Meta Demo Info Page


Introduction

The Meta Project (www.openresource.com/Meta) is, ultimately, concerned with documenting and tracking every significant version of every interesting Open Source package (including operating system releases). As a parallel effort, the project envisions distributed support systems that take advantage of this information, folding in issues such as local configurations and practices.

Obviously, this is a huge project; however intriguing it may be as a fantasy, it is appallingly large as an implementation effort. Far too challenging, in fact, for the current resources at hand. So, we have chosen to implement a realizable subset.

Using Perl-based CGI scripts, the Meta Demo indexes the file system and documentation of a "vanilla" FreeBSD installation. Initially, this covered only FreeBSD 4.1, but we have added coverage for FreeBSD 4.2 and plan to cover following FreeBSD releases. We are also looking into covering other operating system variants and versions; see below for more information.

We hope to assess the utility of this information base, using random inquiries from an interested user community. In the process, we will take advantage of both usage logs and user feedback to improve our information base and presentation software.

The Meta Demo

Each page of the demo displays a variety of information about a specified path name or manual page. This information can help to inform (or remind) the user of related resources for the item in question. It can also serve as a starting point for some serious documentation browsing (:-).

The demo is quite limited, to be sure. It is, after all, an early version of a proof-of-concept demonstration of a realizable subset of a much larger fantasy. Despite all of these limitations, it already appears to be both interesting and useful.

Implementation

The demo is implemented as a Perl-based CGI script, fed by a host of information-gathering filters. BerkeleyDB::Btree is used to store a tied hash of (line-oriented and XML format) metadata, covering a given OS instance (e.g., FreeBSD 4.2). Perl and XML::Simple are used to parse the returned data. The metadata comes from:

  • annotation files
  • hard-coded rules
  • system information

Thus, a link might be displayed because of an annotation file entry, a rule in the CGI script, or a "FILES" entry in a man page. To help users understand the demo's "reasoning" in listing a given item, each link has an associated "?" link. This brings up a "linkage trail" page, which lists and explains the trail of links that led from the original item to the item in question.

As of mid-March 2001, the demo's code base contains (roughly):

  • 2,500 lines of Perl
  • 20,000 lines of hand-edited XML
  • 5,000,000 lines of generated XML

Until very recently, the tied hash consisted of XML entries, indexed by the full path names of files and directories. It now contains a mix of line-oriented and XML formats, and indexing goes down to the variable level. This change increased the size of the hash, but halved the run time of the CGI script. In short, a worthwhile trade-off...

Futures

Although the demo's long-term goals are subsumed in those of the Meta Project, some short-term goals are worth mentioning:

  • better tuning of the graph traversal rules
  • graphical display of selected "subsystems"
  • increased coverage and detail in the metadata

If the demo appears to be useful, several steps can be taken to improve its performance:

  • switch to faster (and perhaps more) hardware
  • supply the server(s) with better bandwidth
  • force production and caching of popular pages
  • recode to use FastCGI, an RDBMS, etc.

In the medium term, we hope to greatly expand the demo's depth and breadth. Online browsing of cross-linked files (e.g., source code and documentation) is quite feasible. Coverage of add-on packages and other operating systems (e.g., BSD, Linux, or even proprietary Unix variants) is also a strong possibility. Given sufficient resources, the demo could be expanded into an industry-wide resource. If you are interested in helping us in this effort, please get in touch (rdm@cfcl.com).