OAI-PMH is intended as a standard way to move metadata from one point to another within the virtual information space of the World Wide Web. It supports interaction between a data provider and a service provider—a renaming of the client-server model that emphasizes that interaction is driven by the client (the service provider) and that the client alone has the onus of deciding what services are offered to users. Data providers, in contrast, are in the business of managing repositories. They export on-demand data records in a standardized form, unencumbered by any consideration of how the information will be used. If other operations, such as text searching, are to be supported, they must be performed by the service provider, not the data provider. Of course, a site may choose to be both a service provider and a data provider. It may also manage more than one repository.

A key technical goal of the protocol is that it be easily implemented using readily available software tools and support. The protocol provides a framework, using HTTP, in which requests are encoded in URLs and executed by programs on the server. Results are returned as XML records. Figure 7.1 shows the basic form of interaction using an example that requests a record pertaining to a particular document identifier and expressed using Dublin Core metadata. First the arguments are encoded into a URL (characters like : and / must be specially coded) and dispatched over HTTP. The response is an XML document that draws heavily on XML namespaces and schemas.

XML namespaces are a way of ensuring that markup tags intended for a particular purpose do not conflict with other tag sets, and XML Schema is a way of extending the idea of predefined tag sets and document structure with typed content. For example, responseDate in OAI is defined to be the complete date, including hours, minutes, and seconds, using the schema type xsd:timeInstant.

The top-level element in the example record is GetRecord. Through its attributes this sets up a suitable namespace using a URL in the Open Archives Web site and specifies the defining XML Schema stored at the same site. Nested deeper in the GetRecord structure is a metadata element that uses the Dublin Core namespace and a schema specifically for this metadata standard that is also defined at the Open Archives site.

Within the metadata element is the main content—expressed, as requested, using Dublin Core metadata—for the digital item that matches document identifier oai:nzdl:hdl/018cf2f4256b4c8827e747b8. The book is titled Farming Snails and was written for the Food and Agriculture Organization of the United Nations (and part of the Humanity Development Library featured in Chapters 1 and 3). Further information supplies a brief description of its content, gives the year of publication, declares that it is principally text, and states its format as HTML. This particular record is rich in metadata, but the protocol does not require this level of richness.

The protocol supports six services through the verb argument, and Table 7.2 summarizes them. Identify and ListSets are services that are typically called early on in a client's interchange with a server to establish a broad picture of the repository. Figure 7.2 shows an example of a response to an Identify request, which contains administrative information about the server: in this case, the OCLC's OAICat Repository Framework, whose repository name is IDEALS @ UIUC. ListIdentifiers is a way of receiving all the document identifiers or a group that matches a stipulated set name. ListMetadataFormats can be applied to the repository as a whole or to a particular document within it to establish which metadata formats are supported. Dublin Core is mandatory, but other formats, such as MARC, which has the capacity to export a greater volume of metadata per record, may also be supported. We have already seen GetRecord in action in the example just given— ListRecords is similar except that more than one record can be returned, and group selection is possible with the same set-naming technique used by ListIdentifiers.

These are the general facilities supported by the protocol. Greater flexibility can be achieved using the input arguments and the set and resumption mechanisms. First, repository items can be categorized into sets—multiple hierarchies, where an item can be in more than one hierarchy or in none at all. Each node in the hierarchy has a set name, such as beekeeping, and a descriptive name, such as Beekeeping and honey extraction. Its hierarchical position is specified by concatenating set names separated by colons—like husbandry:beekeeping. ListSets returns this information for the complete repository, marked up as XML.

The set mechanism means that instead of retrieving all the document identifiers, a service provider can restrict the information returned by ListIdentifiers to a particular set. (The same applies to ListRecords.) Because records in the repository are date-stamped, the information returned can be restricted to a particular range of dates using from and until arguments. Even so, the list of data returned may still be excessively long. If so, a data provider might transmit part of the data and include a resumption token in the record. This mechanism enables the service provider to contact the data provider again and request the next installment (which might include another resumption token).

The OAI protocol makes use of the HTTP status code mechanism to indicate the success or failure of a request. Status code 400, in this context, indicates a syntactic error in the request, such as an invalid input argument. Other forms of failure, such as a repository item being unavailable in the requested metadata format, produce null data in the XML record that is returned.

OAI-PMH imposes no requirements on how metadata is represented on the server of a data provider. The metadata can be stored in a database, as simple XML files, or even embedded within other documents. All that matters is that the data provider responds to protocol requests with appropriate responses; in the case of metadata embedded in documents, the metadata would first have to be extracted and then packaged up as an OAI-PMH response.

It is even possible to have a static metadata repository. A static repository is a single XML file that contains all the metadata a data provider wishes to share. This file needs to be registered with a Static Repository Gateway, which then performs the appropriate transformations to allow service providers to access the metadata. Static repositories are only really appropriate for small collections that do not change frequently. Most digital library systems come with OAI-PMH support built in, although they may only support basic Dublin Core metadata. Support for qualified Dublin Core (or MODS, METS, etc.) is rarer; as a result, although a digital collection may have detailed internal metadata, only a subset may be available via OAI-PMH. Naturally, such restrictions limit the applications that OAI service providers can create.

OAI-PMH was designed to allow structured retrieval of metadata on the Web. We can usefully contrast this with the unstructured approach of Web harvesting, or Web mirroring. When the remote resources are as varied as Web content, then most attempts to gather metadata will produce uneven results. There is no standard metadata scheme for Web content, and although metadata like Dublin Core is sometimes embedded within HTML, it is not present in sufficient quantity or with enough reliability to be of much use. OAI-PMH, on the other hand, provides structure through a defined protocol, which is a necessary condition for effective metadata sharing. However, a protocol by itself does not guarantee that the metadata available is of good quality.

Creating and sharing quality metadata is not a straightforward task. It requires appropriate tools, trained staff, and a long-term commitment to resource availability. Although the technologies may be relatively simple, this is only a necessary condition for success, and without the associated human support it will not be sufficient. One example of a large-scale effort to harvest OAI metadata has been the U.S. National Science Digital Library (NSDL), which began in 2003 to implement a digital library based on metadata aggregation using Dublin Core and OAI-PMH. However, a retrospective analysis in 2006 found that these standards, which are supposed to be easy to adopt with a low human cost, were harder to deploy than expected.

The comparison here is with WorldCat, the world's largest bibliographic database. WorldCat combines the collections of more than 10,000 libraries from over 100 countries that participate in the Online Computer Library Center global cooperative. It comprises 125 million different records pointing to physical and digital assets in more than 500 languages. (These statistics quickly become outdated, however, because a new record is added to the database on average every 10 seconds.)

Overall, OAI-PMH has been a successful method of sharing metadata. Its widespread adoption is due to the simplicity of the protocol and its building on top of other existing standards: HTTP, XML, and Dublin Core.

An early technical solution to the problem of persistence was the Persistent URL (PURL), operated by the Online Computer Library Center (OCLC). Instead of describing the location of the resource to be retrieved, a PURL gives an intermediate, persistent, location which, when retrieved, is automatically redirected to the current location of the resource. A similar technical solution called Handle, from the Communications Network Research Institute (CNRI), is implemented as a Java-based program for assigning, managing, and resolving persistent identifiers for digital objects on the Internet.

The PURL and Handle systems (and the DOIs discussed below) are similar in that they maintain a database of the relationship between a name and an actual URL. When a user or a piece of software has the name, it uses a resolution service to convert the name into an actual URL. If the actual URLs need to be changed, typically because the content has moved to a new server, only the database is updated.

Handles can be resolved by typing into any Web browser the string

followed by the Handle itself. For example, a paper called Building digital library collections with Greenstone can be found in the University of Waikato Research Commons at http://hdl.handle.net/10289/1825. The resolution service (at http://hdl.handle.net) takes the object's Handle (10289/1825) and returns a URL (at the time of writing, http://waikato.researchgateway.ac.nz/handle/10289/1825). When used in a Web browser, the effect is to seamlessly navigate to the actual URL. However, Handles can be used just as easily by software applications outside of a browser, provided they use the HTTP protocol.

Achieving persistence is not really a technical problem, or even a naming problem. It's an organizational problem. To make URLs persistent, all you really need is a mechanism for assigning names, and an organization that administers them and records their locations—an organization that is expected to persist over time and is scalable enough to meet the world's needs. A scheme that is actually in use is the Digital Object Identifier (DOI), which is a de facto practical implementation of the URN and URI specifications using the Handle system. In fact, DOIs are Handles with the prefix “10.” (The Handle system also incorporates other namespaces.)

The International DOI Foundation defines a Digital Object Identifier as a permanent identifier for any object of intellectual property and explains that it is used to identify a piece of intellectual property on a digital network and associate related data with it in an extensible way. A typical use is to give a scholarly article a unique identification number that anyone can use to obtain information about its location on a digital network. DOIs can easily be resolved: just type in the address bar of any Web browser the string

followed by the DOI of the document. For example, the paper mentioned above has the DOI 10.1145/1065385.1065530, and you can see it at http://dx.doi.org/10.1145/1065385.1065530. The resolution service (at http://dx.doi.org/) takes the object's name (10.1145/1065385.1065530) and returns a URL (http://portal.acm.org/citation.cfm?doid=1065385.1065530). This identifies a copy of the document in the ACM Digital Library, rather than the one mentioned above at the University of Waikato.

DOIs are administered by the International DOI Foundation, a consortium that includes both commercial and non-commercial partners, and are currently being standardized through ISO. The structure of the DOI conveys these pieces of information:

As noted earlier, DOIs build upon the Handle system, and DOIs are Handles with the directory code “10,” which refers to the International DOI Foundation. The publisher ID 1145 indicates the Association for Computing Machinery (ACM). Other examples include 1000, which is used by the International DOI Foundation itself, and 1007, which is used by the Springer publishing house.

The prefixes are guaranteed unique because they are registered with the global Handle Registry, and the Handle syntax ensures that suffixes are unique within the namespace defined by the prefix. The DOI directory code (10 above) is a prefix in the Handle system, and the DOI prefix 10.1145 is known as a derived prefix. Because DOIs are themselves Handles, the same document that we discussed above can also be found in the ACM Digital Library by resolving the DOI through the central Handle server: http://hdl.handle.net/10.1145/1065385.1065530.

DOIs are administered by agencies recognized by the International DOI Foundation. They register identifiers, allocate prefixes, and allow registrants to declare and maintain metadata. The agencies are commercial: they levy a small charge to establish each new identifier. Practically, a publisher using this scheme uploads a file to a DOI Registration Agency, which links a DOI with the actual URL. If the URL changes, the publisher uploads the revised DOI-URL association.

The purpose of persistent URLs is to create reliable identifiers that can be used to link to a document, no matter where it is stored. A different problem is to discover a link to a document when all you have is its metadata. This is accomplished by a scheme called OpenURL, which is often run as a service by libraries. An OpenURL is generated by a bibliographic citation or bibliographic record; its target is a resource that satisfies the user's information need.

An OpenURL consists of a base URL that usually addresses an institutional link server and a query string that specifies metadata in the form of key–value pairs. For example, http://www.oxfordjournals.org/content?genre=article&issn=0006-8950&volume=126&issue=2&spage=413 specifies a particular article at the base URL www.oxfordjournals.org by giving its serial number, volume, issue, and page number. In the case of this particular OpenURL resolver, the first four parameters—genre, ISSN (International Standard Serial Number), volume, and issue—are mandatory, but metadata like title and author can be given as an alternative to the page number.

The idea of OpenURLs is that a user can query an online bibliographic database and end up with a pointer to the full text of the article he wants to read. Whether the user is able to download the article depends on whether his institution has an appropriate subscription. To determine this, the link needs to be resolved by the user's institution to point to the appropriate institutional copy. An OpenURL can be made to resolve to a copy of a resource in a different library simply by changing the base URL. Thus, the same OpenURL can easily be adjusted by either library to provide access to its own copy of the resource.

OpenURLs work well for documents that have standard metadata associated with them. In order to refer to general documents on the Web, a scheme involving associative links has been proposed. Pages can be identified not just by a name or location, but also by their content—the words they contain. A few well-chosen words or phrases are sufficient to identify almost every Web page exactly, with high reliability. These snippets—called the page's signature—could be added to the URL. Browsers would locate pages in the normal way, but if they encountered a broken-link error, they would pass the signature to a search engine to find the page's new location. The same technique identifies all copies of the same page on the Web—an important facility for search engines, so they need not bother to index duplicates. The scheme relies on the uniqueness of the signature: there is a trade-off between signature size and reliability.

As noted above, persistence is really an organizational problem, and any solutions require the organizations involved to be perceived as stable and long-lasting. Some organizations, such as the Library of Congress or the International Organization for Standardization (ISO), have obvious longevity and national/international stature. Newer organizations, such as the Communications Network Research Institute that runs the Handle system and the International DOI Foundation that runs the DOI system, rely on technical competence and their wide user base.

As described in Section 7.1, Z39.50 is a complex protocol that was developed before the Web really took off. More recently, its basic ideas have been re-engineered for the Web environment, in the form of the Search/Retrieval via URL (SRU) protocol. SRU can operate in two flavors, with requests in HTTP (either GET or POST requests) or as a SOAP Web Service. The latter used to be called the Search/Retrieval Web Service (SRW).

Whereas OAI has six verbs, SRU has three main operations:

Figure 7.5 shows an SRU response to a searchRetrieve query. The response returns a metadata record, and in this case the metadata is couched as a MODS record (Section 6.2). The recordData contains origin information (publisher, place and date issued), title information ( Five Songs), personal name and role ( Mary Lane Bayliff, creator), language, physical description, and information about local library holdings. The explain operation mirrors Z39.50's Explain Facility mentioned in Section 7.1. The scan operation allows the client to request a range of the available terms at a given point within a list of indexed terms. It enables clients to present users with an ordered list of values and to show how many hits there would be for each one. This operation is often used to select terms for subsequent searching or to verify a negative search result.

SRU services are more fine-grained than those of OAI-PMH. They allow clients to retrieve records with greater precision. It is perfectly reasonable for a digital library to provide access to its resources through both protocols and to allow clients to choose the one that best suits their needs.

In the year 2000, Hewlett-Packard teamed up with MIT's library to develop a system to meet the latter's need for an institutional repository. The result was called DSpace, for “Document space.” When the repository was launched in 2002, the software was simultaneously released as open source for others to use. Chapter 3 illustrated DSpace from the end-user's perspective for searching and browsing. We return to one of the documents viewed there, Kofi Annan's Master's thesis, and consider the steps required to enter it into the system.

Figure 7.6 shows snapshots from the document submission process. To have reached this point, the digital librarian (or end-user, since institutional repositories often allow registered users to submit documents) has first had to authenticate herself using a facility built into DSpace. Stages in the upload sequence are displayed along the top of each screenshot. The first three stages, labeled “describe,” gather metadata from the user. Figure 7.6a shows the second step, soliciting the title, author, and type of document (article, book, thesis, etc.).

Next, the librarian selects the file to upload. In Figure 7.6b she has pressed the browse button and located the file—in this case a PDF document—in a pop-up window. Pressing Next produces Figure 7.6c (after an uploading delay), where she can check the uploaded data—for example, its file size (roughly 6 MB in this case) and type. Clicking on the hyperlinked filename shows the uploaded version; through this page the librarian can also rectify errors, such as uploading the wrong file or its type being incorrectly assigned. The Show checksums button below computes the MD5 checksum of the uploaded file, which allows the librarian an opportunity to verify the integrity of the bytes that have been transferred. To do this, the librarian computes the checksum for the original file on her computer and then checks that the value she arrived at matches the value displayed in DSpace.

Figure 7.6d, the next page, shows a summary of the information entered (and derived). Each section has a button that allows the librarian to edit the information. An additional step (not shown) displays the license that is to be used and requires the librarian to confirm that permission is granted, whereupon the item is finally submitted to DSpace and (ultimately) can be viewed as illustrated in Chapter 3 (Figure 3.22h). The exact procedure depends on how DSpace is configured. Often there is a further round of checking and quality control before the item is published in the repository. If the librarian leaves the submission sequence partway through, the information she has entered so far is stored under her DSpace username and the next time she logs in she can pick up the process where she left off.

The metadata values entered in DSpace are matched to Dublin Core elements, and the software includes an OAI-PMH server. Because it's written in Java, it is Unicode compliant, and the interface has been translated into different languages. DSpace is servlet-based and makes extensive use of JavaServer Pages (JSP). Its developers recommend that it be used in conjunction with the Tomcat Web server, using PostgreSQL or Oracle as the database management system.

DSpace treats documents as black boxes to which metadata is attached. It is therefore important to identify each document's MIME type correctly, because this dictates which helper application a Web browser will use to view it when it is opened. However, DSpace does recognize Word, PDF, HTML, and plain text documents and can be configured to support full-text indexing using the open source indexing package Lucene. Documents can be submitted by librarians or registered users individually over the Web or may be ingested as a batch by command-line scripts that run on the DSpace server.

Fedora (Flexible Extensible Digital Object Repository Architecture) began in 1997 at Cornell University as a conceptual design backed up by a reference implementation. In 1999, the University of Virginia library used this reference implementation as a foundation for developing a digital library tailored to their needs. The two groups (note, incidentally, another pairing of a technology research group with a large-scale library, as for DSpace) ended up working together to produce an open source digital library toolkit for others to use.

Fedora is based on a powerful digital object model, encapsulated as a repository that is extremely flexible and configurable. Moreover, a repository stores all kinds of objects, not just the documents placed in it for presentation to the end-user (we shall see examples shortly). In contrast to DSpace, Fedora is not ready to use out of the box, a deliberate design decision by its developers that means Fedora can be used to develop a wide variety of digital libraries with radically different functionality. However, there is a price to be paid: you cannot use Fedora without programming support by IT staff.

Figure 7.7 shows the default Web interface for a repository that contains a collection of historic maps. This interface is not intended for end-users, but rather to indicate the possibilities and to allow developers to explore the system's capabilities. In Figure 7.7a, the user has called up the search page and sought documents that include the word Africa (searching across all indexed fields). The array of check-boxes determine which fields are displayed in the result list, in this case the document's identifier (persistent identifier, or pid) and title. In Figure 7.7b, the user has clicked on the third item, Africae tabula nova, and is viewing its associated information.

A datastream in Fedora is MIME-encoded data associated with an object: for instance, a source file, such as an image or PDF file, or XML data. Disseminators are methods that act upon an object, and they are linked to Web Services that provide dynamic capabilities that access the object's datastreams. Figure 7.7b, called the Item Index view, is the result of running the default disseminator, which lists the object's datastreams. Datastream names are hyperlinked, and clicking them serves up the underlying MIME-encoded data source. For instance, clicking DC yields an XML datastream representing the object's Dublin Core metadata. In this example we would see that <dc:Subject> is set to Africa, which explains why the item was found despite the fact that its title ( Africae) is a Latin derivative of the stem ( Africa). Clicking on the url datastream brings up Figure 7.7c, which displays a high-resolution version of the digitized image.

Figure 7.7d shows the disseminators associated with the object. This image is associated with the content model UVA_STD_IMAGE, which is why these disseminators appear. This content model, provided by Fedora, supports a wide range of image manipulation operations—resizing, zooming, brightening, watermarking, grayscale conversion, cropping, and so forth—through the RELS-EXT datastream (listed in Figure 7.7c). The default Web interface lists the disseminators along with input components (text boxes, radio buttons, and the like) that match the disseminators’ parameters. To crop the image, for example, the user enters the x, y, width, and height parameters and presses the run button to generate a cropped excerpt.

Ingesting, modifying, and presenting information from this rich repository of objects is accomplished through a set of Web Services. The services fall into two core groups, one for access and the other for management, and both RESTful and full SOAP-based versions of the services (Section 7.4) are provided. A further protocol exists for expressing relationships between objects in terms of the Resource Description Framework (RDF, Section 6.4). These protocols have allowed a variety of digital libraries to been developed, including Pergamos (Section 1.3) and the U.S. National Science Digital Library mentioned in Section 7.2, which contains over 4 million items.

The capabilities provided by these protocols can be investigated using the Fedora Administration application illustrated in Figure 7.7e. The Administration application is another aspect of Fedora that is targeted toward IT developers rather than end-users or digital librarians. After they log in, users are presented with a blank virtual desktop upon which windows appear as they interact. For example, new objects can be created and existing ones can be edited or purged from the repository. The repository can be searched, and items can be selected and manipulated. Sets of files can be ingested or modified; they are streamed to the Fedora server using the protocol. Files must be in either FOXML, a native Fedora XML format; Fedora METS, a METS extension defined by the project; or the Atom Syndication Format, an XML extension used for Web feeds. Authentication is provided by the Extensible Access Control Markup Language. The Administration application also gives access to consoles that connect directly with Fedora's management and access facilities.

In Figure 7.7e the user has repeated the query for Africa: the result list is displayed near the bottom. Double-clicking the first item produces the partially obscured window at the top left, through which datastreams and disseminators can be viewed and altered. Here the DC datastream is selected, which presents XML metadata that can be edited directly within the text box or using an external XML editor. Exploring the datastreams for this object, the user views the RELS-EXT datastream (not shown) and finds that the associated content model is UVA_STD_IMAGE, which means that disseminators for cropping, watermarking, and so on are available for this image as well.

Fedora's Content Model Architecture is a way of establishing properties that are shared by a set of digital objects. It is built upon Service Definitions and Service Deployments, which, like the content model itself, are merely additional objects stored in the repository. The former provide abstract methods; the latter, concrete implementations. For example, a Service Definition tailored for images might provide a method for generating thumbnails at runtime suitable for previewing in a Web browser. Further definitions would be provided to handle different image types: perhaps one for JPEG, GIF, and PNG formats (since Java provides standard support for these) and a separate one for JPEG 2000 files, which require additional library support and may not mesh well with the built-in ones. Since the mechanism is accessed through Web Services, it might be implemented in an entirely different programming language. Regardless of which mechanism is used, the fact that images can have associated thumbnails can be included in the content model and shared by a variety of source documents.

Seeking to understand more about the content model associated with the map, our user calls up the UVA_STD_IMAGE content model object, displayed on the right of Figure 7.7e. The RELS-EXT datastream in this object shows the RDF statements that are currently assigned, such as the hasService predicate. Through the interface, additional statements can be added or existing ones can be edited or deleted. Alternatively, the displayed identifier information can be used as the source of new retrieval requests to continue the user's exploration of the repository.

Like DSpace, Fedora is written in Java and makes use of servlets. It ships with the McKoi relational database, a light-weight Java implementation, but can be configured to use other database management systems. Full-text indexing is achieved through a third-party package (GSearch) into which the Lucene, Solr (built on top of Lucene), or Zebra indexing packages can be plugged.

Because Fedora is not a turnkey software solution, additional development is necessary to shape it into a usable end product. However, following the open source philosophy, such modifications can be packaged up and made available to others in the form of ready-to-run software tools. Fez is a Web interface to Fedora that provides an institutional repository. It uses PHP and MySQL and includes notes for users migrating from DSpace or other repository systems. Muradora provides a Web front-end to a Fedora repository that re-factors authentication and authorization into pluggable middleware components. It provides a Shibboleth authentication module (Section 7.5) and also extends the above-mentioned Extensible Access Control Markup Language; in addition, it has rich user-oriented search and browse capabilities. For many of these systems, installation is quite complex. For example, in addition to a Fedora installation, Muradora requires software support for LDAP, DB XML, and XForms.