NAME¶
HTTP::OAI::Harvester - Agent for harvesting from Open Archives version 1.0, 1.1,
2.0 and static ('2.0s') compatible repositories
DESCRIPTION¶
"HTTP::OAI::Harvester" is the harvesting front-end in the OAI-PERL
library.
To harvest from an OAI-PMH compliant repository create an
"HTTP::OAI::Harvester" object using the baseURL option and then call
OAI-PMH methods to request data from the repository. To handle version 1.0/1.1
repositories automatically you
must request "Identify()"
first.
It is recommended that you request an Identify from the Repository and use the
"repository()" method to update the Identify object used by the
harvester.
When making OAI requests the underlying HTTP::OAI::UserAgent module will take
care of automatic redirection (http code 302) and retry-after (http code 503).
OAI-PMH flow control (i.e. resumption tokens) is handled transparently by
"HTTP::OAI::Response".
Static Repository Support¶
Static repositories are automatically and transparently supported within the
existing API. To harvest a static repository specify the repository XML file
using the baseURL argument to HTTP::OAI::Harvester. An initial request is made
that determines whether the base URL specifies a static repository or a normal
OAI 1.x/2.0 CGI repository. To prevent this initial request state the OAI
version using an HTTP::OAI::Identify object e.g.
$h = HTTP::OAI::Harvester->new(
repository=>HTTP::OAI::Identify->new(
baseURL => 'http://arXiv.org/oai2',
version => '2.0',
));
If a static repository is found the response is cached, and further requests are
served by that cache. Static repositories do not support sets, and will result
in a noSetHierarchy error if you try to use sets. You can determine whether
the repository is static by checking the version
($ha->repository->version), which will be "2.0s" for static
repositories.
FURTHER READING¶
You should refer to the Open Archives Protocol version 2.0 and other OAI
documentation, available from
http://www.openarchives.org/.
Note OAI-PMH 1.0 and 1.1 are deprecated.
BEFORE USING EXAMPLES¶
In the examples I use arXiv.org's and cogprints OAI interfaces. To avoid causing
annoyance to their server administrators please contact them before performing
testing or large downloads (or use other, less loaded, servers for testing).
SYNOPSIS¶
use HTTP::OAI;
my $h = new HTTP::OAI::Harvester(baseURL=>'http://arXiv.org/oai2');
my $response = $h->repository($h->Identify)
if( $response->is_error ) {
print "Error requesting Identify:\n",
$response->code . " " . $response->message, "\n";
exit;
}
# Note: repositoryVersion will always be 2.0, $r->version returns
# the actual version the repository is running
print "Repository supports protocol version ", $response->version, "\n";
# Version 1.x repositories don't support metadataPrefix,
# but OAI-PERL will drop the prefix automatically
# if an Identify was requested first (as above)
$response = $h->ListIdentifiers(
metadataPrefix=>'oai_dc',
from=>'2001-02-03',
until=>'2001-04-10'
);
if( $response->is_error ) {
die("Error harvesting: " . $response->message . "\n");
}
print "responseDate => ", $response->responseDate, "\n",
"requestURL => ", $response->requestURL, "\n";
while( my $id = $response->next ) {
print "identifier => ", $id->identifier;
# Only available from OAI 2.0 repositories
print " (", $id->datestamp, ")" if $id->datestamp;
print " (", $id->status, ")" if $id->status;
print "\n";
# Only available from OAI 2.0 repositories
for( $id->setSpec ) {
print "\t", $_, "\n";
}
}
# Using a handler
$response = $h->ListRecords(
metadataPrefix=>'oai_dc',
handlers=>{metadata=>'HTTP::OAI::Metadata::OAI_DC'},
);
while( my $rec = $response->next ) {
print $rec->identifier, "\t",
$rec->datestamp, "\n",
$rec->metadata, "\n";
print join(',', @{$rec->metadata->dc->{'title'}}), "\n";
}
if( $rec->is_error ) {
die $response->message;
}
# Offline parsing
$I = HTTP::OAI::Identify->new();
$I->parse_string($content);
$I->parse_file($fh);
METHODS¶
- HTTP::OAI::Harvester->new( %params )
- This constructor method returns a new instance of
"HTTP::OAI::Harvester". Requires either an HTTP::OAI::Identify
object, which in turn must contain a baseURL, or a baseURL from which to
construct an Identify object.
Any other parameters are passed to the HTTP::OAI::UserAgent module, and from
there to the LWP::UserAgent module.
$h = HTTP::OAI::Harvester->new(
baseURL => 'http://arXiv.org/oai2',
resume=>0, # Suppress automatic resumption
)
$id = $h->repository();
$h->repository($h->Identify);
$h = HTTP::OAI::Harvester->new(
HTTP::OAI::Identify->new(
baseURL => 'http://arXiv.org/oai2',
));
- $h->repository()
- Returns and optionally sets the HTTP::OAI::Identify object used by the
Harvester agent.
- $h->resume( [1] )
- If set to true (default) resumption tokens will automatically be handled
by requesting the next partial list during "next()" calls.
OAI-PMH Verbs¶
The 6 OAI-PMH Verbs are the requests supported by an OAI-PMH interface.
Error Messages¶
Use "is_success()" or "is_error()" on the returned object to
determine whether an error occurred (see HTTP::OAI::Response).
"code()" and "message()" return the error code (200 is
success) and a human-readable message respectively. Errors returned by the
repository can be retrieved using the "errors()" method:
foreach my $error ($r->errors) {
print $error->code, "\t", $error->message, "\n";
}
Note: "is_success()" is true for the OAI Error Code
"noRecordsMatch" (i.e. empty set), although "errors()"
will still contain the OAI error.
Flow Control¶
If the response contained a resumption token this can be retrieved using the
$r->resumptionToken method.
Methods¶
These methods return an object subclassed from HTTP::Response (where the class
corresponds to the verb requested, e.g. "GetRecord" requests return
an "HTTP::OAI::GetRecord" object).
- $r = $h->GetRecord( %params )
- Get a single record from the repository identified by identifier, in
format metadataPrefix.
$gr = $h->GetRecord(
identifier => 'oai:arXiv:hep-th/0001001', # Required
metadataPrefix => 'oai_dc' # Required
);
$rec = $gr->next;
die $rec->message if $rec->is_error;
printf("%s (%s)\n", $rec->identifier, $rec->datestamp);
$dom = $rec->metadata->dom;
- $r = $h->Identify()
- Get information about the repository.
$id = $h->Identify();
print join ',', $id->adminEmail;
- $r = $h->ListIdentifiers( %params )
- Retrieve the identifiers, datestamps, sets and deleted status for all
records within the specified date range (from/until) and set spec (set).
1.x repositories will only return the identifier. Or, resume an existing
harvest by specifying resumptionToken.
$lr = $h->ListIdentifiers(
metadataPrefix => 'oai_dc', # Required
from => '2001-10-01',
until => '2001-10-31',
set=>'physics:hep-th',
);
while($rec = $lr->next)
{
{ ... do something with $rec ... }
}
die $lr->message if $lr->is_error;
- $r = $h->ListMetadataFormats( %params )
- List available metadata formats. Given an identifier the repository should
only return those metadata formats for which that item can be
disseminated.
$lmdf = $h->ListMetadataFormats(
identifier => 'oai:arXiv.org:hep-th/0001001'
);
for($lmdf->metadataFormat) {
print $_->metadataPrefix, "\n";
}
die $lmdf->message if $lmdf->is_error;
- $r = $h->ListRecords( %params )
- Return full records within the specified date range (from/until), set and
metadata format. Or, specify a resumption token to resume a previous
partial harvest.
$lr = $h->ListRecords(
metadataPrefix=>'oai_dc', # Required
from => '2001-10-01',
until => '2001-10-01',
set => 'physics:hep-th',
);
while($rec = $lr->next)
{
{ ... do something with $rec ... }
}
die $lr->message if $lr->is_error;
- $r = $h->ListSets( %params )
- Return a list of sets provided by the repository. The scope of sets is
undefined by OAI-PMH, so therefore may represent any subset of a
collection. Optionally provide a resumption token to resume a previous
partial request.
$ls = $h->ListSets();
while($set = $ls->next)
{
print $set->setSpec, "\n";
}
die $ls->message if $ls->is_error;
AUTHOR¶
These modules have been written by Tim Brody
<tdb01r@ecs.soton.ac.uk>.