.\" Automatically generated by Pod::Man 2.27 (Pod::Simple 3.28) .\" .\" Standard preamble: .\" ======================================================================== .de Sp \" Vertical space (when we can't use .PP) .if t .sp .5v .if n .sp .. .de Vb \" Begin verbatim text .ft CW .nf .ne \\$1 .. .de Ve \" End verbatim text .ft R .fi .. .\" Set up some character translations and predefined strings. \*(-- will .\" give an unbreakable dash, \*(PI will give pi, \*(L" will give a left .\" double quote, and \*(R" will give a right double quote. \*(C+ will .\" give a nicer C++. Capital omega is used to do unbreakable dashes and .\" therefore won't be available. \*(C` and \*(C' expand to `' in nroff, .\" nothing in troff, for use with C<>. .tr \(*W- .ds C+ C\v'-.1v'\h'-1p'\s-2+\h'-1p'+\s0\v'.1v'\h'-1p' .ie n \{\ . ds -- \(*W- . ds PI pi . if (\n(.H=4u)&(1m=24u) .ds -- \(*W\h'-12u'\(*W\h'-12u'-\" diablo 10 pitch . if (\n(.H=4u)&(1m=20u) .ds -- \(*W\h'-12u'\(*W\h'-8u'-\" diablo 12 pitch . ds L" "" . ds R" "" . ds C` "" . ds C' "" 'br\} .el\{\ . ds -- \|\(em\| . ds PI \(*p . ds L" `` . ds R" '' . ds C` . ds C' 'br\} .\" .\" Escape single quotes in literal strings from groff's Unicode transform. .ie \n(.g .ds Aq \(aq .el .ds Aq ' .\" .\" If the F register is turned on, we'll generate index entries on stderr for .\" titles (.TH), headers (.SH), subsections (.SS), items (.Ip), and index .\" entries marked with X<> in POD. Of course, you'll have to process the .\" output yourself in some meaningful fashion. .\" .\" Avoid warning from groff about undefined register 'F'. .de IX .. .nr rF 0 .if \n(.g .if rF .nr rF 1 .if (\n(rF:(\n(.g==0)) \{ . if \nF \{ . de IX . tm Index:\\$1\t\\n%\t"\\$2" .. . if !\nF==2 \{ . nr % 0 . nr F 2 . \} . \} .\} .rr rF .\" .\" Accent mark definitions (@(#)ms.acc 1.5 88/02/08 SMI; from UCB 4.2). .\" Fear. Run. Save yourself. No user-serviceable parts. . \" fudge factors for nroff and troff .if n \{\ . ds #H 0 . ds #V .8m . ds #F .3m . ds #[ \f1 . ds #] \fP .\} .if t \{\ . ds #H ((1u-(\\\\n(.fu%2u))*.13m) . ds #V .6m . ds #F 0 . ds #[ \& . ds #] \& .\} . \" simple accents for nroff and troff .if n \{\ . ds ' \& . ds ` \& . ds ^ \& . ds , \& . ds ~ ~ . ds / .\} .if t \{\ . ds ' \\k:\h'-(\\n(.wu*8/10-\*(#H)'\'\h"|\\n:u" . ds ` \\k:\h'-(\\n(.wu*8/10-\*(#H)'\`\h'|\\n:u' . ds ^ \\k:\h'-(\\n(.wu*10/11-\*(#H)'^\h'|\\n:u' . ds , \\k:\h'-(\\n(.wu*8/10)',\h'|\\n:u' . ds ~ \\k:\h'-(\\n(.wu-\*(#H-.1m)'~\h'|\\n:u' . ds / \\k:\h'-(\\n(.wu*8/10-\*(#H)'\z\(sl\h'|\\n:u' .\} . \" troff and (daisy-wheel) nroff accents .ds : \\k:\h'-(\\n(.wu*8/10-\*(#H+.1m+\*(#F)'\v'-\*(#V'\z.\h'.2m+\*(#F'.\h'|\\n:u'\v'\*(#V' .ds 8 \h'\*(#H'\(*b\h'-\*(#H' .ds o \\k:\h'-(\\n(.wu+\w'\(de'u-\*(#H)/2u'\v'-.3n'\*(#[\z\(de\v'.3n'\h'|\\n:u'\*(#] .ds d- \h'\*(#H'\(pd\h'-\w'~'u'\v'-.25m'\f2\(hy\fP\v'.25m'\h'-\*(#H' .ds D- D\\k:\h'-\w'D'u'\v'-.11m'\z\(hy\v'.11m'\h'|\\n:u' .ds th \*(#[\v'.3m'\s+1I\s-1\v'-.3m'\h'-(\w'I'u*2/3)'\s-1o\s+1\*(#] .ds Th \*(#[\s+2I\s-2\h'-\w'I'u*3/5'\v'-.3m'o\v'.3m'\*(#] .ds ae a\h'-(\w'a'u*4/10)'e .ds Ae A\h'-(\w'A'u*4/10)'E . \" corrections for vroff .if v .ds ~ \\k:\h'-(\\n(.wu*9/10-\*(#H)'\s-2\u~\d\s+2\h'|\\n:u' .if v .ds ^ \\k:\h'-(\\n(.wu*10/11-\*(#H)'\v'-.4m'^\v'.4m'\h'|\\n:u' . \" for low resolution devices (crt and lpr) .if \n(.H>23 .if \n(.V>19 \ \{\ . ds : e . ds 8 ss . ds o a . ds d- d\h'-1'\(ga . ds D- D\h'-1'\(hy . ds th \o'bp' . ds Th \o'LP' . ds ae ae . ds Ae AE .\} .rm #[ #] #H #V #F C .\" ======================================================================== .\" .IX Title "GETDATA 1" .TH GETDATA 1 "2013-10-28" "perl v5.18.1" "User Contributed Perl Documentation" .\" For nroff, turn off justification. Always turn off hyphenation; it makes .\" way too many mistakes in technical documents. .if n .ad l .nh .SH "NAME" getData \- retrieves databases from the Internet .SH "SYNOPSIS" .IX Header "SYNOPSIS" getData [ \-\-mirrordir ] .PP getData \-\-list .SH "DESCRIPTION" .IX Header "DESCRIPTION" Bioinformatics has the intrinsic problem to bring the biological data to the end user. Astronomers have the equivalent problem and particle physicists, well, they haven come up with (first) the web and (second) the computational grids to address their problems. Debian helps with the programs but will not provide such huge datasets that are even frequently updated \- not even in volatile.debian.org. Most bioinformatics researchers will not need too many of such databases. And even more so will gladly continue in using public services remotely. .PP For those who need a set of databases on a regular basis, this script shall be a start to automate the burden to download the data and update indices and the like. The world has seen such magic before with the Lion Biosciences Prisma tool (http://bib.oxfordjournals.org/cgi/reprint/3/4/389.pdf) but how about something simpler (as a start) that at least gets close to what we desire and is Free. The aim must be to address the needs of all (most) communities, not only of the bioinformatics world. The seed was hence made with databases from astronomy. .PP Please contact the Debian-Med community if you consider this program to be almost ready for your needs and explain what still needs to be added. Public databases that you managed to integrate with this system are also very warmly welcomed as feedback. .SH "OPTIONS" .IX Header "OPTIONS" .IP "\-\-help" 4 .IX Item "--help" .Vb 1 \& this help .Ve .IP "\-\-man" 4 .IX Item "--man" Present a more detailed description in form of a man page. .IP "\-\-verbose" 4 .IX Item "--verbose" Say one or two words more than required. .IP "\-\-mirrordir " 4 .IX Item "--mirrordir " Specifies destination directory. The data will be mirrored to the folder \f(CW$mirrordir\fR/$dbname/. Please be aware that this mirrordir is nowhere stored. The directory can consequently be moved to arbitrary locations at any time, if the users of the data are only informed about that moving. .IP "\-\-list" 4 .IX Item "--list" Lists all databases that may be requested to be installed. .IP "" 4 .IX Item "" Only those databases that are explicitly requested to be downloaded will be downloaded. Such databases may require considerable bandwidth, so please make sure you know you are doing the right thing. .IP "\-\-post" 4 .IX Item "--post" Perform only the unpacking/indexing, but do not retrieve/update the databases. This option is considered useful when adding a new database management system to the system, e.g. after installing \s-1EMBOSS.\s0 .IP "\-\-source" 4 .IX Item "--source" Perform only the unpacking/indexing, but do not retrieve/update the databases. This option may be beneficial when the site administator is aware of current analyses that should not be disturbed by the indexing process but the downloading from the net can already be started. .IP "\-\-confd " 4 .IX Item "--confd " Allows for the specification of a directory in which multiple files can be stored that will be read by getData upon its invocation. These may add values to the global variable \f(CW%toBeMirrored\fR that specifies the databases and their download scripts. .IP "\-\-config " 4 .IX Item "--config " Preparation of the configuration file that would be reuired for a particular system that deals with the database. The configuration is printed to stdout and is expected to be copied manually to the proper file or folder. One could imagine this process to be automated, though this is not yet implemented. Currently available is support for two systems: .RS 4 .IP "emboss" 8 .IX Item "emboss" This specifies the \s-1EMBOSS\s0 suite of tools for bioinformatics (www.emboss.org) that is also available as a Debian package. The configuration for the Uniprot databases will allow the sequence retrieval with the seqret tool. .IP "dre \- \s-1ARC\s0 Grid Runtime Environment" 8 .IX Item "dre - ARC Grid Runtime Environment" Runtime environments (REs) are a concept of the \s-1ARC\s0 grid middleware of which more can be learned on http://www.nordugrid.org. A script is needed to indicate the presence of a runtime environment. Here, the name of the script is important, which is not definable by getData though since it only writes to stdout. .RE .RS 4 .Sp Unfortunately, the configuration was not yet be found to be modularised. It all needs to happen within the getData script itself. .RE .IP "\-\-remove " 4 .IX Item "--remove " This command removes folders that store the data. In principle this could be perfomed manually, though some databases may have special requirements pre\- or post-removal, which can be specified individually for every database. .SH "SPECIFICATION OF DATABASES" .IX Header "SPECIFICATION OF DATABASES" Databases for download and their post-processing are specified at two different locations. One is the getData script itself, the other are files stored in /etc/getData.d. Either will define elements of a considerably large hash. The key is the identifier which is also shown by the 'getData \-\-list' directive. The value is a reference to another hash, which assigns values to all the properties that a database has for its download and post-processing: .IP "name \- a human-readable pretty-printed name or short description that makes clear to the world what this database is about." 4 .IX Item "name - a human-readable pretty-printed name or short description that makes clear to the world what this database is about." A bad example is the mere assignment of \*(L"\s-1DE405\*(R",\s0 which few people understand. A better example is \*(L"Pfam-A : Manually curated protein families and domains, only the seed is presented.\*(R". One could argue that one should have that field renamed to \*(L"description\*(R". .IP "source \- shell commands to perform the initial download and subsequent updates" 4 .IX Item "source - shell commands to perform the initial download and subsequent updates" Commonly the wget tool is used for download. The such presented little script is executed underneath the mirrordir directory. One simple example is \*(L"wget \-\-mirror ftp://ssd.jpl.nasa.gov/pub/eph/export/unix/unxp2[01]*.405\*(R". With increasing proficiency in using wget, one is tempted to substitute \&\*(L"\-\-mirror\*(R" with \*(L"\-\-recursive \-\-no\-host\-directories \-\-no\-directories \&\-\-level 1 \-\-no\-parent\*(R". .IP "post-download \- shell commands to perform after the data has been downloaded." 4 .IX Item "post-download - shell commands to perform after the data has been downloaded." A simple (and unnecessary when used the right flags to wget) example is the mere setting of a symbolic link: .Sp .Vb 1 \& "post\-download" => "ln \-s ssd.jpl.nasa.gov/pub/eph/export/unix/unxp*.405 ." .Ve .Sp Some more effort has been put into TrEMBL for the merging of releases with subsequent updates and the indexing for \s-1EMBOSS:\s0 .Sp .Vb 6 \& "d=uncompressed; if [ ! \-d \e$d ]; then mkdir \e$d; fi; " \& ."rm \-rf \e$d/trembl.dat; " \& ."(find ftp.ebi.ac.uk \-name \*(Aq*.dat.gz\*(Aq | xargs \-r zcat ) > \e$d/trembl.dat; " \& ."[ \-x /usr/bin/dbxflat ] " \& . "&& cd \e$d && " \& . "dbxflat \-dbresource embl \-dbname trembllocal \-idformat swiss \-filenames=trembl.dat \-fields id,acc \-auto", .Ve .Sp The dots are connecting strings in Perl. This helps the readability of the code. When writing these scripts, please be aware the newlines don't separate the individual commands here. Semicolon are required. .IP "recommends \- suggests a series of packages to be present for the use of the database or the performance of the indexing." 4 .IX Item "recommends - suggests a series of packages to be present for the use of the database or the performance of the indexing." This information is not used at the moment, also to render this script more useful for other Linux distributions than Debian. .IP "getWgetOptions \- private command to get wget options" 4 .IX Item "getWgetOptions - private command to get wget options" This is used at download time by makefiles, is not intended to be used interactively, and could be removed anytime. .SH "EXAMPLES" .IX Header "EXAMPLES" The following will list the identifiers and the descriptions of the first 4 databases that area available via getData on your system. .PP .Vb 1 \& ./getData \-\-mirrordir=/local/databases/mirrored \-\-list | head 4 .Ve .PP To install any particular database, only give its name as an argument. If the installation is performed at another directory than the default, then the \-\-mirrordir needs again to be set. .PP .Vb 1 \& ./getData swiss.dat .Ve .PP To remove the database again, give the script a hint with the \-\-remove flag .PP .Vb 1 \& ./getData \-\-remove swiss.dat .Ve .PP To perform the indexing only and circumvent the download (attention, this is dangerous since the index files will look newer than the database is), do .PP .Vb 1 \& ./getData \-\-post swiss.dat .Ve .PP A special exception to these extra scripts is the \-\-config flag in that it takes a list of extra arguments. Each shall denote a particular system that this database may be of interest for. There are today two systems supported: .SH "TODO" .IX Header "TODO" We now need a mechanism with which packages can specify hooks that shall be called upon an update of a database. But we cannot assume that every indexing that can be performed because of the installation of some package is also desired by the user. How to configure this properly is left to be decided. .SH "SEE ALSO" .IX Header "SEE ALSO" http://debian\-med.alioth.debian.org, http://wiki.debian.org/DebianMed, /etc/getData.conf .SH "AUTHORS" .IX Header "AUTHORS" This script was prepared by Steffen Moeller and Charles Plessy and is distributed under the terms of the \s-1GNU\s0 Public License (\s-1GPL\s0). On Debian systems, this license can be found under /usr/share/common\-licenses/GPL.