table of contents
PEGASUS-TRANSFER(1) | Pegasus Manual | PEGASUS-TRANSFER(1) |
NAME¶
pegasus-transfer - Handles data transfers in Pegasus workflows.SYNOPSIS¶
pegasus-transfer [-h] [ --file inputfile] [ --threads number_threads] [ --max-attempts attempts] [ --debug]
DESCRIPTION¶
pegasus-transfer takes a list of url pairs, either on stdin or with an input file, determines the correct tool to use for the transfer and executes the transfer. Some of the protocols pegasus-transfer can handle are GridFTP, SRM, Amazon S3, HTTP, and local cp/symlinking. Failed transfers are retried. Note that pegasus-transfer is a tool mostly used internally in Pegasus workflows, but the tool can be used stand alone as well.OPTIONS¶
-h, --helpPrints a usage summary with all the available
command-line options.
-f inputfile, --file inputfile
File with input pairs. If not given, stdin will be
used.
-m, --max-attempts attempts
Maximum number of attempts for retrying failed
transfers.
-t, --threads number_threads
The number of threads to use. This controls the
parallelism of transfers.
-d, --debug
Enables debugging output.
EXAMPLE¶
$ pegasus-transfer # src 1 local_site file:///etc/hosts # dst 1 local_site file:///tmp/foo CTRL+D
CREDENTIAL HANDLING¶
Credentials used for transfers can be specified with a combination of comments in the input file format and environment variables. For example, give the following input file:# src 1 isi gsiftp://workflow.isi.edu/data/file.dat # dst 1 tacc_stampede gsiftp://gridftp.stampede.tacc.utexas.edu/scratch/file.dat
THREADING¶
In order to speed up data transfers, pegasus-transfer will start a set of transfers in parallel using threads. Threads are turned off when retrying failed transfers.AUTHOR¶
Pegasus Team http://pegasus.isi.edu07/30/2014 | Pegasus 4.4.0 |