.TH "chirp_stream_files" 1 "" "CCTools 7.0.9 FINAL" "Cooperative Computing Tools"

.SH NAME
.LP
\fBchirp_stream_files\fP - move data to/from chirp servers in parallel

.SH SYNOPSIS
.LP
\FC\fBchirp_stream_files [options] <copy|split|join> <localfile> { <hostname[:port]> <remotefile>\fP\FT

.SH DESCRIPTION
.LP

\fBchirp_stream_files\fP is a tool for moving data from one machine to and from many machines, with the option to split or join the file along the way.  It is useful for constructing scatter-gather types of applications on top of Chirp.
.PP
\FCchirp_stream_files copy\FT duplicates a single file to multiple hosts.  The <localfile> argument names a file on the local filesystem.  The command will then open a connection to the following list of hosts, and stream the file to all simultaneously.
.PP
\FCchirp_stream_files split\FT divides an ASCII file up among multiple hosts.  The first line of <localfile> is sent to the first host, the second line to the second, and so on, round-robin.
.PP
\FCchirp_stream_files join\FT collects multiple remote files into one.  The argument <localfile> is opened for writing, and the remote files for reading.  The remote files are read line-by-line and assembled round-robin into the local file.
.PP
In all cases, files are accessed in a streaming manner, making this particularly efficient for processing large files.  A local file name of \FC-\FT indicates standard input or standard output, so that the command can be used in a pipeline.
.SH OPTIONS
.LP

.LP
.TP
\fB-a\fP, \fB-\-auth\fP=\fI<flag>\fP
.
Require this authentication mode.
.TP
\fB-b\fP, \fB-\-block-size\fP=\fI<size>\fP
.
Set transfer buffer size. (default is 1048576 bytes)
.TP
\fB-d\fP, \fB-\-debug\fP=\fI<flag>\fP
.
Enable debugging for this subsystem.
.TP
\fB-i\fP, \fB-\-tickes\fP=\fI<files>\fP
.
Comma-delimited list of tickets to use for authentication.
.TP
\fB-t\fP, \fB-\-timeout\fP=\fI<time>\fP
.
Timeout for failure. (default is 3600s)
.TP
.B \ -v, --version
.
Show program version.
.TP
.B \ -h, --help
.
Show help text.


.SH ENVIRONMENT VARIABLES
.LP
List any environment variables used or set in this section.

.SH EXIT STATUS
.LP
On success, returns zero.  On failure, returns non-zero.

.SH EXAMPLES
.LP

To copy the file \FCmydata\FT to three locations:

.fam C
.nf
.nh
.IP "" 8
% chirp_stream_files copy mydata server1.somewhere.edu /mydata
                                 server2.somewhere.edu /mydata
                                 server2.somewhere.edu /mydata
.fi
.hy
.fam
.P

To split the file \FCmydata\FT into subsets at three locations:

.fam C
.nf
.nh
.IP "" 8
% chirp_stream_files split mydata server1.somewhere.edu /part1
                                  server2.somewhere.edu /part2
                                  server2.somewhere.edu /part3
.fi
.hy
.fam
.P

To join three remote files back into one called \FCnewdata\FT:

.fam C
.nf
.nh
.IP "" 8
% chirp_stream_files join newdata server1.somewhere.edu /part1
                                  server2.somewhere.edu /part2
                                  server2.somewhere.edu /part3
.fi
.hy
.fam
.P

.SH COPYRIGHT
.LP

The Cooperative Computing Tools are Copyright (C) 2003-2004 Douglas Thain and Copyright (C) 2005-2015 The University of Notre Dame.  This software is distributed under the GNU General Public License.  See the file COPYING for details.

.SH SEE ALSO
.LP


.IP \(bu 4
\fBCooperative Computing Tools Documentation\fP
.IP \(bu 4
\fBChirp User Manual\fP
.IP \(bu 4
\fBchirp(1)\fP  \fBchirp_status(1)\fP  \fBchirp_fuse(1)\fP  \fBchirp_get(1)\fP  \fBchirp_put(1)\fP  \fBchirp_stream_files(1)\fP  \fBchirp_distribute(1)\fP  \fBchirp_benchmark(1)\fP  \fBchirp_server(1)\fP