NAME¶
cstream —
direct data streams, with
bandwidth limiting, FIFO, audio, duplication and extended reporting
support.
SYNOPSIS¶
cstream |
[-b num]
[-B num]
[-i
filename]
[-I
string]
[-l]
[-n num]
[-o
filename]
[-O
string]
[-p
filename]
[-t num]
[-T num]
[-v num]
[-V]
[filename] |
DESCRIPTION¶
Cstream filters data streams, much like the UNIX tool
dd(1). It has a more traditional command line syntax,
support for precise bandwidth limiting and reporting and support for FIFOs.
Data limits and throughput rate calculation will work for files > 4 GB.
Cstream reads from the standard input and writes to the
standard output, if no filenames are given. It will also 'generate' or 'sink'
data if desired.
Options:
- -b
num
- Set the block size used for read/write to
num. The default is 8192 bytes.
- -B
num
- Buffer input up to num bytes before
writing. The default is the blocksize. It is an error to set this to
anything below the blocksize. Useful when writing tapes and similar that
prefer few large writes of many small.
- -c
num
- Concurrent operation. Use a separate process for output.
This is especially useful in combination with the -B option.
- 0 = use one process only
(default)
-
- 1 = read process will
buffer
-
- 2 = write process will
buffer
-
- 3 = both processes will
buffer.
- In combination with a large buffer size this will often
load your memory heavily, every time the reader transfers the buffer
it collected to the writer. If you use -c 3 and have a buffer size of
128 Megabytes 256 MB of memory will be touched at once.
- -i
num
-
- -o
num
- Set the file names to use for input or output,
respectively. If the output file name is "-", data will just be
discarded. If the input file name is "-", data will be generated
'out of the void'. If these options aren't given, stdin/stout will be
used. If you need to give -o or -i
options and want stdin/stdout, specify the empty string, like this:
cstream -i''
If TCP support has been compiled in (default), hostname:portnumber will try
to connect to the specified host at the specified port and :portnumber
will open a TCP socket on the local machine and wait for a connection to
arrive. SECURITY NOTE: cstream includes no mechanism to restrict the hosts
that may connect to this port. Unless your machine has other network
filters, anyone will be able to connect.
- -I
string
-
- -O
string
- Specify the type of input and output file, respectively.
- If string
- includes 'f', a fifo will be created.
- If string
- includes 'a', the file will be assumed to be an
opensound-compatible audio device and will be switched to CD-like
settings.
- If string
- includes 't', a copy of the stream will be sent to file
descriptor 3.
- If string
- includes 'N', TCP will not be used for that file even
if the name has a ":".
- -l
- Include line count in statistics.
- -n
num
- Limit the total amount of data to
num. If there is more input available, it will be
discarded, cstream will exit after the limit has been
reached. If there is less input, the limit will not be reached and no
error will be signaled.
num may have a trailing 'k', 'm' or 'g' which means
Kilobytes, Megabytes or Gigabytes (where Kilo = 1024). This applies to all
numeric options.
- -p
filename
- Write the process id of cstream to
filename. If cstream uses a separate writer process
(option -c), this is the pid of the parent (reader) process.
- -t
num
- Limit the throughput of the data stream to
num bytes/second. Limiting is done at the input
side, you can rely on cstream not accepting more than this rate. If the
number you give is positive, cstream accumulates errors and tries to keep
the overall rate at the specified value, for the whole session. If you
give a negative number, it is an upper limit for each read/write system
call pair. In other words: the negative number will never exceed that
limit, the positive number will exceed it to make good for previous
underutilization.
- -T
num
- Report throughput every num seconds.
- -v
num
- Set verbose level to num. By default,
it is set to 0, which means no messages are displayed as long as no errors
occur. A value of 1 means that total amount of data and throughput will be
displayed at the end of program run. A value of 2 means the transfer rate
since the end of the first read/write pair will also be reported (useful
when there is an initial delay). A value of 3 means there will also be
separate measurements for read and write. This option is
resource-consuming and currently isn't implemented. A value of 4 means
that notices about each single read/write will be displayed. High values
include all message types of lower values.
- -V
- Print version number to stdout and exit with 0.
- filename
- A single filename as the last argument without an option
switch will be used as input file if -i has not been used.
- SIGUSR1
-
- SIGINFO
- Sending SIGUSR1 (or SIGINFO, which is usually mapped to
Control-T on you keyboard) to cstream causes it to display throughput
rates to stderr. The stream will continue as if nothing happened.
- SIGUSR2
- Exit and report throughput rates, if requested.
- SIGHUP
- I found myself sending SIGHUP accidentally too often. But
ignoring or misusing SIGHUP is not an option for me. Thus, when
cstream received SIGHUP, it will wait 5 seconds for
another SIGHUP, to give users a chance to correct a possible mistake. If
no additional SIGHUP is received, cstream kills itself
with SIGHUP.
EXAMPLES¶
- cstream -o
tmpfile -v 1 -n 384m -i -
- Writes 384 Megabytes of unspecified data to file
tmpfile and display verbose throughput rate. Makes a
good benchmark, the speed of /dev/null varies too much
from system to system.
- cstream -i
tmpfile -v 1 -n 384m -o -
- Read the same file back in and discard data.
- cstream -b 2000
-t 10000 /var/log/messages
- Will display the file in a more or less watchable
speed.
- dump 0sf 400000
- / | cstream -v 1 -b 32768 -o /dev/rst0 -p pidfile
-
- kill -USR1 `cat
pidfile`
- Write the output from dump(1) to tape. Each time the signal
is sent, the throughput and data rate so far will be displayed.
- cstream -t
176400 -i /dev/dsp0 -I f -o -
- Makes kind of a soundcard emulator which may be used to
test audio applications that need something to write to that limits the
data rate as a real soundcard does. This obviously doesn't work when the
application tries to write data using mmap(2) and the application has to
ignore errors when it tries to set soundcard parameters using
ioctl(2).
- cstream -t
176400 -i /dev/dsp0 -I f -o /dev/dsp1 -O f
- Similar soundcard emulator, except that it allows you to
grab the data your applications sends to it from the other fifo, while
still having precise timing.
- cstream -Oa -o
/dev/dsp0 myhost.mydomain.com:17324
- Connects port 3333 on host myhost.mydomain.com and whatever
data it finds there will be sent to the soundcard, with appropriate
settings for CD quality stero play.
- cstream -i
myaudiofile.raw -o :17324
- This will open a TCP server on port 17324 and waits until
someone connects (for example, the command line from the previous
example). Then it will send the contents of myaudiofile.raw down the TCP
stream (for the previous audio example, typically a CD audiotrack like you
get from the tosha or cdparanoia utilities).
- cstream -OD -o
myfile
-
Write to file myfile with O_DIRECT. That usually means that the filesystem
buffer cache will not try to cache this file. You can use that to prevent
copying operations from eating up physical memory. Note that when cstream
encounters a write error it will switch the output file from O_DIRECT to a
normal file and write all further blocks without O_DIRECT if writes
without O_DIRECT succeed. In practice that usually means that your last
block, if not a multiple of the filesystem block size, will still be
written into the file (the maximum amount of data written without O_DIRECT
is your blocksize minus one). That way cstream ensures that the output
file has the length of the input, however odd the length was and no matter
what restrictions your OS places on O_DIRECT output. Again, cstream will
*not* pad the output to the block size, you get the same file and file
size as if not using O_DIRECT, at the cost of switching to non-O_DIRECT
whenever a block is not the right size.
- cstream -i
:3333 | dd obs=8192 | ./cstream -omyfile
-v7 -OD
- This is what you need to do to buffer TCP input, so that
the last cstream will not switch away from O_DIRECT prematurely because of
short reads. If your input can do short reads (e.g. from TCP), and you
want to ensure that O_DIRECT stays in effect, you need a buffer between
the TCP stream and the O_DIRECT stream. Since cstream does not yet support
different input and output block sizes, dd is suitable here. Note that
this is only necessary if the OS requires multiples of the filesystem
block size for O_DIRECT. At the time of this writing this construct is
needed on Linux for using TCP streams with O_DIRECT, but it is not needed
on FreeBSD.
- cstream -OS -o
myfile
- Writes to file myfile with O_SYNC. This means by the time
the system call returns the data is known to be on disk. This is not the
same thing as O_DIRECT. O_DIRECT can do its own buffering, with O_SYNC
there is no buffering at all. At the time of this writing, O_SYNC on both
Linux and FreeBSD is very slow (1/5th to 1/10th of normal write) and
O_DIRECT is reasonably fast (1/4th to 1/2 of normal write). You can
combined O_SYNC and O_DIRECT.
ERRORS¶
- Exit code 0 means
success.
-
- Exit code 1 means a command
line syntax usage error.
-
- Exit code 2 means other
errors, especially system errors.
-
Bugs¶
There should be an option to begin writing directly after the first read ended
and then fill the buffer with reads in the background. Right now writing will
not begin before the reader has filled the buffer completely for the first
time.
Not a bug: the code to do O_DIRECT is reasonably sophisticated. It will fall
back to normal I/O on errors. But before doing that it knows about both
filesystem blocksize requirements (will default I/O blocksize to whatever the
filesystem of the output file is in) and page alignment requirements (I/O will
happen from a page-aligned buffer). However, the combination of concurrent
read/writes (-c options) and O_DIRECT has not been tested beyond basic
verification that it gets some tests right.
SEE ALSO¶
dd(1),
mkfifo(2)
HISTORY¶
cstream was initially written by Martin Cracauer in 1998. For
updates and more information see
http://www.cons.org/cracauer/cstream.html