.\" Copyright (c) 2000-2016 QoSient, LLC .\" All rights reserved. .\" .\" This program is free software; you can redistribute it and/or modify .\" it under the terms of the GNU General Public License as published by .\" the Free Software Foundation; either version 2, or (at your option) .\" any later version. .\" .\" This program is distributed in the hope that it will be useful, .\" but WITHOUT ANY WARRANTY; without even the implied warranty of .\" MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the .\" GNU General Public License for more details. .\" .\" You should have received a copy of the GNU General Public License .\" along with this program; if not, write to the Free Software .\" Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA. .\" .TH RASTREAM 1 "12 August 2003" "rastream 3.0.8" .SH NAME \fBrastream\fP \- stream block processor for \fBargus(8)\fP data. .SH SYNOPSIS .B rastream [[\fB\-M\fP \fIsplitmode\fP] [\fIsplitmode options\fP]] [\fB\-f\fP \fIfile processing program\fP -B secs] [\fBraoptions\fP] [\fB--\fP \fIfilter-expression\fP] .SH DESCRIPTION .IX "rastream command" "" "\fBrastream\fP \(em argus data" .LP \fBRastream\fP reads .BR argus data from an \fIargus-data\fP source, and splits the resulting output into consecutive sections of records based on size, count time, or flow event, writing the output into a set of output-files. \fBRastream\fP provides the option to run a program against the output files, \fBseconds\fP after the file is understood to be finished. The program must be specified in a manner so that \fBrastream\fP can find it, either within the system $PATH, or provided as a full pathname. By default, \fBrastream\fP splits the stream by output file record count, putting 10,000 records of input into each \fBargus\fP output file, or standard out, as needed. The behavior is similar to the unix split.1 command. The output files' name consists of a prefix, which is specified using the \fI-w\fP \fIra option\fP, and a suffix, which is created for each resulting file. If no prefix is provided, then \fBrastream\fP will use 'x' as the default prefix. The suffix that is used is determined by the mode of operation. When \fBrastream\fP is using the default count mode or the size mode, the suffix is a group of letters 'aa', \'ab\', and so on, such that concatenating the output files in sorted order by file name produces the original input file. If \fBrastream\fP will need to create more output files than are allowed by the default suffix strategy, more letters will be added, in order to accomodate the needed files. When the mode is \fBtime\fP mode, the default output filename suffix is '%Y.%m.%d.%h.%m.%s', which is used by strftime() to create an output filename that is time oriented. This default is overrided by adding a '%' extension to the name provided on the commandline using the \fI-w\fP option. When standard out is specified, using \fI-w -\fP, \fBrastream\fP will output a single \fBargus-stream\fP with START and STOP argus management records inserted appropriately to indicate where the output is split. See \fBargus(8)\fP for more information on output stream formats. When \fBrastream\fP is spliting on output record count (the default), the number of records is specified as an ordinal counter, the default is 10,000 records. When \fBrastream\fP is spliting based on the maximum output file size, the size is specified as bytes. The scale of the bytes can be specified by appending 'b', 'k' and 'm' to the number provided. When \fBrastream\fP is spliting based on time, the time period is specified with the option, and can be any period based in seconds (s), minutes (m), hours (h), days (d), weeks (w), months (M) or years (y). \fBRastream\fP will create and modify records as required to split on prescribed time boundaries. If any record spans a time boundary, the record is split and the metrics are adjusted using a uniform distribution model to distribute the statistics between the two records. Care is taken to avoid records with zero packet and byte counts, that could result from roundoff error. When \fBrastream\fP is spliting based on flow event, the flow that acts as the event marker is specified using a standard \fBra\fP filter expression, that is bounded by quotes ("). Records that preceed the first flow event in the data stream are written to the specified output file, and then new files are generated with the flow event record being the first record of the new file. This method will allow you to use wire events as triggers for spliting data. .SH RASTREAM SPECIFIC OPTIONS Rastream, like all ra based clients, supports a number of \fBra options\fP including remote data access, reading from multiple files and filtering of input argus records through a terminating filter expression. \fBrastream(1)\fP specific options are: .TP 4 4 .BI \-a "\| suffix length\^" Starting append suffix length. The default is 2 characters. .TP 4 4 .BI \-B "\| duration\^" Buffer hold time before processing. This value is usually in the 5-15 second range and provides time for rastream to sort records and schedule outputfile processing. The number is derived from the larges FAR status interval of all the argus data sources encountered. .TP 4 4 .BI \-f "\| program\^" Post processing program. \fBrastream\fP, will execute this program just after closing the output file, passing the full path to the closed output file as a parameter, using this convention: .nf program -r /full/path/to/closed/file .fi This allows you to post-process the output file in an automated fashion. Generally, this program can do anything you like, such as aggregating and correcting flow records, labeling records for semantic enhancement, indexing the files, using programs like rasqltimeindex(), and compressing the files. Traditionally, the program has been a shell-script, perl program, or php script, so that it can be easily modified, on the fly, but it can be any executable that can handle the "-r filename" parameter convention. The program should provides its own accountability and error logging, so that you know that things are working as you expect. \fBrastream\fP must have a path to the program, the program must be executable, and \fBrastream\fP must have permission to run the program for this strategy to be successful. An example rastream.sh is provided in the ./support/Config directory. .TP 4 4 .BI \-M "\| splitmode\^" Supported spliting modes are: .nf \fB count \fP \fB size \fP \fB time \fP \fB flow "filter-expression"\fP .fi .TP 4 4 .BI \-w "\| filename\^" \fBRastream\fP supports an extended \fI-w\fP option that allows for output record contents to be inserted into the output filename. Specified using '$' (dollar) notation, any printable field can be used. Care should be taken to honor any shell escape requirements when specifying on the command line. See \fBra(1)\fP for the list of printable fields. Another extended feature, when using \fBtime\fP mode, \fBrastream\fP will process the supplied filename using \fBstrftime(3)\fP, so that time fields can be inserted into the resulting output filename. .SH INVOCATION This invocation reads \fBargus(8)\fP data from \fBinputfile\fP and splits the \fBargus(8)\fP data stream based on output file size of no greater than 1 Megabyte. The resulting output files have a prefix of \fIargus.\fP and suffix that starts with 'aa'. The single trailing '.' is significant. .nf \fBrastream\fP -r inputfile -M size 1m -w argus. .fi This invocation splits \fBinputfile\fP based on hard 10 minute time boundaries. The resulting output files are created with a prefix of \fI/archive/%Y/%m/%d/argus.\fP and the suffix is \fI%H.%M.%S\fP. The values will be supplied based on the time in the record being written out. .nf \fBrastream\fP -r * -M time 10m -w "/archive/%Y/%m/%d/argus.%H.%M.%S" .fi This invocation splits \fBinputfile\fP based on the argus source identifier. The resulting output files are created with a prefix of \fI/archive/Source Identifier/argus.\fP and the default suffix starting with "aa". The source identifier will be supplied based on the contents of the record being exported. .nf \fBrastream\fP -r * -M time 10m -w "/archive/$srcid/argus." .fi This invocation splits \fBinputfile\fP based on a flow event marker. The resulting output files are created with a prefix of 'outfile.' and the default suffix starting with "aa". Whenever a ping to a specific host is seen in the stream, a new output file is generated. .nf \fBrastream\fP -r * -M flow "echo and host 1.2.3.4" -w outfile. .fi .SH COPYRIGHT Copyright (c) 2000-2016 QoSient. All rights reserved. .SH SEE ALSO .BR ra(1), .BR rarc(5), .BR argus(8), .SH AUTHORS .nf Carter Bullard (carter@qosient.com). .fi