\" Copyright (c) 2023, Peter Haag .\" All rights reserved. .\" .\" Redistribution and use in source and binary forms, with or without .\" modification, are permitted provided that the following conditions are met: .\" .\" * Redistributions of source code must retain the above copyright notice, .\" this list of conditions and the following disclaimer. .\" * Redistributions in binary form must reproduce the above copyright notice, .\" this list of conditions and the following disclaimer in the documentation .\" and/or other materials provided with the distribution. .\" * Neither the name of the author nor the names of its contributors may be .\" used to endorse or promote products derived from this software without .\" specific prior written permission. .\" .\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" .\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE .\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE .\" ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE .\" LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR .\" CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF .\" SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS .\" INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN .\" CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) .\" ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE .\" POSSIBILITY OF SUCH DAMAGE. .\" .Dd $Mdocdate$ .Dt NFDUMP 1 .Os .Sh NAME .Nm nfdump .Nd flow display and analysis program .Sh SYNOPSIS .Nm .Fl r Ar flowpath .Op Fl w Ar outfile .Op Fl f Ar filterfile .Op Fl C Ar config .Op Fl R Ar filelist .Op Fl M Ar dirlist .Op Fl O Ar order .Op Fl t Ar timewin .Op Fl c Ar num .Op Fl a .Op Fl A Ar aggregation .Op Fl b .Op Fl B .Op Fl I .Op Fl D Ar nameserver .Op Fl G Ar geoDB .Op Fl s Ar statistic .Op Fl n Ar num .Op Fl o Ar format .Op Fl 6 .Op Fl q .Op Fl N .Op Fl i Ar ident .Op Fl v Ar flowfile .Op Fl E Ar flowfile .Op Fl x Ar flowfile .Op Fl z= .Op Fl J Ar compress .Op Fl X .Op Fl Z .Op Fl T .Op Fl V .Op Fl h .Op filter .Sh DESCRIPTION .Nm reads the flow data from one or more binary files, created by any nfdump collector .Ar nfcapd, nfpcapd and .Ar sfcapd . It processes and lists the flows in many different output formats and can create a wide range of statistics. .Pp .Nm has a very powerful flow filter to process flows. The filter syntax is very similar to tcpdump, but adapted and extended for flow filtering. A flow filter may also contain arrays of many thousand IP addresses etc. to search for specific records. .Pp .Nm can aggreagte flows according to a user defined number of elements. This masks certain elements and allows to sum up flow records matching the same values. .Pp The combination of flow filtering and aggregation as input for any flow statistics allows complex flow processing. Pre-filtered and aggregated flow data may also be written back into a binary flow file, which again may be processed with .Nm .Pp .Nm can enrich the listing of flows with geo location information and AS information, unless AS information is already available in the flow records. IP addresses can be tagged with a two letter country code, or with a longer location label containing the geographic region, country and city. The geo location and AS information is retrieved from the optional .Ar geoDB database, created by the .Ar geolookup program from the nfdump tools. .Ar geolookup uses the .Ar Maxmind database .Ar GeoDB or .Ar GeoLite2 to create a binary lookup database for .Nm Please check the .Ar geolooup(1) man page for more details. .Pp The options are as follows: .Bl -tag -width Ds .It Fl r Ar flowpath Reads flow records from this path. .Ar flowpath may be a single file, or a directory containing any number of flow files or sub directories. All files are processed in the order, as listed by the OS. .It Fl w Ar outfile Writes all processed records into .Ar outfile instead of printing. The .Ar flowfile is a binary flow file and may be processed again with .Nm This can be useful to limit flows according to a flow filter and/or specific flow aggregation. .It Fl f Ar filterfile Reads the flow filter from .Ar filterfile. This can be useful for very long or structured filters, with comments and long lists. .Ar Note: Any filter specified directly on the command line takes precedence over the .Ar filterfile. .It Fl C Ar config Read more options from file .Ar config. .Nm tries to read by default .Ar %prefix/etc/nfdump.config. This may be overwritten by the environment valiable .Ar NFCONF which again may be overwritten by this option .Fl C. In order to prevent reading any config file, even if it would exist set .Fl C .Sy none. A config file is not required, but may be handy for often used output formats etc. .It Fl O Ar order Sets an output order for records to be printed as text output. This order applies after all records processing, such as filtering, and aggregation and before printing. .Bl -tag -width "duration" -compact .It Cm flows Sort according to the number of flows .It Cm packets Sort according to (in)packets .It Cm ipkg Same as packets .It Cm opkg Sort according to output packets .It Cm bytes Sort according to (in)bytes .It Cm ibyte Same as bytes .It Cm obyte Sort according to output bytes .It Cm pps Sort according to (in)packets per second .It Cm ipps Same as ipps .It Cm opps Sort according to out packets per second .It Cm bps Sort according to (in)bytes per second .It Cm ibps Same as bps .It Cm obps Sort according to output bytes per second .It Cm bpp Sort according to (in)bytes per packet .It Cm ibpp Same as bpp .It Cm obpp Sort according to output packets .It Cm tstart Sort according to start time of flow - former -m .It Cm tend Sort according to end time of flows .It Cm duration Sort according to duration of flows .El .It Fl t Ar timewin Set time window to process flows. This option is considered legacy andmay be replaced with a .Ar filter primitive in future rleases. The time window is specified as: YYYY/MM/dd.hh:mm:ss[-YYYY/MM/dd.hh:mm:ss]. Any parts of the time spec may be omitted e.g YYYY/MM/dd expands to YYYY/MM/dd.00:00:00-infinity and processes all flow from a given day onwards. The time window may also be specified as +/- n. In this case it is relative to the beginning or end of all flows. +10 means the first 10 seconds of all flows, -10 means the last 10 seconds of all flows. .It Fl c Ar num Limit the number of records to be processed to the first .Ar num records, which passwd the .Ar filter. .It Fl a Aggregate flow records. The default aggregation is done at connection level by taking the 5-tuple .Ar protocol, srcip, dstip, srcport and .Ar dstport. This way of aggregation may be overwritten by option .Fl A .It Fl A Ar aggregation Sets the list of elements in a flow record to be aggregated. .Ar aggregation is a ',' separated list of any number of v9/ipfix elements. The following elements are accepted: .Bl -tag -width "srcip4/net" -compact .It Cm proto IP protocol .It Cm srcip Source IP address .It Cm dstip Destination IP address .It Cm srcip4/net IPv4 source IP address with applied netmask .It Cm srcip6/net IPv6 source IP address with applied netmask .It Cm dstip4/net IPv4 destination IP address with applied netmask .It Cm dstip6/net IPv6 destination IP address with applied netmask .It Cm srcnet Apply netmask srcmask in netflow record for source IP .It Cm dstnet Apply netmask dstmask in netflow record for dest IP .It Cm srcport Source port .It Cm dstport Destination port .It Cm srcmask Source mask .It Cm dstmask Destination mask .It Cm srcvlan Source vlan label .It Cm dstvlan Destination vlan label .It Cm srcas Source AS number .It Cm dstas Destination AS number .It Cm nextas BGP Next AS .It Cm prevas BGP Previous AS .It Cm inif SNMP input interface number .It Cm outif SNMP output interface number .It Cm next IP next hop .It Cm bgpnext BGP next hop .It Cm insrcmac In source MAC address .It Cm outdstmac out destination MAC address .It Cm indstmac In destination MAC address .It Cm outsrcmac Out source MAC address .It Cm tos Source type of service .It Cm srctos Source type of Service .It Cm dsttos Destination type of Service .It Cm mpls1 MPLS label 1 .It Cm mpls2 MPLS label 2 .It Cm mpls3 MPLS label 3 .It Cm mpls4 MPLS label 4 .It Cm mpls5 MPLS label 5 .It Cm mpls6 MPLS label 6 .It Cm mpls7 MPLS label 7 .It Cm mpls8 MPLS label 8 .It Cm mpls9 MPLS label 9 .It Cm mpls10 MPLS label 10 .It Cm router IP address of exporting router .It Cm odid observation domain ID .It Cm opid observation point ID .It Cm xsrcip X-late source IP address, if compiled with NSEL support .It Cm xdstip X-late destination IP address, if compiled with NSEL support .It Cm xsrcport X-late source port, if compiled with NSEL support .It Cm xdstport X-late destination port, if compiled with NSEL support .El .Pp .Nm automatically compiles the appropriate output format for the selected aggregation elements unless an explicit output format .Fl o is given. The automatic output format is identical to .Pp .Dl -o 'fmt:%ts %td %pkt %byt %bps %bpp %fl' .Pp where represents the selected aggregation tags. .It Fl b Aggregate flow records as bidirectional flows. This automatically implies -a. Aggregation is done on connection level by taking the 5-tuple .Ar protocol, srcip, dstip, srcport and .Ar dstport The reverse order applies for the corresponding reverse flow. Input and output packets/bytes are counted and reported separately. Both flows are merged into a single record with corresponding input and output counters. An appropriate output format is selected automatically, which may be overwritten by any .Fl o Ar format option. .It Fl B Similar to option .Fl b but tries to guess the correct client to server direction. Automagically swaps flows if src port is < dst port for TCP and UDP flows and src port < 1024 and dst port > 1024. Some exporters do not really care sending the flows in proper order. It's considered to be a conveniency option. .It Fl I Print flow statistics of a single file or the summary of all the files specified by .Fl r Ar flowpath. .It Fl g Print for each flow file given by .Fl r Ar flowpath a one line summary, which can be easily used by gnu plot. .It Fl D Ar nameserver Sets the .Ar nameserver to translate hostnames into IP addresses in filter expressions. See .Ar filter below for more details. .It Fl G Ar geoDB Use .Ar geoDB as geo lookup DB for geo location and AS lookups. .Nm tries to read the environment variable .Ar NFGEODB for the path of .Ar geoDB. The option .Fl G overwrites .Ar NFGEODB. In order to prevent reading any .Ar geoDB file, even if it would exist set .Fl G .Sy none. .It Fl s Ar statistic Op Ar :p Op Ar /orderby Generate the Top N flow record or flow element statistic. By optionally adding .Sy :p to .Ar statistic, the statistic is additionally split up into the transport layer protocols. By default the statistic is transport protocol independent. Each .Ar statistic may be ordered by the optional parameter .Ar orderby This can be .Sy flows, packets, bytes, pps, bps or .Sy bpp. You may specify more than one .Ar orderby option, which results in the same statistic but ordered differently. If no orderby is given, the statistic is ordered by flows. You can specify as many -s flow element statistics as needed on the command line for the same run. .Pp .Ar statistic can be: .Pp .Bl -tag -width "outsrcmac" -compact .It Cm record aggregated netflow records. .It Cm srcip source IP addresses .It Cm dstip destination IP addresses .It Cm ip any (src or dst) IP addresses .It Cm nhip next hop IP addresses .It Cm nhbip BGP next hop IP addresses .It Cm router exporting router IP address .It Cm srcport source ports .It Cm dstport destination ports .It Cm port any (source or destination) ports .It Cm tos type of service - default src .It Cm srctos src type of service .It Cm dsttos dst type of service .It Cm dir flow directions ingress/egress .It Cm srcas source AS numbers .It Cm dstas destination AS numbers .It Cm srcgeo 2 letter geo source country code .It Cm dstgeo 2 letter geo destination country code .It Cm as any (source or destination) AS numbers .It Cm inif input interface .It Cm outif output interface .It Cm if any interface .It Cm inam input interface name .It Cm onam output interface name .It Cm srcmask src mask .It Cm dstmask dst mask .It Cm srcvlan src vlan label .It Cm dstvlan dst vlan label .It Cm vlan any vlan label .It Cm insrcmac input src MAC address .It Cm outdstmac output dst MAC address .It Cm indstmac input dst MAC address .It Cm outsrcmac output src MAC address .It Cm srcmac any src MAC address .It Cm dstmac any dst MAC address .It Cm inmac any input MAC address .It Cm outmac any output MAC address .It Cm mask any mask .It Cm proto IP protocols .It Cm mpls1 MPLS label 1 .It Cm mpls2 MPLS label 2 .It Cm mpls3 MPLS label 3 .It Cm mpls4 MPLS label 4 .It Cm mpls5 MPLS label 5 .It Cm mpls6 MPLS label 6 .It Cm mpls7 MPLS label 7 .It Cm mpls8 MPLS label 8 .It Cm mpls9 MPLS label 9 .It Cm mpls10 MPLS label 10 .It Cm sysid Internal SysID of exporter .It Cm nbar nbar ID .It Cm ja3 ja3 hashes .It Cm odid observation domain ID .It Cm opid observation point ID .It Cm vrf/ivrf ingress vrf .It Cm evrf egress vrf .It Cm ivrfnam ingress vrf name .It Cm evrfnam egress vrf name .It .Pp NSEL/ASA statistics .It Cm event NSEL/ASA event .It Cm xevent NSEL/ASA extended event .It Cm xsrcip NSEL/ASA translated src IP address .It Cm xsrcport NSEL/ASA translated src port .It Cm xdstip NSEL/ASA translated dst IP address .It Cm xdstport NSEL/ASA translated dst port .It Cm iacl NSEL/ASA ingress ACL .It Cm iace NSEL/ASA ingress ACE .It Cm ixace NSEL/ASA ingress xACE .It Cm eacl NSEL/ASA egress ACL .It Cm eace NSEL/ASA egress ACE .It Cm exace NSEL/ASA egress xACE .It .Pp NAT statistics .It Cm nevent NAT event .It Cm nsrcip NAT src IP address .It Cm nsrcport NAT src port .It Cm ndstip NAT dst IP address .It Cm ndstport NAT dst port .El .Pp Example: .Pp .Dl % nfdump -s srcip -s ip/flows/bytes -s record/bytes .Pp .It Fl n Ar num Set the number of records to be printed to .Ar num. This option applies to .Fl s statistics as well as to ordered output .Fl O or aggregated records .Fl a The default is set to 10 for statistics and unlimited for the other use cases. To disable the limit, set .Ar num to 0. .It Fl o Ar format Sets the output format to print flow records. .Nm has many different output formats already predefined. .Ar format may be one of the options below: .Pp .Bl -tag -width "extended " -compact .It Cm raw Print the full flow record on multiple lines. This prints all available information. .It Cm fmt: Ar user Print the flow records according the format .Ar user. This is a very flexible and powerful way to format flow records. See the section .Sy OUTPUT below for more details on how to compile your own format. .It Cm json Print full record as a separate json object. .It Cm csv Legacy .csv format - will get removed in future releases. Please use .Sy json instead. .It Cm pipe Legacy '|' separated format - will get removed in future releases. Please use .Sy json instead. .El .Pp Already predefined fmt formats: .Pp .Bl -tag -width "extended" -compact .It Cm line Print each flow on one line. Default format. .It Cm long Print each flow on one line with more details .It Cm biline Same as .Ar line, but for bi-directional flows .It Cm bilong Same as .Ar long, but for bi-directional flows .It Cm gline Same as .Ar line, but add country code to IPs. If a geoDB file is supplied this is the default output format .It Cm glong Same as .Ar long, but add country code to IPs .It Cm extended Print each flow on one line with even more details. .It Cm nsel Print format for NSEL event records. Default format if NSEL/NAT support has been compiled in. .It Cm nel Print format for NAT event records. .El .Pp The .Nm config file may contain additional formats. If you want to add new formats or change existing ones, check the config file. .Pp IPv6 addresses are printed condensed in any .Sy fmt defined format to prevent cluttering the output with large blank blocks. A condensed IPV6 uses max 16 characters. If it is longer, then the middle part of the IP is cut out and replaced be "..". For previewing an output, this fits most needs. For a listing with the full IPV6 addresses add option .Fl 6. .It Fl 6 Print full length of IPv6 addresses in output instead of condensed. .It Fl q Quiet mode. Suppress the header line and the statistics at the bottom of text outputs. .It Fl N Print plain numbers in output without scaling. Easier for output parsing with 3rd party tools. .It Fl i Ar ident Change the ident label in the file, specified by .Fl r to .Ar ident .It Fl v Ar flowfile Verify the consistency of .Ar flowfile and print the file parameters and number of records. .It Fl E Ar flowfile Print the exporter and sampler list if found in .Ar flowfile. Additional statistics per exporter are printed with number of flows, packets and sequence errors. .It Fl x Ar flowfile This options works on nfdump version 1.6.x files only and may get removed in future. Scans and prints extension maps located in .Ar flowfile .It Fl z=lzo Compress flow files with LZO1X-1 compression. Fastest compression. .It Fl z=lz4 Compress flow files with LZ4 compression. Fast and efficient. .It Fl z=bz2 Compress flow files with bz2 compression. Slow but most efficient. May be used for archiving files or if you are really short of spce. .It Fl J Ar compress Change compression for any number of files given by option .Fl r Ar flowpath Set .Ar compress to 0 for no compression or to any of: 1 or LZO, 2 or BZ2, 3 or LZ4. This option may be used for archiving flow files and changing the compression to use less disk space. .It Fl X Compiles the .Ar filter syntax and dumps the filter engine table to stdout. This is for debugging purpose only. .It Fl Z Check .Ar filter syntax and exit. Sets the return value accordingly. .It Fl R Ar filelist Select a range of files. This option is mainly used by old NfSen and documented here as legacy option. .Bl -item -compact .It /any/dir Read recursively all files in directory dir. .It /dir/file Read all files beginning with file. .It /dir/file1:file2 Read all files from file1 to file2. .El When using in combination with a sub hierarchy: /dir/sub1/sub2/file1:sub3/sub4/file2 Read all files from sub1/sub2/file1 sub3/sub4/file2 iterating over all required hierarchy levels. Note: files are read in alphabetical order. .It Fl M Ar dirlist Read the same file hierarchy from multiple directories. This option is mainly used by old NfSen and documented here as legacy option. Example: /any/path/to/dir1:dir2:dir3 etc. and will be expanded to the directories: /any/path/to/dir1, /any/path/to/dir2 and /any/path/to/dir3. Any number of colon separated directories may be given. The files to read are specified by -r or -R and are expected to exist in all the given directories. The options -r and -R must not contain any directories when used in combination with -M. .It Fl T Tag IP addresses with a prepending cntrl-A character, to allow output parsers to hook in. This option is mainly used by old NfSen and documented here as legacy option. .It Fl V Print .Nm version and exit. .It Fl h Print help text on stdout with all options and exit. .El .Pp .Ar filter selects, which records will be further processed. If no filter is given, all records will be processed. Otherwise, only those flows matching the filter will be processed. Any IP address in a filter may be specified as IPv4 or IPv6. .Pp The filter syntax is similar to tcpdump but adapted and extended for flow records. The filter can be either specified on the command line after all options or in a separate file. It can span several lines. Anything after a '#' is treated as a comment and ignored to the end of the line. There is virtually no limit in the length of the filter expression. All keywords are case insensitive. .Pp A single filter primitive filters a single element of a flow record. A filter consists of one or more primitives, which are linked together: .Pp .Dl Ar expr Sy and Ar expr .Dl Ar expr Sy or Ar expr .Dl Sy not Ar expr Sy and Ar (expr) .Pp Possible filter primitives: .Bl -tag -width "## spacer ##" -compact .It Cm @include Ar file Expands the content of .Ar file into the current filter .Pp .It Cm count Ar comp number True if the comparison with the record counter matches .Ar number Each record gets assigned a record number at the time it is read from file. Therefore this record number is not unique and may change, depending on the order files are read. .It Cm ident Ar string True if the record ident field matches .Ar string. This filter can be used to filter out different sources. .Pp .It Cm inet .It Cm ipv4 True if source and destination IP of a record are IPv4 IPs. .Pp .It Cm inet6 .It Cm ipv6 True if source and destination IP of a record are IPv6 IPs. .Pp .It Cm proto Ar protocol True if the record protocol field matches .Ar protocol. protocol can be a symbolic name such as .Cm tcp , .Cm udp , .Cm icmp , .Cm ah , .Cm esp , .Cm ipip , and many more or a protocol number, such as 6, 17 for protocol .Sy tcp and .Sy udp . .Pp .It Cm tun proto Ar protocol True if the record tunnel protocol field matches .Ar protocol. protocol may be a symbolic name or protocol number. .Pp .It Cm ip Ar ipaddr .It Cm src ip Ar ipaddr .It Cm dst ip Ar ipaddr True if the respective IP field of the record matches .Ar ipaddr . ipaddr may be an IPv4 or IPv6 address or a symbolic hostname. In this case a DNS lookup resolves the hostname to one or more IP addresses. If more than one IP results, all IPs are chained together in an .Cm or chain. (IP or IP or IP). If .Cm ip is not specified with .Cm src or .Cm dst the source or destination IP may match. .It Cm host Ar ipaddr .Cm host is just a synonym for .Cm ip (See above) .Pp .It Cm ip in Ar [ iplist ] .It Cm src in ip Ar [ iplist ] .It Cm dst ip Ar [ iplist ] True if the respective IP field of the record is in .Ar iplist. iplist is a space or ',' separated list of IP addresses or networks in CIDR notation. This is the preferred way to search in large list of IP addresses and networks and is much more efficient than to chain all IP addresses together. (IP1 or IP2 or IP3). The .Ar iplist may contain several hundreds to thousand IPs and/or networks. For just a few IPs use an .Cm or chain, otherwise use an .Ar iplist If .Cm ip is not specified with .Cm src or .Cm dst the source or destination IP may match. .Pp .It Cm net Ar network netmask .It Cm src net Ar network netmask .It Cm dst net Ar network netmask .It Cm net Ar network/netbits .It Cm src net Ar network/netbits .It Cm dst net Ar network/netbits True if the respective IP field of the record matches the .Ar network if the corresponding .Ar netmask or .Ar netbits are applied to the IP address. If .Cm net is not specified with .Cm src or .Cm dst the source or destination IP may match. .Pp .It Cm geo Ar geoloc .It Cm src geo Ar geocode .It Cm dst geo Ar geocode True, if the 2-letter country code resolved by geolookup of the source or destination IP address matches .Ar geocode. This filter works only, if a valid geoDB is specified. See geo location option above. The 2-letter country code corresponds to the maxmin DB definitions. if .Cm geo is not specified with .Cm src or .Cm dst the source or destination geo location code may match. .Pp .It Cm tunip Ar ipaddr .It Cm src tunip Ar ipaddr .It Cm dst tunip Ar ipaddr True if the respective tunnel IP field of the record matches .Ar ipaddr . If .Cm tunip is not specified with .Cm src or .Cm dst the source or destination tunnel IP may match. .Pp .It Cm port Ar comp num .It Cm src port Ar comp num .It Cm dst port Ar comp num True if the comparison of the respective port field matches .Ar num See .Ar comp for the comparator details. If .Cm port is not specified with .Cm src or .Cm dst the source or destination port may match. .Pp .It Cm port in Ar [ portlist ] .It Cm src port in Ar [ portlist ] .It Cm dst port in Ar [ portlist ] True if the respective port field of the record is in .Ar portlist. portlist is a space or ',' separated list of port numbers. This is the preferred way to search in large list of port numbers and is much more efficient than to chain all ports together. (PORT1 or PORT2 or PORT3). .Ar portlist may contain several hundreds to thousand of port numbers. If .Cm port is not specified with .Cm src or .Cm dst the source or destination port may match. .Pp .It Cm icmp-type Ar num .It Cm icmp-code Ar num True if the respective icmp field of the record matches .Ar num. This automatically implies .Cm proto icmp. .Pp .It Cm engine-type Ar num .It Cm engine-id Ar num .It Cm sysid Ar num True if the respective fields of the record matches .Ar num engine type and ID are set by the exporting device, sysid refers to the .Nm collector internal assigned number. See also option .Fl E above. .Pp .It Cm if Ar num .It Cm in if Ar num .It Cm out if Ar num True if the respective interface fields of the record matches .Ar num. This ID may correspond to the SNMP ID of the interface but depends on the exporter. If .Cm if is not specified with .Cm in or .Cm out the input or output interface may match. .Pp .It Cm as Ar comp num .It Cm src as Ar comp num .It Cm dst as Ar comp num .It Cm prev as Ar comp num .It Cm next as Ar comp num True if the comparison of the respective AS fields matches .As num .Nm supports 32-bit AS numbers every where. Without .CM src, dst, prev or .CM next the source or destination AS may match. See .Ar comp for the comparator details. .Pp .It Cm as in Ar [ aslist ] .It Cm src as in Ar [ aslist ] .It Cm dst as in Ar [ aslist ] .It Cm prev as in Ar [ aslist ] .It Cm next as in Ar [ aslist ] True if the respective AS field of the record is in .Ar aslist. aslist is a space or ',' separated list of AS numbers. This is the preferred way to search in large list of AS numbers and is much more efficient than to chain all ports together. .Ar aslist may contain several hundreds to thousand of AS numbers. If .Cm as is not specified with .Cm src, dst, prev or .Cm next the source or destination AS may match. .Pp .It Cm mask Ar bits .It Cm src mask Ar bits .It Cm dst mask Ar bits True if the respective mask bit field of the record matches .Ar bits If .Cm mask is not specified with .Cm src or .Cm dst the source or destination mask bits may match. .Pp .It Cm vlan Ar num .It Cm src vlan Ar num .It Cm dst vlan Ar num True if the respective vlan field of the record matches .Ar num If .Cm vlan is not specified with .Cm src or .Cm dst the source or destination vlan may match. .Pp .It Cm flags Ar tcpflags True if the respective tcp flags field of the record matches any of the given .Ar tcpflags. tcpflags is a string combination of all flags to be tested: .Bl -tag -width "## " -offset indent -compact .It Cm A ACK. .It Cm S SYN. .It Cm F FIN. .It Cm R Reset. .It Cm P Push. .It Cm U Urgent. .It Cm X All flags on. .El The order of the flags within .Ar tcpflags is not relevant. Flags not mentioned are treated as don't care. In order to get those flows with only the SYN flag set, use the syntax .Pp .Dl flags S and not flags AFRPU .Pp .It Cm router ip Ar ipaddr True if the ip address of the sending router matches .Ar ipaddr as valid IPv4/IPv6 address. .Pp .It Cm next ip Ar ipaddr True if the field next-ip of the record matches .Ar ipaddr as valid IPv4/IPv6 address. .Pp .It Cm bgpnext ip Ar ipaddr True if the field bgpnext-ip of the record matches .Ar ipaddr as valid IPv4/IPv6 address. .Pp .It Cm mac Ar macaddr .It Cm in mac Ar macaddr .It Cm in src mac Ar macaddr .It Cm in dst mac Ar macaddr .It Cm out mac Ar macaddr .It Cm out src mac Ar macaddr .It Cm out dst mac Ar macaddr True if the respective mac address field of the record matches .Ar macaddr By prepending .Cm mac with any combination of a direction specifier as defined by CISCO v9 the test is limited to those .Cm mac addresses only. Otherwise multiple matches are possible. Without any specifiers any .Cm mac address is tested against .Ar macaddr .Pp .It Cm mpls labelN Ar comp number True if the comparison of the mpls label .Cm N with .Cm N as mpls label number 1..10 matches .Ar number Filters according a specific number in the mpls label stack. .Pp .It Cm mpls eos Ar comp number True if the comparison of the end of stack mpls label matches .Ar number .Pp .It Cm mpls expN Ar comp number True if the comparison of the experimental bits 0..7 of mpls label .Cm N with .Cm N as mpls label number 1..10 matches .Ar number .Pp .It Cm packets Ar comp num .It Cm in packets Ar comp num .It Cm out packets Ar comp num True if the comparison of the packet counter in the flow record matches .Ar num. num may contain any valid scaling factor such as .Ar k, m, g Example: packets > 1k. For a single flow .Cm packets and .Cm in packets is equivalent and describes the number of packets from source to destination. In case of a bi-directional flow (sent by an exporter or combined by option .Fl -B ) the packet counter for the reverse flow can be tested with .Cm out packet .Pp .It Cm bytes Ar comp num .It Cm in bytes Ar comp num .It Cm out bytes Ar comp num True if the comparison of the byte counter in the flow record matches .Ar num. num may contain any valid scaling factor such as .Ar k, m, g Example: bytes > 1k .Cm bytes and .Cm in bytes is equivalent and describes the number of bytes from source to destination. In case of a bi-directional flow (sent by an exporter or combined by option .Fl -B ) the byte counter for the reverse flow can be tested with .Cm out bytes .Pp .It Cm flows Ar comp num True if the comparison of the flow counter in the flow record matches .Ar num. num may contain any valid scaling factor such as .Ar k, m, g For each received flow, the flow counter is set to 1, unless the exporter sends this information. If multiple flows are aggregated, this counter is increased respectively. .Pp .It Cm tos Ar num True if the type of service field of the flow record matches .Ar num .Pp .It Cm flowdir Ar direction True, if the flow direction field in the flow record matches .Ar direction. direction may be .Ar ingress, egress, 0 for ingress, or .Ar 1 for egress .Pp .It Cm duration Ar comp time True if the calculated duration of a flow (tend - tstart) compares to .Ar time. The duration is specified in msec (milliseconds) .Pp .It Cm pps Ar comp num True if the calculated value of in-packets/duration (packets per second) compares with the number .Ar num. num may contain any valid scaling factor such as .Ar k, m, g .Pp .It Cm bps Ar comp num True if the calculated value of 8*in-bytes/duration (bits per second) compares with the number .Ar num. num may contain any valid scaling factor such as .Ar k, m, g .Pp .It Cm bpp Ar comp num True if the calculated value of in-bytes/in-packets (bytes per packet) compares with the number .Ar num. num may contain any valid scaling factor such as .Ar k, m, g .Pp .It Cm observation domain id Ar comp number .It Cm observation point id Ar comp number True if the comparison of the observation domain ID or point ID field respectively matches .Ar number .Pp .It payload filters Some exporters, such as .Cm yaf or the nfdump collector .Cm nfpcap can send payload data along the netflow information. If such payloads are sent it can be filtered according the filter primitives below: .Pp .It Cm payload content Ar 'string' True if the string .Ar string is found in the payload data. .Ar string must be quoted with single or double quotes: 'string', .Dq string .Pp .It Cm payload regex Ar 'regex' .It Cm payload regex Ar 'regex' flags True if .Ar regex matches the payload data. .Cm regex searches over the full payload length. A '\0' byte does not stop the match process. .Ar regex must be quoted with single or double quotes: 'regex' or .Dq regex The regex engine understands the following reduced syntax: .Pp .Bl -item -offset indent -compact .It .Sy (...) subexpressions/capture ranges .It .Sy | the "or" operator .It .Sy ^and .Sy $ anchors .It . match any single character .It .Sy [...] and .Sy [^...] character classes .It .Sy ?, *, +, simple quantifiers .It .Sy *?, +?, ?? lazy quantifiers .It .Sy {}, {,} complex quantifiers .El .Pp .Ar flags are optional can be: .Bl -item -offset indent -compact .It .Sy m multiline .It .Sy i case insensitive matching .It .Sy s '.' includes newlines .El .Pp .It Cm payload ja3 Ar md5string True, if the payload contains the start of an SSL/TLS handshake and the calculated jas value of the handshake matches .Ar md5string .Pp .It Cm payload ja3 defined True, if the payload contains the start of an SSL/TLS handshake and a valid ja3 value can be calculated. Useful to mask out all flow records with no SSL/TLS traffic in order to generate a .Fl s .Sy ja3 statistic. .Pp .It OpenBSD pflog implemented elements .Pp .It Cm pf action Ar action True, if the respective pflog action field compares to one of .Ar pass, block, scrub, noscrub, nat, nonat, binat, nobinat, rdr, .Ar nordr, synblock, defer, match, divert, rt, afrt .It Cm pf reason Ar reason True, if the respective pflog reason field compares to one of .Ar match, bad-offset, fragment, short, normalize, memory, .Ar bad-timestamp, congestion, ip-option, proto-cksum, state-mismatch, state-insert, .Ar state-limit, src-limit, synproxy, translate, no-route .It Cm pf rule Ar ruleNr True, if the respective pflog rule number field matches .Ar ruleNr .It Cm pf dir Ar in|out True, if the respective pflog rule direction field matches .Ar in or .Ar out .It Cm pf interface Ar interfaceName True, if the respective pflog rule interface name field matches the string .Ar interfaceName .Pp .It nprobe implemented elements .Pp .It Cm client latency Ar comp time .It Cm server latency Ar comp time True, if the respective latency field in the flow record compares to .Ar time. time is specified in msec. .Pp .It CISCO ASA, network security event logging (NSEL) and NAT event logging (NEL) specific filters: .It NSEL specific filters: .Pp .It Cm asa event Ar event True if the NSEL event type of an event record matches .Ar event which may be: .Ar ignore, create, term, delete, deny .Pp .It Cm asa event Ar comp number True if the comparison of the NSEL event type of an event records matches .Ar number as a number. .Pp .It Cm asa event denied Ar reason True if the event denied type of an event records matches .Ar reason which may be .Ar ingress, egress, interface, nosyn .Pp .It Cm asa xevent Ar comp num True, if the comparison of the extended event field of the event record matches .Ar num .Pp .It Cm xip Ar ipaddr .It Cm src xip Ar ipaddr .It Cm dst xip Ar ipaddr True, if the field of the translated source or destination IP address matches .Ar ipaddr if .Cm xip is specified without .Cm src or .Cm dst both IP addresses may match. .Pp .It Cm xport Ar ipaddr .It Cm src xport Ar ipaddr .It Cm dst xport Ar ipaddr True, if the field of the translated source or destination IP address matches .Ar ipaddr if .Cm xport is specified without .Cm src or .Cm dst both ports may match. .Pp .It Cm xnet Ar network/mask .It Cm src xnet Ar network/mask .It Cm dst xnet Ar network/mask True if the translated source or destination IP address matches .Ar network if mask .Ar mask is applied. if .Cm xnet is specified without .Cm src or .Cm dst both IP addresses may match. .Pp .It Cm ingress ACL Ar comp number .It Cm ingress ACE Ar comp number .It Cm ingress XACE Ar comp number True if the comparison of the respective ingress field matches .Ar number .Pp .It Cm egress ACL Ar comp number True if the comparison of the egress field matches .Ar number .Pp .It NEL specific filters: .It Cm nat event Cm event True if the NEL event type of an event record matches .Ar event. event may be .Ar add, delete .Pp .It Cm nat event Ar comp number True if the comparison of the NEL event type of an event records matches .Ar number as a number. .Pp .It Cm nip Ar ipaddr .It Cm src nip Ar ipaddr .It Cm dst nip Ar ipaddr True, if the field of the nat source or destination IP address matches .Ar ipaddr if .Cm nip is specified without .Cm src or .Cm dst both IP addresses may match. .Pp It Cm nport Ar number .It Cm src nport Ar number .It Cm dst nport Ar number True, if the field of the nat source or destination port matches .Ar number if .Cm nip is specified without .Cm src or .Cm dst both ports may match. .Pp .It Cm ingress vrf Ar number True, if the field of the ingess vrf field of the event record matches .Ar number .Pp .It Cm pblock start Ar comp number .It Cm pblock step Ar comp number .It Cm pblock end Ar comp number True if the comparison of the start, step or end of the NAT port block in the event record matches .Ar number .It Cm port in pblock .It Cm src port in pblock .It Cm dst port in pblock True, if the source or destination port field matches the NAT port block range .Pp .It Ar comp Many filter elements support the comparison with a number. The following comparators are supported for each of those filters: .Cm =, ==, >, <, >=, <= To prevent collisions with bash interpretation, alternative comparators are available: .Cm EQ, LT, GT, LE, GE If comp is omitted, '==' is assumed. .Pp .El .Sh OUTPUT FORMAT This section describes how output formats are compiled. .Nm has a lot of already pre-defined output formats such as .Ar raw, json, csv etc. One line formats as described for option .Fl o can be compiled from various elements of a flow record. As a flow record can contains man different elements it is often useful to compile an output format for specific needs. .Pp .Ss Format description The output format is specified by .Cm -o Do fmt: Ar string Dc .Ar string contains the field .Ar tags to be printed as well as other characters if needed. A .Ar tag starts with a .Cm % sign followed by the field name. .Ar tags are separated by spaces from other tags. Characters or other strings, not starting with a .Cm % sign are copied literally to the output. .Pp Example: .Dl Fl o Do fmt:%ts %td %pr %sap -> %dap %pkt %byt %fl Dc .Pp This is the definition of the predined format .Cm line. It adds the elements .Ar tstart duration protocol source IP address/port followed by the literal characters -> and .Ar destination ip address/port packets, bytes, flows counter. Depending on the task, different output formats are required to see the required fields of a flow record. You can either extend a predefined format or specify a new one at the command line. .Pp Example: Extend the predefined format .Cm long with the the IP address of the sending router .Dl Fl o Do fmt:%long %ra Dc .Pp Predefined formats can be extended by simply add their name with a .Cm % sign somewhere in the format string. As described under the output option .Fl o .Pp .Ss Format definition .Nm has already many formats predefined. Most of the time, these format are good enough. Sometimes you may need different formats, which can be compiled as described above. In order to prevent adding the same often used output format each time you run .Nm a new output format may be define in the config file .Ar nfdump.conf The file nfdump.conf.dist contains the definition of the already hard coded formats. These may be uncommented and changed according to the specific needs. New formats may be added using the following syntax: .Pp .Dl fmt.newname = Do fmt:%ts %td %pr %sap -> %dap %pkt %byt %fl Dc .Pp with .Ar newname any new or existing definition of output formats. Existing formats are overwritten with the new definition. .Pp .Ss Tag definition The following list contains all tags, which are available to compile the output format: .Pp .Bl -tag -width "## ##" -offset indent -compact .It Cm % Inserts the predefined format at this position. e.g. %line .It Cm %cnt Record counter. record numbers are assigned dynamically assigned while reading read from file. .It Cm %nfv Netflow version. .It Cm %ts Start Time - first seen .It Cm %tfs First seen - identical to %ts .It Cm %tsr Start Time, but in fractional seconds since the epoch (1970-01-01) UNIX format. .It Cm %te End Time - last seen .It Cm %ter End Time, in fractional seconds .It Cm %tr Time the flow was received by the collector .It Cm %trr Time the flow was received, in fractional seconds .It Cm %td Duration of flow. Displayed in ddHHMMSS.msec .It Cm %pr Transort protocol .It Cm %exp Exporter ID .It Cm %eng Engine Type/ID .It Cm %lbl Flowlabel .It Cm %sa Source Address .It Cm %da Destination Address .It Cm %sap Source Address:Port .It Cm %dap Destination Address:Port .It Cm %gsap Source Address(country code):Port .It Cm %gdap Destination Address(country code):Port .It Cm %sp Source Port .It Cm %dp Destination Port .It Cm %it ICMP-type .It Cm %ic ICMP-code .It Cm %sn Source Network, mask applied .It Cm %dn Destination Network, mask applied .It Cm %nh Next-hop IP Address .It Cm %nhb BGP Next-hop IP Address .It Cm %ra Router IP Address .It Cm %sas Source AS .It Cm %das Destination AS .It Cm %nas Next AS .It Cm %pas Previous AS .It Cm %in Input Interface num .It Cm %out Output Interface num .It Cm %pkt Packets - default input .It Cm %ipkt Input Packets .It Cm %opkt Output Packets .It Cm %byt Bytes - default input .It Cm %ibyt Input Bytes .It Cm %obyt Output Bytes .It Cm %fl Flows .It Cm %flg TCP Flags .It Cm %tos Tos - default src .It Cm %stos Src Tos .It Cm %dtos Dst Tos .It Cm %dir Direction: ingress, egress .It Cm %smk Src mask .It Cm %dmk Dst mask .It Cm %fwd Forwarding Status .It Cm %svln Src vlan label .It Cm %dvln Dst vlan label .It Cm %ismc Input Src Mac Addr .It Cm %odmc Output Dst Mac Addr .It Cm %idmc Input Dst Mac Addr .It Cm %osmc Output Src Mac Addr .It Cm %mpls1 MPLS label 1 .It Cm %mpls2 MPLS label 2 .It Cm %mpls3 MPLS label 3 .It Cm %mpls4 MPLS label 4 .It Cm %mpls5 MPLS label 5 .It Cm %mpls6 MPLS label 6 .It Cm %mpls7 MPLS label 7 .It Cm %mpls8 MPLS label 8 .It Cm %mpls9 MPLS label 9 .It Cm %mpls10 MPLS label 10 .It Cm %mpls MPLS labels 1-10 .It Cm %bps bps - bits per second .It Cm %pps pps - packets per second .It Cm %bpp bps - Bytes per package .It Cm %sc src IP 2 letter country code .It Cm %dc dst IP 2 letter country code .It Cm %sloc src IP geo location info .It Cm %dloc dst IP geo location info .It Cm %sasn src AS organisation name .It Cm %dasn dst AS organisation name .It Cm %n new line char \\n .It Cm %ipl input payload .It Cm %opl output payload .It Cm %nbid nbar ID .It Cm %ja3 ja3 hash .It Cm %sni sni name in tls handshake .It Cm %nbnam nbar name .It Cm %odid observation domainID .It Cm %opid observation pointID .Pp .It OpenBSD pflog specific formats .It Cm %pfifn pflog interface name .It Cm %pfact pflog action .It Cm %pfrea pflog reason .It Cm %pfdir pflog direction .It Cm %pfrule pflog rule nr .Pp .It NSEL specific formats .It Cm %nfc NSEL connection ID .It Cm %evt NSEL event .It Cm %xevt NSEL extended event .It Cm %sgt NSEL Source security group tag .It Cm %msec NSEL event time in msec .It Cm %iacl NSEL ingress ACL .It Cm %eacl NSEL egress ACL .It Cm %xsa NSEL XLATE src IP address .It Cm %xda NSEL XLATE dst IP address .It Cm %xsp NSEL XLATE src port .It Cm %xdp NSEL SLATE dst port .It Cm %xsap Xlate Source Address:Port .It Cm %xdap Xlate Destination Address:Port .It Cm %uname NSEL user name .Pp .It NEL/NAT specific formats .It Cm %nevt NAT event - same as %evt .It Cm %ivrf NAT ingress VRF ID .It Cm %evrf NAT egress VRF ID .It Cm %nsa NAT src IP address .It Cm %nda NAT dst IP address .It Cm %nsp NAT src port .It Cm %ndp NAT dst port .It Cm %pbstart NAT pool block start .It Cm %pbend NAT pool block end .It Cm %pbstep NAT pool block step .It Cm %pbsize NAT pool block size .Pp .It Nprobe formats .It Cm %cl Client latency .It Cm %sl Server latency .It Cm %al Application latency .El .Sh EXAMPLES .Nm processes files created by any previous version of nfdump 1.6.x with some limitations for versions < 1.6.17. In order to convert flow files to the new 1.7.x binary format use the following command to read//write files: .Pp .Dl % nfdump -r oldfile -w newfile .Pp Print a statistic about the top 20 IP addresses, once sorted by flows and once by bytes .Pp .Dl % nfdump -r flowfile -s ip/flows/bytes -n 20 .Pp Print two statistics, one about the source IP and one about the destination IP address limited to flow with either source or destination port 443 .Pp .Dl % nfdump -r flowfile -s srcip/bytes -s dstip/bytes -n 20 'port 443' .Pp Print a statistic about the IP pairs, which exchanged most traffic. .Pp .Dl % nfdump -r flowfile -s record/bytes -A srcip,dstip .Pp Print all flows in raw format with a HTTP header in the payload even if flow is not on port 80. .Pp .Dl % nfdump -r flowfile -o raw Do payload regex 'GET|POST' Dc .Pp Print a statistic about all ja3 md5 sums for those flows, which a valid ja3 can be calculated .Pp .Dl % nfdump -r flowfile -s ja5 -n 0 'payload ja3 defined' .Pp Aggregate all flows and write the result back to a binary file, sorted by the start time .Pp .Dl % nfdump -r flowfile -a -Otstart -w newfile .Pp .Sh RETURN VALUES .Nm returns 0 on success and 255 if processing failed. .Sh SEE ALSO https://www.iana.org/assignments/ipfix/ipfix.xhtml .Pp https://www.cisco.com/en/US/technologies/tk648/tk362/technologies_white_paper09186a00800a3db9.html .Pp .Xr nfcapd 1 .Xr nfpcapd 1 .Xr sfcapd 1 .Xr geolookup 1 .Sh BUGS No software without bugs! Please report any bugs back to me.