.\" DO NOT MODIFY THIS FILE!  It was generated by help2man 1.46.4.
.TH HFST-TOKENIZE "1" "January 2016" "HFST" "User Commands"
.SH NAME
hfst-tokenize \- =perform matching/lookup on text streams
.SH SYNOPSIS
.B hfst-tokenize
[\fI\,--segment | --xerox | --cg\/\fR] [\fI\,OPTIONS\/\fR...] \fI\,RULESET\/\fR
.SH DESCRIPTION
perform matching/lookup on text streams
.SS "Common options:"
.TP
\fB\-h\fR, \fB\-\-help\fR
Print help message
.TP
\fB\-V\fR, \fB\-\-version\fR
Print version info
.TP
\fB\-v\fR, \fB\-\-verbose\fR
Print verbosely while processing
.TP
\fB\-q\fR, \fB\-\-quiet\fR
Only print fatal erros and requested output
.TP
\fB\-s\fR, \fB\-\-silent\fR
Alias of \fB\-\-quiet\fR
.TP
\fB\-n\fR  \fB\-\-newline\fR
Newline as input separator (default is blank line)
.TP
\fB\-a\fR  \fB\-\-print\-all\fR
Print nonmatching text
.TP
\fB\-w\fR  \fB\-\-print\-weight\fR
Print weights
.TP
\fB\-\-tokenize\-multichar\fR
Tokenize multicharacter symbols
(by default only one utf\-8 character is tokenized at a time
regardless of what is present in the alphabet)
.TP
\fB\-t\fR, \fB\-\-time\-cutoff\fR=\fI\,S\/\fR
Limit search after having used S seconds per input
.TP
\fB\-\-segment\fR
Segmenting / tokenization mode (default)
.TP
\fB\-\-xerox\fR
Xerox output
.TP
\fB\-\-cg\fR
cg output
.TP
\fB\-\-finnpos\fR
FinnPos output
.PP
Use standard streams for input and output (for now).
.SH "REPORTING BUGS"
Report bugs to <hfst\-bugs@helsinki.fi> or directly to our bug tracker at:
<https://sourceforge.net/tracker/?atid=1061990&group_id=224521&func=browse>
.PP
hfst\-tokenize home page:
<https://kitwiki.csc.fi/twiki/bin/view/KitWiki//HfstTokenize>
.br
General help using HFST software:
<https://kitwiki.csc.fi/twiki/bin/view/KitWiki//HfstHome>
.SH COPYRIGHT
Copyright \(co 2010 University of Helsinki,
License GPLv3: GNU GPL version 3 <http://gnu.org/licenses/gpl.html>
.br
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.