.TH ucto 1 "2014 december 2"

.SH NAME
ucto \- Unicode Tokenizer
.SH SYNOPSIS
ucto [[options]] [input\(hyfile] [[output\(hyfile]]

.SH DESCRIPTION
.B ucto
ucto tokenizes text files: it separates words from punctuation, splits
sentences (and optionally paragraphs), and finds paired quotes.
Ucto is preconfigured with tokenisation rules for several languages.

.SH OPTIONS

.BR \-c " configfile"
.RS
read settings from a file
.RE

.BR \-d " value"
.RS
set debug mode to 'value'
.RE

.BR \-e " value"
.RS
set input encoding. (default UTF8)
.RE

.BR \-N " value"
.RS
set UTF8 output normalization. (default NFC)
.RE

.BR \-f
.RS
disable filtering of special characters
.RE

.BR \-L " language"
.RS
 Automatically selects a configuration file by language code. 
 The language code is generally a three-letter iso-639-3 code.
For example, 'fra' will select the file tokconfig\(hyfra from the installation directory
.RE

.BR \-l
.RS
Convert to all lowercase
.RE

.BR \-u
.RS
Convert to all uppercase
.RE

.BR \-n
.RS
Emit one sentence per line on output
.RE

.BR \-m
.RS
Assume one sentence per line on input
.RE

.BR \-\-passthru
.RS
Don't tokenize, but perform input decoding and simple token role detection
.RE

.BR \-\-filterpunct
.RS
remove most of the punctuation from the output. (not from abreviations!)
.RE

.B \-P
.RS
Disable Paragraph Detection
.RE

.B \-Q
.RS
Enable Quote Detection. (this is experimental and may lead to unexpected results)
.RE

.B \-S
.RS
Disable Sentence Detection
.RE

.B \-s
<string>
.RS
Set End\(hyof\(hysentence marker. (Default <utt>)
.RE

.B \-V
.RS
Show version information
.RE

.B \-v
.RS
set Verbose mode
.RE

.B \-F
.RS
Read a FoLiA XML document, tokenize it, and output the modified doc. (this disables usage of most other options: \-nulPQvsS)
.RE

.BR \-\-textclass "cls"
.RS
When tokenizing a FoLiA XML document, search for text nodes of class 'cls'
.RE

.B \-X
.RS
Output FoLiA XML. (this disables usage of most other options: \-nulPQvsS)
.RE

.B \-\-id
<DocId>
.RS
Use the specified Document ID for the FoLiA XML
.RE

.B \-x
<DocId>
.B (obsolete)
.RS
Output FoLiA XML, use the specified Document ID. (this disables usage of most other options: \-nulPQvsS)

.B obsolete
Use
.B \-X
and
.B \-\-id
instead
.RE

.SH BUGS
likely

.SH AUTHORS
Maarten van Gompel proycon@anaproy.nl

Ko van der Sloot Timbl@uvt.nl