NAME¶
ids2ngram - generate n-gram data file from ids file
SYNOPSIS¶
ids2ngram [
option]...
ids_file...
DESCRIPTION¶
ids2ngram generates idngram file, which is a sorted [id1,..,idN,freq]
array, from binary id stream files. Here, the id stream files are always
generated by
mmseg or
slmseg. Basically, it finds all occurrence
of n-words tuples (i.e. the tuple of (id1,..,idN)), and sorts these tuples by
the lexicographic order of the ids make up the tuples, then write them to
specified output file.
The input file is presented as a binary id stream, which looks like:
[id0,...,idX]
OPTIONS¶
All the following options are mandatory.
- -n,--NMax N
- Generates N-gram result. ids2ngram does only support
uni-gram, bi-gram, and trigram, so any number not in the range of 1..3 is
not valid.
- -s,--swap swap-file
- Specify the temporary intermediate file.
- -o, --out output-file
- Specify the result idngram file, e.g. the array of [id1, ..., idN,
freq]
- -p, --para N
- Specify the maximum n-gram items per paragraph. ids2ngram writes to
the temporary file on a per-paragraph basis. Every time it writes a
paragraph out, it frees the corresponding memory allocated for it. When
your computer system permits, a higher N is suggested. This can
speed up the processing speed because of less I/O.
EXAMPLE¶
Following example will use three input idstream file idsfile[1,2,3] to generate
the idngram file all.id3gram. Each para (internal map size or hash size) would
be 1024000, using swap file for temp result. All temp para result would
eventually be merged to got the final result.
ids2ngram -n 3 -s /tmp/swap -o all.id3gram -p 1024000 idsfile1 idsfile2
idsfile3
AUTHOR¶
Originally written by Phill.Zhang <phill.zhang@sun.com>. Currently
maintained by Kov.Chai <tchaikov@gmail.com>.
SEE ALSO¶
mmseg(1),
slmseg(1),
slmbuild (1).