table of contents
Plucene::Analysis::Standard::StandardTokenizer(3pm) | User Contributed Perl Documentation | Plucene::Analysis::Standard::StandardTokenizer(3pm) |
NAME¶
Plucene::Analysis::Standard::StandardTokenizer - standard tokenizerSYNOPSIS¶
# isa Plucene::Analysis::CharTokenizer
DESCRIPTION¶
This is the standard tokenizer.This should be a good tokenizer for most European-language documents.
METHODS¶
token_re¶
The regular expression for tokenising.normalize¶
Remove 's and .2018-04-02 | perl v5.26.1 |