NAME¶

Plucene::Analysis::Standard::StandardTokenizer - standard tokenizer

SYNOPSIS¶

        # isa Plucene::Analysis::CharTokenizer

DESCRIPTION¶

This is the standard tokenizer. This should be a good tokenizer for most European-language documents.

METHODS¶

token_re¶

The regular expression for tokenising.

normalize¶

Remove 's and .