NAME¶
String::Tokenizer - A simple string tokenizer.
SYNOPSIS¶
use String::Tokenizer;
# create the tokenizer and tokenize input
my $tokenizer = String::Tokenizer->new("((5+5) * 10)", '+*()');
# create tokenizer
my $tokenizer = String::Tokenizer->new();
# ... then tokenize the string
$tokenizer->tokenize("((5 + 5) - 10)", '()');
# will print '(, (, 5, +, 5, ), -, 10, )'
print join ", " => $tokenizer->getTokens();
# create tokenizer which retains whitespace
my $st = String::Tokenizer->new(
'this is a test with, (significant) whitespace',
',()',
String::Tokenizer->RETAIN_WHITESPACE
);
# this will print:
# 'this', ' ', 'is', ' ', 'a', ' ', 'test', ' ', 'with', ' ', '(', 'significant', ')', ' ', 'whitespace'
print "'" . (join "', '" => $tokenizer->getTokens()) . "'";
# get a token iterator
my $i = $tokenizer->iterator();
while ($i->hasNextToken()) {
my $next = $i->nextToken();
# peek ahead at the next token
my $look_ahead = $i->lookAheadToken();
# ...
# skip the next 2 tokens
$i->skipTokens(2);
# ...
# then backtrack 1 token
my $previous = $i->prevToken();
# ...
# get the current token
my $current = $i->currentToken();
# ...
}
DESCRIPTION¶
A simple string tokenizer which takes a string and splits it on whitespace. It
also optionally takes a string of characters to use as delimiters, and returns
them with the token set as well. This allows for splitting the string in many
different ways.
This is a very basic tokenizer, so more complex needs should be either addressed
with a custom written tokenizer or post-processing of the output generated by
this module. Basically, this will not fill everyone's needs, but it spans a
gap between simple "split / /, $string" and the other options that
involve much larger and complex modules.
Also note that this is not a lexical analyser. Many people confuse tokenization
with lexical analysis. A tokenizer merely splits its input into specific
chunks, a lexical analyzer classifies those chunks. Sometimes these two steps
are combined, but not here.
METHODS¶
- new ($string, $delimiters,
$handle_whitespace )
- If you do not supply any parameters, nothing happens, the
instance is just created. But if you do supply parameters, they are passed
on to the "tokenize" method and that method is run. For
information about those arguments, see "tokenize" below.
- setDelimiter ($delimiter)
- This can be used to set the delimiter string, this is used
by "tokenize".
- handleWhitespace ($value)
- This can be used to set the whitespace handling. It accepts
one of the two constant values "RETAIN_WHITESPACE" or
"IGNORE_WHITESPACE".
- tokenize ($string, $delimiters,
$handle_whitespace )
- Takes a $string to tokenize, and optionally a set of
$delimiter characters to facilitate the tokenization and the type of
whitespace handling with $handle_whitespace. The $string parameter and the
$handle_whitespace parameter are pretty obvious, the $delimiter parameter
is not as transparent. $delimiter is a string of characters, these
characters are then separated into individual characters and are used to
split the $string with. So given this string:
(5 + (100 * (20 - 35)) + 4)
The "tokenize" method without a $delimiter parameter would return
the following comma separated list of tokens:
'(5', '+', '(100', '*', '(20', '-', '35))', '+', '4)'
However, if you were to pass the following set of delimiters "(,
)" to "tokenize", you would get the following comma
separated list of tokens:
'(', '5', '+', '(', '100', '*', '(', '20', '-', '35', ')', ')', '+', '4', ')'
We now can differentiate the parens from the numbers, and no globbing
occurs. If you wanted to allow for optionally leaving out the whitespace
in the expression, like this:
(5+(100*(20-35))+4)
as some languages do. Then you would give this delimiter "+*-()"
to arrive at the same result.
If you decide that whitespace is significant in your string, then you need
to specify that like this:
my $st = String::Tokenizer->new(
'this is a test with, (significant) whitespace',
',()',
String::Tokenizer->RETAIN_WHITESPACE
);
A call to "getTokens" on this instance would result in the
following token set.
'this', ' ', 'is', ' ', 'a', ' ', 'test', ' ', 'with', ' ', '(', 'significant', ')', ' ', 'whitespace'
All running whitespace is grouped together into a single token, we make no
attempt to split it into its individual parts.
- getTokens
- Simply returns the array of tokens. It returns an array-ref
in scalar context.
- iterator
- Returns a String::Tokenizer::Iterator instance, see
below for more details.
INNER CLASS¶
A
String::Tokenizer::Iterator instance is returned from the
String::Tokenizer's "iterator" method and serves as yet
another means of iterating through an array of tokens. The simplest way would
be to call "getTokens" and just manipulate the array yourself, or
push the array into another object. However, iterating through a set of tokens
tends to get messy when done manually. So here I have provided the
String::Tokenizer::Iterator to address those common token processing
idioms. It is basically a bi-directional iterator which can look ahead, skip
and be reset to the beginning.
NOTE: String::Tokenizer::Iterator is an inner class, which means
that only
String::Tokenizer objects can create an instance of it. That
said, if
String::Tokenizer::Iterator's "new" method is called
from outside of the
String::Tokenizer package, an exception is thrown.
- new ($tokens_array_ref)
- This accepts an array reference of tokens and sets up the
iterator. This method can only be called from within the
String::Tokenizer package, otherwise an exception will be
thrown.
- reset
- This will reset the internal counter, bringing it back to
the beginning of the token list.
- hasNextToken
- This will return true (1) if there are more tokens to be
iterated over, and false (0) otherwise.
- hasPrevToken
- This will return true (1) if the beginning of the token
list has been reached, and false (0) otherwise.
- nextToken
- This dispenses the next available token, and move the
internal counter ahead by one.
- prevToken
- This dispenses the previous token, and moves the internal
counter back by one.
- currentToken
- This returns the current token, which will match the last
token retrieved by "nextToken".
- lookAheadToken
- This peeks ahead one token to the next one in the list.
This item will match the next item dispensed with "nextToken".
This is a non-destructive look ahead, meaning it does not alter the
position of the internal counter.
- skipToken
- This will jump the internal counter ahead by 1.
- skipTokens ($number_to_skip)
- This will jump the internal counter ahead by
$number_to_skip.
- skipTokenIfWhitespace
- This will skip the next token if it is whitespace.
- skipTokensUntil ($token_to_match)
- Given a string as a $token_to_match, this will skip all
tokens until it matches that string. If the $token_to_match is never
matched, then the iterator will return the internal pointer to its initial
state.
- collectTokensUntil ($token_to_match)
- Given a string as a $token_to_match, this will collect all
tokens until it matches that string, at which point the collected tokens
will be returned. If the $token_to_match is never matched, then the
iterator will return the internal pointer to its initial state and no
tokens will be returned.
TO DO¶
- Inline token expansion
- The Java StringTokenizer class allows for a token to be
tokenized further, therefore breaking it up more and including the results
into the current token stream. I have never used this feature in this
class, but I can see where it might be a useful one. This may be in the
next release if it works out.
Possibly compliment this expansion with compression as well, so for instance
double quoted strings could be compressed into a single token.
- Token Bookmarks
- Allow for the creation of "token bookmarks".
Meaning we could tag a specific token with a label, that index could be
returned to from any point in the token stream. We could mix this with a
memory stack as well, so that we would have an ordering to the bookmarks
as well.
BUGS¶
None that I am aware of. Of course, if you find a bug, let me know, and I will
be sure to fix it.
CODE COVERAGE¶
I use
Devel::Cover to test the code coverage of my tests, below is the
Devel::Cover report on this module's test suite.
------------------------ ------ ------ ------ ------ ------ ------ ------
File stmt branch cond sub pod time total
------------------------ ------ ------ ------ ------ ------ ------ ------
String/Tokenizer.pm 100.0 100.0 64.3 100.0 100.0 100.0 97.6
------------------------ ------ ------ ------ ------ ------ ------ ------
Total 100.0 100.0 64.3 100.0 100.0 100.0 97.6
------------------------ ------ ------ ------ ------ ------ ------ ------
SEE ALSO¶
The interface and workings of this module are based largely on the
StringTokenizer class from the Java standard library.
Below is a short list of other modules that might be considered similar to this
one. If this module does not suit your needs, you might look at one of these.
- String::Tokeniser
- Along with being a tokenizer, it also provides a means of
moving through the resulting tokens, allowing for skipping of tokens and
such. But this module looks as if it hasn't been updated from 0.01 and
that was uploaded in since 2002. The author (Simon Cozens) includes it in
the section of Acme::OneHundredNotOut entitled "The Embarrassing
Past". From what I can guess, he does not intend to maintain it
anymore.
- Parse::Tokens
- This one hasn't been touched since 2001, although it did
get up to version 0.27. It looks to lean over more towards the parser side
than a basic tokenizer.
- Text::Tokenizer
- This one looks more up to date (updated as recently as
March 2004), but is both a lexical analyzer and a tokenizer. It also uses
XS, mine is pure perl. This is something maybe to look into if you were to
need a more beefy solution that what String::Tokenizer provides.
THANKS¶
- Thanks to Stephan Tobias for finding bugs and suggestions
on whitespace handling.
AUTHOR¶
stevan little, <stevan@iinteractive.com>
COPYRIGHT AND LICENSE¶
Copyright 2004 by Infinity Interactive, Inc.
<
http://www.iinteractive.com>
This library is free software; you can redistribute it and/or modify it under
the same terms as Perl itself.