NAME¶
PPIx::Regexp::Tokenizer - Tokenize a regular expression
SYNOPSIS¶
use PPIx::Regexp::Dumper;
PPIx::Regexp::Dumper->new( 'qr{foo}smx' )
->print();
INHERITANCE¶
"PPIx::Regexp::Tokenizer" is a PPIx::Regexp::Support.
"PPIx::Regexp::Tokenizer" has no descendants.
DESCRIPTION¶
This class provides tokenization of the regular expression.
METHODS¶
This class provides the following public methods. Methods not documented here
(or documented below under "EXTERNAL TOKENIZERS") are private, and
unsupported in the sense that the author reserves the right to change or
remove them without notice.
new¶
my $tokenizer = PPIx::Regexp::Tokenizer->new( 'xyzzy' );
This static method instantiates the tokenizer. You must pass it the regular
expression to be parsed, either as a string or as a PPI::Element of some sort.
You can also pass optional name/value pairs of arguments. The option names are
specified
without a leading dash. Supported options are:
- default_modifiers array_reference
- This argument specifies default statement modifiers. It is optional, but
if specified must be an array reference. See the PPIx::Regexp new()
documentation for the details.
- encoding name
- This option specifies the encoding of the string to be tokenized. If
specified, an "Encode::decode" is done on the string (or the
"content" of the PPI class) before it is tokenized.
- trace number
- Specifying a positive value for this option causes a trace of the
tokenization. This option is unsupported in the sense that the author
reserves the right to alter it without notice.
If this option is unspecified, the value comes from environment variable
"PPIX_REGEXP_TOKENIZER_TRACE" (see "ENVIRONMENT
VARIABLES"). If this environment variable does not exist, the default
is 0.
Undocumented options are unsupported.
The returned value is the instantiated tokenizer, or "undef" if
instantiation failed. In the latter case a call to "errstr" will
return the reason.
content¶
print $tokenizer->content();
This method returns the string being tokenized. This will be the result of the
PPI::Element->
content() method if the object was instantiated with
a PPI::Element.
default_modifiers¶
print join ', ', @{ $tokenizer->default_modifiers() };
This method returns a reference to a copy of the array passed to the
"default_modifiers" argument to
new(). If this argument was
not used to instantiate the object, the return is a reference to an empty
array.
encoding¶
This method returns the encoding of the data being parsed, if one was set when
the class was instantiated; otherwise it simply returns undef.
errstr¶
my $tokenizer = PPIx::Regexp::Tokenizer->new( 'xyzzy' )
or die PPIx::Regexp::Tokenizer->errstr();
This static method returns an error description if tokenizer instantiation
failed.
failures¶
print $tokenizer->failures(), " tokenization failures\n";
This method returns the number of tokenization failures encountered. A
tokenization failure is represented in the output token stream by a
PPIx::Regexp::Token::Unknown.
modifier¶
$tokenizer->modifier( 'x' )
and print "Tokenizing an extended regular expression\n";
This method returns true if the given modifier character was found on the end of
the regular expression, and false otherwise.
next_token¶
my $token = $tokenizer->next_token();
This method returns the next token in the token stream, or nothing if there are
no more tokens.
significant¶
This method exists simply for the convenience of PPIx::Regexp::Dumper. It always
returns true.
tokens¶
my @tokens = $tokenizer->tokens();
This method returns all remaining tokens in the token stream.
EXTERNAL TOKENIZERS¶
This class does very little of its own tokenization. Instead the token classes
contain external tokenization routines, whose name is '__PPIX_TOKENIZER__'
concatenated with the current mode of the tokenizer ('regexp' for regular
expressions, 'repl' for the replacement string).
These external tokenizers are called as static methods, and passed the
"PPIx::Regexp::Tokenizer" object and the current character in the
character stream.
If the external tokenizer wants to make one or more tokens, it returns an array
containing either length in characters for tokens of the tokenizer's own
class, or the results of one or more "make_token" calls for tokens
of an arbitrary class.
If the external tokenizer is not interested in the characters starting at the
current position it simply returns.
The following methods are for the use of external tokenizers, and
are
not part of the public interface to this class.
capture¶
if ( $tokenizer->find_regexp( qr{ \A ( foo ) }smx ) ) {
foreach ( $tokenizer->capture() ) {
print "$_\n";
}
}
This method returns all the contents of any capture buffers from the previous
call to "find_regexp". The first element of the array (i.e. element
0) corresponds to $1, and so on.
The captures are cleared by "make_token", as well as by another call
to "find_regexp".
cookie¶
$tokenizer->cookie( foo => sub { 1 } );
my $cookie = $tokenizer->cookie( 'foo' );
my $old_hint = $tokenizer->cookie( foo => undef );
This method either creates, deletes, or accesses a cookie.
A cookie is a code reference which is called whenever the tokenizer makes a
token. If it returns a false value, it is deleted. Explicitly setting the
cookie to "undef" also deletes it.
When you call "$tokenizer->cookie( 'foo' )", the current cookie is
returned. If you pass a new value of "undef" to delete the token,
the deleted cookie (if any) is returned.
When the "make_token" method calls a cookie, it passes it the
tokenizer and the token just made. If a token calls a cookie, it is
recommended that it merely pass the tokenizer, though of course the token can
do whatever it wants.
The cookie mechanism seems to be a bit of a crock, but it appeared to be more
work to fix things up in the lexer after the tokenizer got something wrong.
The recommended way to write a cookie is to use a closure to store any necessary
data, and have a call to the cookie return the data; otherwise the ultimate
consumer of the cookie has no way to access the data. Of course, it may be
that the presence of the cookie at a certain point in the parse is all that is
required.
expect¶
$tokenizer->expect( 'PPIx::Regexp::Token::Code' );
This method inserts a given class at the head of the token scan, for the next
iteration only. More than one class can be specified. Class names can be
abbreviated by removing the leading 'PPIx::Regexp::'.
If no class is specified, this method does nothing.
The expectation lasts from the next time "get_token" is called until
the next time make_token makes a significant token, or until the next
"expect" call if that is done sooner.
find_regexp¶
my $end = $tokenizer->find_regexp( qr{ \A \w+ }smx );
my ( $begin, $end ) = $tokenizer->find_regexp(
qr{ \A \w+ }smx );
This method finds the given regular expression in the content, starting at the
current position. If called in scalar context, the offset from the current
position to the end of the matched string is returned. If called in list
context, the offsets to both the beginning and the end of the matched string
are returned.
find_matching_delimiter¶
my $offset = $tokenizer->find_matching_delimiter();
This method is used by tokenizers to find the delimiter matching the character
at the current position in the content string. If the delimiter is an opening
bracket of some sort, bracket nesting will be taken into account.
When searching for the matching delimiter, the back slash character is
considered to escape the following character, so back-slashed delimiters will
be ignored. No other quoting mechanisms are recognized, though, so delimiters
inside quotes still count. This is actually the way Perl works, as
$ perl -e 'qr<(?{ print "}" })>'
demonstrates.
This method returns the offset from the current position in the content string
to the matching delimiter (which will always be positive), or undef if no
match can be found.
get_start_delimiter¶
my $start_delimiter = $tokenizer->get_start_delimiter();
This method is used by tokenizers to access the start delimiter for the regular
expression.
get_token¶
my $token = $tokenizer->make_token( 3 );
my @tokens = $tokenizer->get_token();
This method returns the next token that can be made from the input stream. It is
not part of the external interface, but is intended for the use of an
external tokenizer which calls it after making and retaining its own token to
look at the next token ( if any ) in the input stream.
If any external tokenizer calls get_token without first calling make_token, a
fatal error occurs; this is better than the infinite recursion which would
occur if the condition were not trapped.
An external tokenizer
must return anything returned by get_token;
otherwise tokens get lost.
interpolates¶
This method returns true if the top-level structure being tokenized
interpolates; that is, if the delimiter is not a single quote.
make_token¶
return $tokenizer->make_token( 3, 'PPIx::Regexp::Token::Unknown' );
This method is used by this class (and possibly by individual tokenizers) to
manufacture a token. Its arguments are the number of characters to include in
the token, and optionally the class of the token. If no class name is given,
the caller's class is used. Class names may be shortened by removing the
initial 'PPIx::Regexp::', which will be restored by this method.
The token will be manufactured from the given number of characters starting at
the current cursor position, which will be adjusted.
If the given length would include characters past the end of the string being
tokenized, the length is reduced appropriately. If this means a token with no
characters, nothing is returned.
match¶
if ( $tokenizer->find_regexp( qr{ \A \w+ }smx ) ) {
print $tokenizer->match(), "\n";
}
This method returns the string matched by the previous call to
"find_regexp".
The match is set to "undef" by "make_token", as well as by
another call to "find_regexp".
modifier_duplicate¶
$tokenizer->modifier_duplicate();
This method duplicates the modifiers on the top of the modifier stack, with the
intent of creating a locally-scoped copy of the modifiers. This should only be
called by an external tokenizer that is actually creating a modifier scope. In
other words, only when creating a PPIx::Regexp::Token::Structure token whose
content is '('.
modifier_modify¶
$tokenizer->modifier_modify( name => $value ... );
This method sets new values for the modifiers in the local scope. Only the
modifiers whose names are actually passed have their values changed.
This method is intended to be called after manufacturing a
PPIx::Regexp::Token::Modifier token, and passed the results of its
"modifiers" method.
modifier_pop¶
$tokenizer->modifier_pop();
This method removes the modifiers on the top of the modifier stack. This should
only be called by an external tokenizer that is ending a modifier scope. In
other words, only when creating a PPIx::Regexp::Token::Structure token whose
content is ')'.
Note that this method will never pop the last modifier item off the stack, to
guard against unmatched right parentheses.
peek¶
my $character = $tokenizer->peek();
my $next_char = $tokenizer->peek( 1 );
This method returns the character at the given non-negative offset from the
current position. If no offset is given, an offset of 0 is used.
If you ask for a negative offset or an offset off the end of the sting,
"undef" is returned.
ppi_document¶
This method makes a PPI document out of the remainder of the string, and returns
it.
prior¶
$tokenizer->prior( 'can_be_quantified' )
and print "The prior token can be quantified.\n";
This method calls the named method on the most-recently-instantiated significant
token, and returns the result. Any arguments subsequent to the method name
will be passed to the method.
Because this method is designed to be used within the tokenizing system, it will
die horribly if the named method does not exist.
ENVIRONMENT VARIABLES¶
A tokenizer trace can be requested by setting environment variable
PPIX_REGEXP_TOKENIZER_TRACE to a numeric value other than 0. Use of this
environment variable is unsupported in the same sense that the
"trace" option of "new" is unsupported. Explicitly
specifying the "trace" option to "new" overrides the
environment variable.
The real reason this is documented is to give the user a way to troubleshoot
funny output from the tokenizer.
SUPPORT¶
Support is by the author. Please file bug reports at <
http://rt.cpan.org>,
or in electronic mail to the author.
AUTHOR¶
Thomas R. Wyant, III
wyant at cpan dot org
COPYRIGHT AND LICENSE¶
Copyright (C) 2009-2014 by Thomas R. Wyant, III
This program is free software; you can redistribute it and/or modify it under
the same terms as Perl 5.10.0. For more details, see the full text of the
licenses in the directory LICENSES.
This program is distributed in the hope that it will be useful, but without any
warranty; without even the implied warranty of merchantability or fitness for
a particular purpose.