NAME¶
Syntax::Highlight::Perl::Improved - Highlighting of Perl Syntactical Structures
VERSION¶
This file documents Syntax::Highlight::Perl::Improved version
1.0.
SYNOPSIS¶
# simple procedural
use Syntax::Highlight::Perl::Improved ':BASIC'; # or ':FULL'
print format_string($my_string);
# OO
use Syntax::Highlight::Perl::Improved;
my $formatter = new Syntax::Highlight::Perl::Improved;
print $formatter->format_string($my_string);
DESCRIPTION¶
This module provides syntax highlighting for Perl code. The design bias is
roughly line-oriented and streamed (ie, processing a file line-by-line in a
single pass). Provisions
may be made in the future for tasks related to
"back-tracking" (ie, re-doing a single line in the middle of a
stream) such as speeding up state copying.
Constructors
The only constructor provided is
"new()". When called on an existing object, "new()" will
create a new
copy of that object. Otherwise,
"new()" creates a new copy of the (internal)
Default Object.
Note that the use of the procedural syntax modifies the
Default Object
and that those changes
will be reflected in any subsequent
"new()" calls.
Formatting
Formatting is done using the "format_string()" method. Call
"format_string()" with one or more strings to format, or it will
default to using $_.
Setting and Getting Formats
You can set the text used for formatting a syntax element using
"set_format()" (or set the start and end format individually using
"set_start_format()" and "set_end_format()",
respectively).
You can also retrieve the text used for formatting for an element via
"get_start_format()" or "get_end_format". Bulk retrieval
of the names or values of defined formats is possible via
"get_format_names_list()" (names),
"get_start_format_values_list()" and
"get_end_format_values_list()".
See "FORMAT TYPES" later in this document for information on what
format elements can be used.
Checking and Setting the State
You can check certain aspects of the state of the formatter via the methods:
"in_heredoc()", "in_string()", "in_pod()",
"was_pod()", "in_data()", and "line_count()".
You can reset all of the above states (and a few other internal ones) using
"reset()".
Stable and Unstable Formatting Modes
You can set or check the stability of formatting via "unstable()".
In unstable (TRUE) mode, formatting is not considered to be persistent with
nested formats. Or, put another way, when unstable, the formatter can only
"remember" one format at a time and must reinstate formatting for
each token. An example of unstable formatting is using ANSI color escape
sequences in a terminal.
In stable (FALSE) mode (the default), formatting is considered persistent within
arbitrarily nested formats. Even in stable mode, however, formatting is never
allowed to span multiple lines; it is always fully closed at the end of the
line and reinstated at the beginning of a new line, if necessary. This is to
ensure properly balanced tags when only formatting a partial code snippet. An
example of stable formatting is HTML.
Substitutions
Using "define_substitution()", you can have the formatter substitute
certain strings with others, after the original string has been parsed (but
before formatting is applied). This is useful for escaping characters special
to the output mode (eg, > and < in HTML) without them affecting the way
the code is parsed.
You can retrieve the current substitutions (as a hash-ref) via
"substitutions()".
The Syntax::Highlight::Perl::Improved formatter recognizes and differentiates
between many Perl syntactical elements. Each type of syntactical element has a
Format Type associated with it. There is also a 'DEFAULT' type that is applied
to any element who's Format Type does not have a value.
Several of the Format Types have underscores in their name. This underscore is
special, and indicates that the Format Type can be "generalized."
This means that you can assign a value to just the first part of the Format
Type name (the part before the underscore) and that value will be applied to
all Format Types with the same first part. For example, the Format Types for
all types of variables begin with "Variable_". Thus, if you assign a
value to the Format Type "Variable", it will be applied to any type
of variable. Generalized Format Types take precedence over non-generalized
Format Types. So the value assigned to "Variable" would be applied
to "Variable_Scalar", even if "Variable_Scalar" had a
value explicitly assigned to it.
You can also define a "short-cut" name for each Format Type that can
be generalized. The short-cut name would be the part of the Format Type name
after the underscore. For example, the short-cut for
"Variable_Scalar" would be "Scalar". Short-cut names have
the least precedence and are only assigned if neither the generalized Type
name, nor the full Type name have values.
Following is a list of all the syntactical elements that
Syntax::Highlight::Perl::Improved currently recognizes, along with a short
description of what each would be applied to.
- Comment_Normal
- A normal Perl comment. Starts with '#' and goes until the
end of the line.
- Comment_POD
- Inline documentation. Starts with a line beginning with an
equal sign ('=') followed by a word (eg: '=pod') and continuing until a
line beginning with '=cut'.
- Directive
- Either the "she-bang" line at the beginning of
the file, or a line directive altering what the compiler thinks the
current line and file is.
- Label
- A loop or statement label (to be the target of a goto,
next, last or redo).
- Quote
- Any string or character that begins or ends a String.
Including, but not necessarily limited to: quote-like regular expression
operators ("m//", "s///", "tr///", etc), a
Here-Document terminating line, the lone period terminating a format, and,
of course, normal quotes ("'", """,
"`", "q{}", "qq{}", "qr{}",
"qx{}").
- String
- Any text within quotes, "format"s,
Here-Documents, Regular Expressions, and the like.
- Subroutine
- The identifier used to define, identify, or call a
subroutine (or method). Note that Syntax::Highlight::Perl::Improved cannot
recognize a subroutine if it is called without using parentheses or an
ampersand, or methods called using the indirect object syntax. It formats
those as barewords.
- Variable_Scalar
- A scalar variable.
Note that (theoretically) this format is not applied to non-scalar variables
that are being used as scalars (ie: array or hash lookups, nor references
to anything other than scalars). Syntax::Highlight::Perl::Improved figures
out (or at least tries to) the actual type of the variable being
used (by looking at how you're subscripting it) and formats it
accordingly. The first character of the variable (ie, the "$",
"@", "%", or "*") tells you the type of
value being used, and the color (hopefully) tells you the type of variable
being used to get that value.
(See "KNOWN ISSUES" for information about when this doesn't work
quite right.)
- Variable_Array
- An array variable (but not usually a slice; see
above).
- Variable_Hash
- A hash variable.
- Variable_Typeglob
- A typeglob. Note that typeglobs not beginning with an
asterisk (*) (eg: filehandles) are formatted as barewords. This is
because, well, they are.
- Whitespace
- Whitespace. Not usually formatted but it can be.
- Character
- A special, or backslash-escaped, character. For example:
"\n" (newline), or "\d" (digits).
Only occurs within strings or regular expressions.
- Keyword
- A Perl keyword. Some examples include: my, local, sub,
next.
Note that Perl does not make any distinction between keywords and built-in
functions (at least not in the documentation). Thus I had to make a
subjective call as to what would be considered keywords and what would be
built-in functions.
The list of keywords can be found (and overloaded) in the variable
$Syntax::Highlight::Perl::Improved::keyword_list_re as a pre-compiled
regular expression.
- Builtin_Function
- A Perl built-in function, called as a function (ie, using
parentheses).
The list of built-in functions can be found (and overloaded) in the variable
$Syntax::Highlight::Perl::Improved::builtin_list_re as a pre-compiled
regular expression.
- Builtin_Operator
- A Perl built-in function, called as a list or unary
operator (ie, without using parentheses).
The list of built-in functions can be found (and overloaded) in the variable
$Syntax::Highlight::Perl::Improved::builtin_list_re as a pre-compiled
regular expression.
- Operator
- A Perl operator.
The list of operators can be found (and overloaded) in the variable
$Syntax::Highlight::Perl::Improved::operator_list_re as a pre-compiled
regular expression.
- Bareword
- A bareword. This can be user-defined subroutine called
without parentheses, a typeglob used without an asterisk (*), or just a
plain old bareword.
- Package
- The name of a package or pragmatic module.
Note that this does not apply to the package portion of a fully qualified
variable name.
- Number
- A numeric literal.
- Symbol
- A symbol (ie, non-operator punctuation).
- CodeTerm
- The special tokens that signal the end of executable code
and the begining of the DATA section. Specifically, '"__END__"'
and '"__DATA__"'.
- DATA
- Anything in the DATA section (see
"CodeTerm").
PROCEDURAL vs. OBJECT ORIENTED¶
Syntax::Highlight::Perl::Improved uses OO method-calls internally (and actually
defines a Default Object that is used when the functions are invoked
procedurally) so you will not gain anything (efficiency-wise) by using the
procedural interface. It is just a matter of style.
It is actually recommended that you use the OO interface, as this allows you to
instantiate multiple, concurrent-yet-separate formatters. Though I cannot
think of
why you would
need multiple formatters instantiated.
:-)
One point to note: the "new()" method uses the Default Object to
initialize new objects. This means that any changes to the state of the
Default Object (including Format definitions) made by using the procedural
interface will be reflected in any subsequently created objects. This can be
useful in some cases (eg, call "set_format()" procedurally just
before creating a batch of new objects to define default Formats for them all)
but will most likely lead to trouble.
METHODS¶
- new PACKAGE
- new OBJECT
- Creates a new object. If called on an existing object,
creates a new copy of that object (which is thenceforth totally separate
from the original).
- reset
- Resets the object's internal state. This breaks out of
strings and here-docs, ends PODs, resets the line-count, and otherwise
gets the object back into a "normal" state to begin processing a
new stream.
Note that this does not reset any user options
(including formats and format stability).
- unstable EXPR
- unstable
- Returns true if the formatter is in unstable mode.
If called with a non-zero number, puts the formatter into unstable
formatting mode.
In unstable mode, it is assumed that formatting is not persistent one token
to the next and that each token must be explicitly formatted.
- in_heredoc
- Returns true if the next string to be formatted will be
inside a Here-Document.
- in_string
- Returns true if the next string to be formatted will be
inside a multi-line string.
- in_pod
- Returns true if the formatter would consider the next
string passed to it as begin within a POD structure. This is false
immediately before any POD instigators ("=pod",
"=head1", "=item", etc), true immediately after an
instigator, throughout the POD and immediately before the POD terminator
("=cut"), and false immediately after the POD terminator.
- was_pod
- Returns true if the last line of the string just formatted
was part of a POD structure. This includes the "/^=\w+/" POD
instigators and terminators.
- in_data
- Returns true if the next string to be formatted will be
inside the DATA section (ie, follows a "__DATA__" or
"__END__" tag).
- line_count
- Returns the number of lines processed by the
formatter.
- substitutions
- Returns a reference to the substitution table used. The
substitution table is a hash whose keys are the strings to be replaced,
and whose values are what to replace them with.
- define_substitution HASH_REF
- define_substitution LIST
- Allows user to define certain characters that will be
substituted before formatting is done (but after they have been processed
for meaning).
If the first parameter is a reference to a hash, the formatter will replace
it's own hash with the given one, and subsequent changes to the hash
outside the formatter will be reflected.
Otherwise, it will copy the arguments passed into it's own hash, and any
substitutions already defined (but not in the parameter list) will be
preserved. (ie, the new substitutions will be added, without destroying
what was there already.)
- set_start_format HASH_REF
- set_start_format LIST
- Given either a list of keys/values, or a reference to a
hash of keys/values, copy them into the object's Formats list.
- set_end_format HASH_REF
- set_end_format LIST
- Given either a list of keys/values, or a reference to a
hash of keys/values, copy them into the object's Formats list.
- set_format LIST
- Sets the formatting string for one or more formats.
You should pass a list of keys/values where the keys are the format names
and the values are references to arrays containing the starting and ending
formatting strings (in that order) for that format.
- get_start_format LIST
- Retrieve the string that is inserted to begin a given
format type (starting format string).
The names are looked for in the following order:
First: Prefer the names joined by underscore, from most general to
least. For example, given ("Variable", "Scalar"):
"Variable" then "Variable_Scalar".
Second: Then try each name singly, in reverse order. For example,
"Scalar" then "Variable".
See "FORMAT TYPES" for more information.
- get_end_format LIST
- Retrieve the string that is inserted to end a given format
type (ending format string).
- get_format_names_list
- Returns a list of the names of all the Formats
defined.
- get_start_format_values_list
- Returns a list of the values of all the start
Formats defined (in the same order as the names returned by
"get_format_names_list()").
- get_end_format_values_list
- Returns a list of the values of all the end Formats
defined (in the same order as the names returned by
"get_format_names_list()").
- format_string LIST
- Formats one or more strings of Perl code. If no strings are
specified, defaults to $_. Returns the list of formatted strings (or the
first string formatted if called in scalar context).
Note: The end of the string is considered to be the end of a line,
regardless of whether or not there is a trailing line-break (but trailing
line-breaks will not cause an extra, empty line).
Another Note: The function actually uses $/ to determine
line-breaks, unless $/ is set to "\n" (newline). If $/ is
"\n", then it looks for the first match of
"m/\r?\n|\n?\r/" in the string and uses that to determine
line-breaks. This is to make it easy to handle non-unix text. Whatever
characters it ends up using as line-breaks are preserved.
- format_token TOKEN, LIST
- Returns TOKEN wrapped in the start and end Formats
corresponding to LIST (as would be returned by "get_start_format(
LIST )" and "get_end_format( LIST )", respectively).
No syntax checking is done on TOKEN but substitutions defined with
"define_substitution()" are performed.
KNOWN ISSUES or LIMITATIONS¶
- •
- Barewords used as keys to a hash are formatted as strings.
This is Good. They should not be, however, if they are not the only thing
within the curly braces. That can be fixed.
- •
- This version does not handle formats (see
perlform(1)) very well. It treats them as Here-Documents and
ignores the rules for comment lines, as well as the fact that picture
lines are not supposed to be interpolated. Thus, your picture lines will
look strange with the '@'s being formatted as array variables (albeit,
invalid ones). Ideally, it would also treat value lines as normal Perl
code and format accordingly. I think I'll get to the comment lines and
non-interpolating picture lines first. If/When I do get this fixed, I will
most likely add a format type of 'Format' or something, so that they can
be formatted differently, if so desired.
- •
- This version does not handle Regular Expression significant
characters. It simply treats Regular Expressions as interpolated
strings.
- •
- User-defined subroutines, called without parentheses, are
formatted as barewords. This is because there is no way to tell them apart
from barewords without parsing the code, and would require us to go as far
as perl does when doing the "-c" check (ie, executing BEGIN and
END blocks and the like). That's not going to happen.
- •
- If you are indexing (subscripting) an array or hash, the
formatter tries to figure out the "real" variable class by
looking at how you index the variable. However, if you do something funky
(but legal in Perl) and put line-breaks or comments between the variable
class character ($) and your identifier, the formatter will get confused
and treat your variable as a scalar. Until it finds the index character.
Then it will format the scalar class character ($) as a scalar and your
identifier as the "correct" class.
- •
- If you put a line-break between your variable identifier
and it's indexing character (see above), which is also legal in Perl, the
formatter will never find it and treat your variable as a scalar.
- •
- If you put a line-break between a bareword hash-subscript
and the hash variable, or between a bareword and its associated
"=>" operator, the bareword will not be formatted correctly
(as a string). (Noticing a pattern here?)
BUGS¶
Bug reports are always welcome. Email me at b<davidcyl@cpan.org>.
AUTHOR¶
David C.Y. Liu b<davidcyl@cpan.org>
based on code by Cory Johns
darkness@yossman.net
Copyright (c) 2004 David C.Y. Liu. This library is free software; you can
redistribute and/or modify it under the same conditions as Perl itself.
TO DO¶
Note: This is Cory John's todo list, not mine. Currently none of these
features are planned for the near future.
- 1
- Improve handling of regular expressions. Add support for
regexp-special characters. Recognize the /e option to the substitution
operator (maybe).
- 2
- Improve handling of formats. Don't treat format definitions
as interpolating. Handle format-comments. Possibly format value lines as
normal Perl code.
- 3
- Create in-memory deep-copy routine to replace
"eval(Data::Dumper)" deep-copy.
- 4
- Generalize state transitions ("reset()" and, in
the future, "copy_state()") to use non-hard-coded keys and
values for state variables. Probably will extrapolate them into an
overloadable hash, and use the aforementioned deep-copy to assign
them.
- 5
- Create a method to save or copy states between objects
("copy_state()"). Would be useful for using this module in an
editor.
- 6
- Add support for greater-than-one length special characters.
Specifically, octal, hexidecimal, and control character codes. For
example, "\644", "\x1a4" or "\c[".
REVISIONS¶
05-03-2004 David C.Y. Liu (Version 1.01)
- •
- Added 'our' to the keywords list.
- •
- Fixed bug that prevented interpolation inside qq()
quotes.
- •
- Renamed to Syntax::Highlight::Perl::Improved.
04-04-2001 Cory Johns
- •
- Fixed problem with special characters not formatting inside
of Here-Documents.
- •
- Fixed bug causing hash variables to format inside of
Here-Documents.
03-30-2001 Cory Johns
- •
- Fixed bug where quote-terminators were checked for inside
of Here-Documents.
03-29-2001 Cory Johns
- •
- Moved token processing tests from _format_line()
into _process_token() (where they should've been all along),
generally making _format_line() more logical. Contemplating
extrapolating the tokenizing and token loop into its own subroutine to
avoid all the recursive calls.
- •
- Fixed bug that caused special characters to be recognized
outside of strings.
- •
- Added $VERSION variable.
- •
- Added support for different types of literal numbers:
floating point, exponential notation (eg: 1.3e10), hexidecimal, and
underscore-separated.
- •
- Added the "CodeTerm" and "DATA"
Formats.
03-27-2001 Cory Johns
- •
- Added was_pod() and updated the documentation for
in_pod().
03-20-2001 Cory Johns
- •
- Added support for Perl formats (ie, `"format =
..."').