.\" -*- mode: troff; coding: utf-8 -*-
.\" Automatically generated by Pod::Man 5.01 (Pod::Simple 3.43)
.\"
.\" Standard preamble:
.\" ========================================================================
.de Sp \" Vertical space (when we can't use .PP)
.if t .sp .5v
.if n .sp
..
.de Vb \" Begin verbatim text
.ft CW
.nf
.ne \\$1
..
.de Ve \" End verbatim text
.ft R
.fi
..
.\" \*(C` and \*(C' are quotes in nroff, nothing in troff, for use with C<>.
.ie n \{\
.    ds C` ""
.    ds C' ""
'br\}
.el\{\
.    ds C`
.    ds C'
'br\}
.\"
.\" Escape single quotes in literal strings from groff's Unicode transform.
.ie \n(.g .ds Aq \(aq
.el       .ds Aq '
.\"
.\" If the F register is >0, we'll generate index entries on stderr for
.\" titles (.TH), headers (.SH), subsections (.SS), items (.Ip), and index
.\" entries marked with X<> in POD.  Of course, you'll have to process the
.\" output yourself in some meaningful fashion.
.\"
.\" Avoid warning from groff about undefined register 'F'.
.de IX
..
.nr rF 0
.if \n(.g .if rF .nr rF 1
.if (\n(rF:(\n(.g==0)) \{\
.    if \nF \{\
.        de IX
.        tm Index:\\$1\t\\n%\t"\\$2"
..
.        if !\nF==2 \{\
.            nr % 0
.            nr F 2
.        \}
.    \}
.\}
.rr rF
.\" ========================================================================
.\"
.IX Title "LaTeXML::Core::Token 3pm"
.TH LaTeXML::Core::Token 3pm 2024-02-27 "perl v5.38.2" "User Contributed Perl Documentation"
.\" For nroff, turn off justification.  Always turn off hyphenation; it makes
.\" way too many mistakes in technical documents.
.if n .ad l
.nh
.SH NAME
"LaTeXML::Core::Token" \- representation of a Token:
a pair of character and category code (catcode);
It extends LaTeXML::Common::Object.
.SS "Exported functions"
.IX Subsection "Exported functions"
.ie n .IP """$catcode = CC_ESCAPE;""" 4
.el .IP "\f(CW$catcode = CC_ESCAPE;\fR" 4
.IX Item "$catcode = CC_ESCAPE;"
Constants for the category codes:
.Sp
.Vb 4
\&  CC_BEGIN, CC_END, CC_MATH, CC_ALIGN, CC_EOL,
\&  CC_PARAM, CC_SUPER, CC_SUB, CC_IGNORE,
\&  CC_SPACE, CC_LETTER, CC_OTHER, CC_ACTIVE,
\&  CC_COMMENT, CC_INVALID, CC_CS.
.Ve
.Sp
[The last 2 are (apparent) extensions,
with catcodes 16 and 17, respectively].
.ie n .IP """$token = Token($string,$cc);""" 4
.el .IP "\f(CW$token = Token($string,$cc);\fR" 4
.IX Item "$token = Token($string,$cc);"
Creates a LaTeXML::Core::Token with the given content and catcode.
The following shorthand versions are also exported for convenience:
.Sp
.Vb 4
\&  T_BEGIN, T_END, T_MATH, T_ALIGN, T_PARAM,
\&  T_SUB, T_SUPER, T_SPACE, T_LETTER($letter),
\&  T_OTHER($char), T_ACTIVE($char),
\&  T_COMMENT($comment), T_CS($cs)
.Ve
.ie n .IP """@tokens = Explode($string);""" 4
.el .IP "\f(CW@tokens = Explode($string);\fR" 4
.IX Item "@tokens = Explode($string);"
Returns a list of the tokens corresponding to the characters in \f(CW$string\fR.
All tokens have catcode CC_OTHER, except for spaces which have catcode CC_SPACE.
.ie n .IP """@tokens = ExplodeText($string);""" 4
.el .IP "\f(CW@tokens = ExplodeText($string);\fR" 4
.IX Item "@tokens = ExplodeText($string);"
Returns a list of the tokens corresponding to the characters in \f(CW$string\fR.
All (roman) letters have catcode CC_LETTER, all others have catcode CC_OTHER,
except for spaces which have catcode CC_SPACE.
.ie n .IP """UnTeX($object, $suppress_linebreaks);""" 4
.el .IP "\f(CWUnTeX($object, $suppress_linebreaks);\fR" 4
.IX Item "UnTeX($object, $suppress_linebreaks);"
Converts \f(CW$object\fR to a string containing TeX that created it (or could have).
Note that this is not necessarily the original TeX code; expansions
or other substitutions may have taken place.
.Sp
Line-breaking of the generated TeX can be explicitly requested or disabled
by passing 0 or 1 as the second \f(CW$suppress_linebreaks\fR argument.
The default behavior of line-breaking is controlled by
the global State value \f(CW\*(C`SUPPRESS_UNTEX_LINEBREAKS\*(C'\fR.
.SS Methods
.IX Subsection "Methods"
.ie n .IP """@tokens = $object\->unlist;""" 4
.el .IP "\f(CW@tokens = $object\->unlist;\fR" 4
.IX Item "@tokens = $object->unlist;"
Return a list of the tokens making up this \f(CW$object\fR.
.ie n .IP """$string = $object\->toString;""" 4
.el .IP "\f(CW$string = $object\->toString;\fR" 4
.IX Item "$string = $object->toString;"
Return a string representing \f(CW$object\fR.
.ie n .IP """$string = $token\->getCSName;""" 4
.el .IP "\f(CW$string = $token\->getCSName;\fR" 4
.IX Item "$string = $token->getCSName;"
Return the string or character part of the \f(CW$token\fR; for the special category
codes, returns the standard string (eg. \f(CW\*(C`T_BEGIN\->getCSName\*(C'\fR returns "{").
.ie n .IP """$string = $token\->getString;""" 4
.el .IP "\f(CW$string = $token\->getString;\fR" 4
.IX Item "$string = $token->getString;"
Return the string or character part of the \f(CW$token\fR.
.ie n .IP """$code = $token\->getCharcode;""" 4
.el .IP "\f(CW$code = $token\->getCharcode;\fR" 4
.IX Item "$code = $token->getCharcode;"
Return the character code of the character part of the \f(CW$token\fR,
or 256 if it is a control sequence.
.ie n .IP """$code = $token\->getCatcode;""" 4
.el .IP "\f(CW$code = $token\->getCatcode;\fR" 4
.IX Item "$code = $token->getCatcode;"
Return the catcode of the \f(CW$token\fR.
.SH AUTHOR
.IX Header "AUTHOR"
pBruce Miller <bruce.miller@nist.gov>
.SH COPYRIGHT
.IX Header "COPYRIGHT"
Public domain software, produced as part of work done by the
United States Government & not subject to copyright in the US.