NAME¶
toke - OpenBIOS tokenizer
SYNOPSIS¶
toke [options] forth-file
DESCRIPTION¶
toke is an open-source FCode Tokenizer from the OpenBIOS project.
OPTIONS¶
- -h, -?
- Print a brief help message and then exit.
- -i, --ignore-errors
- Generate a Binary Output even if errors were reported.
- -l, --load-list
- Collect the names of floaded files into an
FLoad-List file. The names collected are in the same form as they were
presented in the fload statements.
The name of the FLoad-List File is derived from the name of the binary
output file, by replacing its extension with .fl , or, if the
binary output file name had no extension, merely appending the extension
.fl.
The binary output file name used for this purpose is either the one
specified on the command line, or the one created by default.
- -o, --output-name OutputFileName,
- Direct the binary output (FCode result of Tokenization) to
the named file instead of to the default-named file. This option is not
valid when multiple input files are named.
- -P, --dependencies
- Collect the fully-resolved pathnames of floaded and ENCODEd
files into a dependency-list file. The names collected are in the form
that is presented to the host operating system: shell environment
Variables and related expressions will be fully expanded, and the
directory within the include-list in which the file was found will be
attached.
The name of the dependency-list file will be the same as that of the
FLoad-list file, except that its extension will be .P instead of
.fl.
- -v, --verbose
- Print additional messages (including advisories) during
tokenization.
- -I, --Include directory
- This tokenizer supports the notion of an include-list. The
user creates the include-list by specifying a number of -I
directory pairs on the command-line. All file-reads, whether for an
fload command or an encode-file directive, will involve a search for the
named file through the directories of the include-list, in the order they
were supplied on the command-line.
If no include-list is created, file-reads are relative to the current
working Directory. If an include-list is created, file-reads are
restricted to the directories within it. For the current working directory
to be included in the file-search, it must be specified explicitly.
-I. will accomplish that quite effectively.
- -T, --Trace symbol
- This tokenizer supports the notion of a trace-list. The
User creates the trace-list by specifying a number of -T symbol
pairs on the command-line.
When a name is defined, whether as an FCode, an alias, a Macro or anything
else, either in normal tokenization mode or "Tokenizer
Escape"-mode, if it matches a symbol that has been added to the trace
list, a trace note message will be issued indicating that a definition of
that name has been created. Subsequent trace note Messages will be issued
when the definition of that name is invoked.
This trace-symbols feature can be helpful during maintenance of legacy code,
for instance, when multiple symbols carry the same name.
- -d, --define Symbol[=Value]
- Define a command-line symbol and optionally, assign a value
to it. If you wish the value to contain spaces or quotes, you can
accomplish that using the shell escape conventions. This sequence may be
repeated. Once a Symbol is defined on the command-line, it stays in effect
for the duration of the entire batch of tokenizations (i.e., if there are
multiple input files named on the command line). Command-line Symbols can
be tested for purposes of conditional tokenization, or their assigned
values can be evaluated.
- -f, --flag [no]<FlagName>
- The tokenizer recognizes a specific set of special-feature
flag-names; each is associated with a specific non-standard variant
behavior. Pass the flag-name as an argument to the -f switch to
enable the behavior; to disable it, precede the flag-name with the
optional string no.
The settings of the special-feature flags can also be changed or displayed
from within the source input file.
The special-feature flags are all initially set to be enabled, except where
noted.
The flag-names and their associated special-features are as follows:
- Local-Values
- Support IBM-style Local Values ("LV"s). Initially
disabled.
- LV-Legacy-Separator
- Allow Semicolon for Local Values Separator
("Legacy").
- LV-Legacy-Message
- Display a Warning Message when Semicolon is used as the
Local Values Separator.
- ABORT-Quote
- Allow ABORT" macro.
- Sun-ABORT-Quote
- ABORT" with implicit IF ... THEN
- Abort-Quote-Throw
- Use -2 THROW, rather than ABORT, in an Abort"
phrase
- String-remark-escape
- Allow "\ (Quote-Backslash) to interrupt string
parsing.
- Hex-remark-escape
- Allow \ (Backslash) to interrupt hex-sequence parsing
within a string.
- C-Style-string-escape
- Allow the C-style String-Escape pairs \n \t and \xx\ to be
treated as special characters in string parsing.
- Always-Headers
- Override occurrences of the standard directive headerless
in the source with -- effectively -- headers to make all definitions have
a header. Occurrences of the directive external will continue to behave in
the standard manner. Initially disabled.
- Always-External
- All definitions will be made as though under the external
directive; occurrences of either Standard directive headerless or headers
in the source will be overridden. This special-feature flag will also
override the Always-Headers special-feature flag in the event that
both have been specified. Initially disabled.
- Warn-if-Duplicate
- Display a WARNING message whenever a definition is made
whose name duplicates that of an existing definition. Disabling this flag
will suspend the duplicate-names test globally, until it is re-enabled. A
Directive is supported that will suspend the test for the duration of only
a single definition, without affecting global behavior.
- Obsolete-FCode-Warning
- Display a WARNING message whenever an FCode function is
invoked that the Standard identifies as obsolete.
- Trace-Conditionals
- Issue advisory messages about the state of conditional
tokenization. (Remember that advisory messages are displayed only if the
verbose option -v is set.) Initially disabled.
- Upper-Case-Token-Names
- Lower-Case-Token-Names When outputting the names of
headered functions ( token-names) to the binary output file,
override the character-case in which the names appeared in the source, and
convert them to Upper- or Lower- -Case, respectively. (These
flags do not affect text string sequences, whose character-case is always
preserved.) Initially disabled.
- Big-End-PCI-Rev-Level
- Save the Revision Level of the Vendor's ROM field of
the PCI Header in big-endian byte-order, rather than little-endian as per
the general PCI Standard convention. (This flag does not affect any other
field
of the PCI Header). Initially disabled.
- Ret-Stk-Interp
- Allow return-stack operations during interpretation. While
the standard specifies that usage of the operators >r,
r@, and r> while interpreting is allowed, actual practice
in the industry is inconsistent. Developers who wish to take a more
cautious approach to this question can disable this flag so that any
attempt to use the operators >r, r@, and r> in
the interpreting state will generate an ERROR Message.
Also, the pseudo-flag-name
help will cause a list of the flag-names and
their associated special-features to be printed.
SEE ALSO¶
detok(1).
AUTHORS¶
Stefan Reinauer <stepan@openbios.org>
David L. Paktor <dlpaktor@us.ibm.com>
This manual page was written by Aurelien Jarno <aurel32@debian.org> for
the Debian GNU/Linux system (but may be used by others).