NAME¶
docstrip_util - Docstrip-related utilities
SYNOPSIS¶
package require
Tcl 8.4
package require
docstrip ?1.2?
package require
docstrip::util ?1.3?
pkgProvide name version terminals
pkgIndex ?
terminal ...?
fileoptions ?
option value ...?
docstrip::util::index_from_catalogue dir pattern
?
option value ...?
docstrip::util::modules_from_catalogue target source
?
option value ...?
docstrip::util::classical_preamble metaprefix message
target ?
source terminals ...?
docstrip::util::classical_postamble metaprefix message
target ?
source terminals ...?
docstrip::util::packages_provided text ?
setup-script?
docstrip::util::ddt2man text
docstrip::util::guards subcmd text
docstrip::util::patch source-var terminals fromtext
diff ?
option value ...?
docstrip::util::thefile filename ?
option value ...?
docstrip::util::import_unidiff diff-text ?
warning-var?
DESCRIPTION¶
The
docstrip::util package is meant for collecting various utility
procedures that are mainly useful at installation or development time. It is
separate from the base package to avoid overhead when the latter is used to
source code.
PACKAGE INDEXING COMMANDS¶
Like raw "
.tcl" files, code lines in docstrip source files can
be searched for package declarations and corresponding indices constructed. A
complication is however that one cannot tell from the code blocks themselves
which will fit together to make a working package; normally that information
would be found in an accompanying "
.ins" file, but parsing
one of those is not an easy task. Therefore
docstrip::util introduces
an alternative encoding of such information, in the form of a declarative Tcl
script: the
catalogue (of the contents in a source file).
The special commands which are available inside a catalogue are:
- pkgProvide name version
terminals
- Declares that the code for a package with name name
and version version is made up from those modules in the source
file which are selected by the terminals list of guard expression
terminals. This code should preferably not contain a package
provide command for the package, as one will be provided by the
package loading mechanisms.
- pkgIndex ?terminal ...?
- Declares that the code for a package is made up from those
modules in the source file which are selected by the listed guard
expression terminals. The name and version of this package is
determined from package provide command(s) found in that
code (hence there must be such a command in there).
- fileoptions ?option value ...?
- Declares the fconfigure options that should be in
force when reading the source; this can usually be ignored for pure ASCII
files, but if the file needs to be interpreted according to some other
-encoding then this is how to specify it. The command should
normally appear first in the catalogue, as it takes effect only for
commands following it.
Other Tcl commands are supported too ? a catalogue is parsed by being evaluated
in a safe interpreter ? but they are rarely needed. To allow for future
extensions, unknown commands in the catalogue are silently ignored.
To simplify distribution of catalogues together with their source files, the
catalogue is stored
in the source file itself as a module selected by
the terminal '
docstrip.tcl::catalogue'. This supports both the style
of collecting all catalogue lines in one place and the style of putting each
catalogue line in close proximity of the code that it declares.
Putting catalogue entries next to the code they declare may look as follows
% First there's the catalogue entry
% \begin{tcl}
%<docstrip.tcl::catalogue>pkgProvide foo::bar 1.0 {foobar load}
% \end{tcl}
% second a metacomment used to include a copyright message
% \begin{macrocode}
%<*foobar>
%% This file is placed in the public domain.
% \end{macrocode}
% third the package implementation
% \begin{tcl}
namespace eval foo::bar {
# ... some clever piece of Tcl code elided ...
% \end{tcl}
% which at some point may have variant code to make use of a
% |load|able extension
% \begin{tcl}
%<*load>
load [file rootname [info script]][info sharedlibextension]
%</load>
%<*!load>
# ... even more clever scripted counterpart of the extension
# also elided ...
%</!load>
}
%</foobar>
% \end{tcl}
% and that's it!
The corresponding set-up with
pkgIndex would be
% First there's the catalogue entry
% \begin{tcl}
%<docstrip.tcl::catalogue>pkgIndex foobar load
% \end{tcl}
% second a metacomment used to include a copyright message
% \begin{tcl}
%<*foobar>
%% This file is placed in the public domain.
% \end{tcl}
% third the package implementation
% \begin{tcl}
package provide foo::bar 1.0
namespace eval foo::bar {
# ... some clever piece of Tcl code elided ...
% \end{tcl}
% which at some point may have variant code to make use of a
% |load|able extension
% \begin{tcl}
%<*load>
load [file rootname [info script]][info sharedlibextension]
%</load>
%<*!load>
# ... even more clever scripted counterpart of the extension
# also elided ...
%</!load>
}
%</foobar>
% \end{tcl}
% and that's it!
- docstrip::util::index_from_catalogue dir
pattern ? option value ...?
- This command is a sibling of the standard
pkg_mkIndex command, in that it adds package entries to "
pkgIndex.tcl" files. The difference is that it indexes
docstrip-style source files rather than raw "
.tcl" or loadable library files. Only packages listed in the
catalogue of a file are considered.
The dir argument is the directory in which to look for files (and
whose " pkgIndex.tcl" file should be amended). The
pattern argument is a glob pattern of files to look into; a
typical value would be *.dtx or *.{dtx,ddt}. Remaining
arguments are option-value pairs, where the supported options are:
- -recursein dirpattern
- If this option is given, then the
index_from_catalogue operation will be repeated in each
subdirectory whose name matches the dirpattern. -recursein
* will cause the entire subtree rooted at dir to be
indexed.
- -sourceconf dictionary
- Specify fileoptions to use when reading the
catalogues of files (and also for reading the packages if the catalogue
does not contain a fileoptions command). Defaults to being empty.
Primarily useful if your system encoding is very different from that of
the source file (e.g., one is a two-byte encoding and the other is a
one-byte encoding). ascii and utf-8 are not very different
in that sense.
- -options terminals
- The terminals is a list of terminals in addition to
docstrip.tcl::catalogue that should be held as true when extracting
the catalogue. Defaults to being empty. This makes it possible to make use
of "variant sections" in the catalogue itself, e.g. gaurd some
entries with an extra "experimental" and thus prevent them from
appearing in the index unless that is generated with
"experimental" among the -options.
- -report boolean
- If the boolean is true then the return value will be
a textual, probably multiline, report on what was done. Defaults to false,
in which case there is no particular return value.
- -reportcmd commandPrefix
- Every item in the report is handed as an extra argument to
the command prefix. Since index_from_catalogue would typically be
used at a rather high level in installation scripts and the like, the
commandPrefix defaults to " puts stdout".
Use list to effectively disable this feature. The return values
from the prefix are ignored.
- The package ifneeded scripts that are generated
contain one package require docstrip command and one
docstrip::sourcefrom command. If the catalogue entry was of the
pkgProvide kind then the package ifneeded script also
contains the package provide command.
Note that index_from_catalogue never removes anything from an
existing " pkgIndex.tcl" file. Hence you may need to
delete it (or have pkg_mkIndex recreate it from scratch) before
running index_from_catalogue to update some piece of information,
such as a package version number.
- docstrip::util::modules_from_catalogue target
source ?option value ...?
- This command is an alternative to
index_from_catalogue which creates Tcl Module ("
.tm") files rather than " pkgIndex.tcl"
entries. Since this action is more similar to what docstrip
classically does, it has features for putting pre- and postambles on the
generated files.
The source argument is the name of the source file to generate "
.tm" files from. The target argument is the directory
which should count as a module path, i.e., this is what the relative paths
derived from package names are joined to. The supported options are:
- -preamble message
- A message to put in the preamble (initial block of
comments) of generated files. Defaults to a space. May be several lines,
which are then separated by newlines. Traditionally used for copyright
notices or the like, but metacomment lines provide an alternative to
that.
- -postamble message
- Like -preamble, but the message is put at the end of
the file instead of the beginning. Defaults to being empty.
- -sourceconf dictionary
- Specify fileoptions to use when reading the
catalogue of the source (and also for reading the packages if the
catalogue does not contain a fileoptions command). Defaults to
being empty. Primarily useful if your system encoding is very different
from that of the source file (e.g., one is a two-byte encoding and the
other is a one-byte encoding). ascii and utf-8 are not very
different in that sense.
- -options terminals
- The terminals is a list of terminals in addition to
docstrip.tcl::catalogue that should be held as true when extracting
the catalogue. Defaults to being empty. This makes it possible to make use
of "variant sections" in the catalogue itself, e.g. gaurd some
entries with an extra "experimental" guard and thus prevent them
from contributing packages unless those are generated with
"experimental" among the -options.
- -formatpreamble commandPrefix
- Command prefix used to actually format the preamble. Takes
four additional arguments message, targetFilename,
sourceFilename, and terminalList and returns a fully
formatted preamble. Defaults to using classical_preamble with a
metaprefix of '##'.
- -formatpostamble commandPrefix
- Command prefix used to actually format the postamble. Takes
four additional arguments message, targetFilename,
sourceFilename, and terminalList and returns a fully
formatted postamble. Defaults to using classical_postamble with a
metaprefix of '##'.
- -report boolean
- If the boolean is true (which is the default) then
the return value will be a textual, probably multiline, report on what was
done. If it is false then there is no particular return value.
- -reportcmd commandPrefix
- Every item in the report is handed as an extra argument to
this command prefix. Defaults to list, which effectively disables
this feature. The return values from the prefix are ignored. Use for
example " puts stdout" to get report items written
immediately to the terminal.
- An existing file of the same name as one to be created will
be overwritten.
- docstrip::util::classical_preamble metaprefix
message target ?source terminals ...?
- This command returns a preamble in the classical
docstrip style
##
## This is `TARGET',
## generated by the docstrip::util package.
##
## The original source files were:
##
## SOURCE (with options: `foo,bar')
##
## Some message line 1
## line2
## line3
- if called as
docstrip::util::classical_preamble {##}\
"\nSome message line 1\nline2\nline3" TARGET SOURCE {foo bar}
- The command supports preambles for files generated from
multiple sources, even though modules_from_catalogue at present
does not need that.
- docstrip::util::classical_postamble
metaprefix message target ?source
terminals ...?
- This command returns a postamble in the classical
docstrip style
## Some message line 1
## line2
## line3
##
## End of file `TARGET'.
- if called as
docstrip::util::classical_postamble {##}\
"Some message line 1\nline2\nline3" TARGET SOURCE {foo bar}
- In other words, the source and terminals
arguments are ignored, but supported for symmetry with
classical_preamble.
- docstrip::util::packages_provided text
?setup-script?
- This command returns a list where every even index element
is the name of a package provided by text when that is
evaluated as a Tcl script, and the following odd index element is the
corresponding version. It is used to do package indexing of extracted
pieces of code, in the manner of pkg_mkIndex.
One difference to pkg_mkIndex is that the text gets evaluated
in a safe interpreter. package require commands are silently
ignored, as are unknown commands (which includes source and
load). Other errors cause processing of the text to stop, in
which case only those package declarations that had been encountered
before the error will be included in the return value.
The setup-script argument can be used to customise the evaluation
environment, if the code in text has some very special needs. The
setup-script is evaluated in the local context of the
packages_provided procedure just before the text is
processed. At that time, the name of the slave command for the safe
interpreter that will do this processing is kept in the local variable
c. To for example copy the contents of the ::env array to
the safe interpreter, one might use a setup-script of
$c eval [list array set env [array get ::env]]
SOURCE PROCESSING COMMANDS¶
Unlike the previous group of commands, which would use
docstrip::extract
to extract some code lines and then process those further, the following
commands operate on text consisting of all types of lines.
- docstrip::util::ddt2man text
- The ddt2man command reformats text from the
general docstrip format to doctools ".man"
format (Tcl Markup Language for Manpages). The different line types are
treated as follows:
- comment and metacomment lines
- The '%' and '%%' prefixes are removed, the rest of the text
is kept as it is.
- empty lines
- These are kept as they are. (Effectively this means that
they will count as comment lines after a comment line and as code lines
after a code line.)
- code lines
- example_begin and example_end commands are
placed at the beginning and end of every block of consecutive code lines.
Brackets in a code line are converted to lb and rb
commands.
- verbatim guards
- These are processed as usual, so they do not show up in the
result but every line in a verbatim block is treated as a code line.
- other guards
- These are treated as code lines, except that the actual
guard is emphasised.
- At the time of writing, no project has employed
doctools markup in master source files, so experience of what works
well is not available. A source file could however look as follows
% [manpage_begin gcd n 1.0]
% [moddesc {Greatest Common Divisor}]
% [require gcd [opt 1.0]]
% [description]
%
% [list_begin definitions]
% [call [cmd gcd] [arg a] [arg b]]
% The [cmd gcd] procedure takes two arguments [arg a] and [arg b] which
% must be integers and returns their greatest common divisor.
proc gcd {a b} {
% The first step is to take the absolute values of the arguments.
% This relieves us of having to worry about how signs will be treated
% by the remainder operation.
set a [expr {abs($a)}]
set b [expr {abs($b)}]
% The next line does all of Euclid's algorithm! We can make do
% without a temporary variable, since $a is substituted before the
% [lb]set a $b[rb] and thus continues to hold a reference to the
% "old" value of [var a].
while {$b>0} { set b [expr { $a % [set a $b] }] }
% In Tcl 8.3 we might want to use [cmd set] instead of [cmd return]
% to get the slight advantage of byte-compilation.
%<tcl83> set a
%<!tcl83> return $a
}
% [list_end]
%
% [manpage_end]
- If the above text is fed through
docstrip::util::ddt2man then the result will be a syntactically
correct doctools manpage, even though its purpose is a bit
different.
It is suggested that master source code files with doctools markup
are given the suffix " .ddt", hence the "ddt"
in ddt2man.
- docstrip::util::guards subcmd
text
- The guards command returns information (mostly of a
statistical nature) about the ordinary docstrip guards that occur in the
text. The subcmd selects what is returned.
- counts
- List the guard expression terminals with counts. The format
of the return value is a dictionary which maps the terminal name to the
number of occurencies of it in the file.
- exprcount
- List the guard expressions with counts. The format of the
return value is a dictionary which maps the expression to the number of
occurencies of it in the file.
- exprerr
- List the syntactically incorrect guard expressions (e.g.
parentheses do not match, or a terminal is missing). The return value is a
list, with the elements in no particular order.
- expressions
- List the guard expressions. The return value is a list,
with the elements in no particular order.
- exprmods
- List the guard expressions with modifiers. The format of
the return value is a dictionary where each index is a guard expression
and each entry is a string with one character for every guard line that
has this expression. The characters in the entry specify what modifier was
used in that line: +, -, *, /, or (for guard without modifier:) space.
This is the most primitive form of the information gathered by
guards.
- names
- List the guard expression terminals. The return value is a
list, with the elements in no particular order.
- rotten
- List the malformed guard lines (this does not include lines
where only the expression is malformed, though). The format of the return
value is a dictionary which maps line numbers to their contents.
- docstrip::util::patch source-var
terminals fromtext diff ?option value
...?
- This command tries to apply a diff file (for example
a contributed patch) that was computed for a generated file to the
docstrip source. This can be useful if someone has edited a
generated file, thus mistaking it for being the source. This command makes
no presumptions which are specific for the case that the generated file is
a Tcl script.
patch requires that the source file to patch is kept as a list of
lines in a variable, and the name of that variable in the calling context
is what goes into the source-var argument. The terminals is
the list of terminals used to extract the file that has been patched. The
diff is the actual diff to apply (in a format as explained below)
and the fromtext is the contents of the file which served as
"from" when the diff was computed. Options can be used to
further control the process.
The process works by "lifting" the hunks in the diff from
generated to source file, and then applying them to the elements of the
source-var. In order to do this lifting, it is necessary to
determine how lines in the fromtext correspond to elements of the
source-var, and that is where the terminals come in; the
source is first extracted under the given terminals, and the
result of that is then matched against the fromtext. This produces
a map which translates line numbers stated in the diff to element
numbers in source-var, which is what is needed to lift the hunks.
The reason that both the terminals and the fromtext must be
given is twofold. First, it is very difficult to keep track of how many
lines of preamble are supplied some other way than by copying lines from
source files. Second, a generated file might contain material from several
source files. Both make it impossible to predict what line number an
extracted file would have in the generated file, so instead the algorithm
for computing the line number map looks for a block of lines in the
fromtext which matches what can be extracted from the source. This
matching is affected by the following options:
- -matching mode
- How equal must two lines be in order to match? The
supported modes are:
- exact
- Lines must be equal as strings. This is the default.
- anyspace
- All sequences of whitespace characters are converted to
single spaces before comparing.
- nonspace
- Only non-whitespace characters are considered when
comparing.
- none
- Any two lines are considered to be equal.
- -metaprefix string
- The -metaprefix value to use when extracting.
Defaults to "%%", but for Tcl code it is more likely that
"#" or "##" had been used for the generated file.
- -trimlines boolean
- The -trimlines value to use when extracting.
Defaults to true.
- The return value is in the form of a unified diff,
containing only those hunks which were not applied or were only partially
applied; a comment in the header of each hunk specifies which case is at
hand. It is normally necessary to manually review both the return value
from patch and the patched text itself, as this command cannot
adjust comment lines to match new content.
An example use would look like
set sourceL [split [docstrip::util::thefile from.dtx] \n]
set terminals {foo bar baz}
set fromtext [docstrip::util::thefile from.tcl]
set difftext [exec diff --unified from.tcl to.tcl]
set leftover [docstrip::util::patch sourceL $terminals $fromtext\
[docstrip::util::import_unidiff $difftext] -metaprefix {#}]
set F [open to.dtx w]; puts $F [join $sourceL \n]; close $F
return $leftover
- Here, "from.dtx" was used as source for
" from.tcl", which someone modified into "
to.tcl". We're trying to construct a "
to.dtx" which can be used as source for "
to.tcl".
- docstrip::util::thefile filename
?option value ...?
- The thefile command opens the file filename,
reads it to end, closes it, and returns the contents (dropping a final
newline if there is one). The option-value pairs are passed on to
fconfigure to configure the open file channel before anything is
read from it.
- docstrip::util::import_unidiff diff-text
?warning-var?
- This command parses a unified (diff flags -U
and --unified) format diff into the list-of-hunks format expected
by docstrip::util::patch. The diff-text argument is the text
to parse and the warning-var is, if specified, the name in the
calling context of a variable to which any warnings about parsing problems
will be appended.
The return value is a list of hunks. Each hunk is a list of five
elements " start1 end1 start2 end2
lines". start1 and end1 are line numbers in the
"from" file of the first and last respectively lines of the
hunk. start2 and end2 are the corresponding line numbers in
the "to" file. Line numbers start at 1. The lines is a
list with two elements for each line in the hunk; the first specifies the
type of a line and the second is the actual line contents. The type is
- for lines only in the "from" file, + for lines
that are only in the "to" file, and 0 for lines that are
in both.
SEE ALSO¶
docstrip, doctools, doctools_fmt
KEYWORDS¶
.ddt, Tcl module, catalogue, diff, docstrip, doctools, documentation, literate
programming, module, package indexing, patch, source
CATEGORY¶
Documentation tools
COPYRIGHT¶
Copyright (c) 2003?2010 Lars Hellström <Lars dot Hellstrom at residenset dot net>