.\" Automatically generated by Pod::Man 4.10 (Pod::Simple 3.35) .\" .\" Standard preamble: .\" ======================================================================== .de Sp \" Vertical space (when we can't use .PP) .if t .sp .5v .if n .sp .. .de Vb \" Begin verbatim text .ft CW .nf .ne \\$1 .. .de Ve \" End verbatim text .ft R .fi .. .\" Set up some character translations and predefined strings. \*(-- will .\" give an unbreakable dash, \*(PI will give pi, \*(L" will give a left .\" double quote, and \*(R" will give a right double quote. \*(C+ will .\" give a nicer C++. Capital omega is used to do unbreakable dashes and .\" therefore won't be available. \*(C` and \*(C' expand to `' in nroff, .\" nothing in troff, for use with C<>. .tr \(*W- .ds C+ C\v'-.1v'\h'-1p'\s-2+\h'-1p'+\s0\v'.1v'\h'-1p' .ie n \{\ . ds -- \(*W- . ds PI pi . if (\n(.H=4u)&(1m=24u) .ds -- \(*W\h'-12u'\(*W\h'-12u'-\" diablo 10 pitch . if (\n(.H=4u)&(1m=20u) .ds -- \(*W\h'-12u'\(*W\h'-8u'-\" diablo 12 pitch . ds L" "" . ds R" "" . ds C` "" . ds C' "" 'br\} .el\{\ . ds -- \|\(em\| . ds PI \(*p . ds L" `` . ds R" '' . ds C` . ds C' 'br\} .\" .\" Escape single quotes in literal strings from groff's Unicode transform. .ie \n(.g .ds Aq \(aq .el .ds Aq ' .\" .\" If the F register is >0, we'll generate index entries on stderr for .\" titles (.TH), headers (.SH), subsections (.SS), items (.Ip), and index .\" entries marked with X<> in POD. Of course, you'll have to process the .\" output yourself in some meaningful fashion. .\" .\" Avoid warning from groff about undefined register 'F'. .de IX .. .nr rF 0 .if \n(.g .if rF .nr rF 1 .if (\n(rF:(\n(.g==0)) \{\ . if \nF \{\ . de IX . tm Index:\\$1\t\\n%\t"\\$2" .. . if !\nF==2 \{\ . nr % 0 . nr F 2 . \} . \} .\} .rr rF .\" .\" Accent mark definitions (@(#)ms.acc 1.5 88/02/08 SMI; from UCB 4.2). .\" Fear. Run. Save yourself. No user-serviceable parts. . \" fudge factors for nroff and troff .if n \{\ . ds #H 0 . ds #V .8m . ds #F .3m . ds #[ \f1 . ds #] \fP .\} .if t \{\ . ds #H ((1u-(\\\\n(.fu%2u))*.13m) . ds #V .6m . ds #F 0 . ds #[ \& . ds #] \& .\} . \" simple accents for nroff and troff .if n \{\ . ds ' \& . ds ` \& . ds ^ \& . ds , \& . ds ~ ~ . ds / .\} .if t \{\ . ds ' \\k:\h'-(\\n(.wu*8/10-\*(#H)'\'\h"|\\n:u" . ds ` \\k:\h'-(\\n(.wu*8/10-\*(#H)'\`\h'|\\n:u' . ds ^ \\k:\h'-(\\n(.wu*10/11-\*(#H)'^\h'|\\n:u' . ds , \\k:\h'-(\\n(.wu*8/10)',\h'|\\n:u' . ds ~ \\k:\h'-(\\n(.wu-\*(#H-.1m)'~\h'|\\n:u' . ds / \\k:\h'-(\\n(.wu*8/10-\*(#H)'\z\(sl\h'|\\n:u' .\} . \" troff and (daisy-wheel) nroff accents .ds : \\k:\h'-(\\n(.wu*8/10-\*(#H+.1m+\*(#F)'\v'-\*(#V'\z.\h'.2m+\*(#F'.\h'|\\n:u'\v'\*(#V' .ds 8 \h'\*(#H'\(*b\h'-\*(#H' .ds o \\k:\h'-(\\n(.wu+\w'\(de'u-\*(#H)/2u'\v'-.3n'\*(#[\z\(de\v'.3n'\h'|\\n:u'\*(#] .ds d- \h'\*(#H'\(pd\h'-\w'~'u'\v'-.25m'\f2\(hy\fP\v'.25m'\h'-\*(#H' .ds D- D\\k:\h'-\w'D'u'\v'-.11m'\z\(hy\v'.11m'\h'|\\n:u' .ds th \*(#[\v'.3m'\s+1I\s-1\v'-.3m'\h'-(\w'I'u*2/3)'\s-1o\s+1\*(#] .ds Th \*(#[\s+2I\s-2\h'-\w'I'u*3/5'\v'-.3m'o\v'.3m'\*(#] .ds ae a\h'-(\w'a'u*4/10)'e .ds Ae A\h'-(\w'A'u*4/10)'E . \" corrections for vroff .if v .ds ~ \\k:\h'-(\\n(.wu*9/10-\*(#H)'\s-2\u~\d\s+2\h'|\\n:u' .if v .ds ^ \\k:\h'-(\\n(.wu*10/11-\*(#H)'\v'-.4m'^\v'.4m'\h'|\\n:u' . \" for low resolution devices (crt and lpr) .if \n(.H>23 .if \n(.V>19 \ \{\ . ds : e . ds 8 ss . ds o a . ds d- d\h'-1'\(ga . ds D- D\h'-1'\(hy . ds th \o'bp' . ds Th \o'LP' . ds ae ae . ds Ae AE .\} .rm #[ #] #H #V #F C .\" ======================================================================== .\" .IX Title "TESH 1" .TH TESH 1 "2018-11-22" "perl v5.28.0" "User Contributed Perl Documentation" .\" For nroff, turn off justification. Always turn off hyphenation; it makes .\" way too many mistakes in technical documents. .if n .ad l .nh .SH "NAME" tesh \-\- testing shell .SH "SYNOPSIS" .IX Header "SYNOPSIS" \&\fBtesh\fR [\fIoptions\fR]... \fItestsuite\fR .SH "DESCRIPTION" .IX Header "DESCRIPTION" Tesh is the testing shell, a specialized shell for running tests. It provides the specified input to the tested commands, and check that they produce the expected output and return the expected value. .SH "OPTIONS" .IX Header "OPTIONS" .Vb 8 \& \-\-help : display help \& \-\-cd some/directory : ask tesh to switch the working directory before \& launching the tests \& \-\-setenv var=value : set a specific environment variable \& \-\-cfg arg : add parameter \-\-cfg=arg to each command line \& \-\-log arg : add parameter \-\-log=arg to each command line \& \-\-ignore\-jenkins : ignore all cruft generated on SimGrid \& continuous integration servers .Ve .SH "TEST SUITE FILE SYTAX" .IX Header "TEST SUITE FILE SYTAX" A test suite is composed of one or several \fIcommand blocks\fR separated by empty lines, each of them being composed of a command to run, its input text and the expected output. .PP The first char of each line specifies the type of line according to the following list. The second char of each line is ignored. .PP .Vb 2 \& \`$\*(Aq command to run in foreground \& \`&\*(Aq command to run in background \& \& \`<\*(Aq input to pass to the command \& \`>\*(Aq output expected from the command \& \& \`!\*(Aq metacommand, which can be one of: \& \`timeout\*(Aq |no \& \`expect signal\*(Aq \& \`expect return\*(Aq \& \`output\*(Aq \& \`output sort\*(Aq [integer] \& \`setenv =\*(Aq \& \`ignore\*(Aq \& \& \`p\*(Aq an informative message to print \& \`#\*(Aq a comment .Ve .PP If the expected output do not match the produced output, or if the command did not end as expected, Tesh provides an error message (see the \s-1OUTPUT\s0 section below) and stops. .SS "Command blocks examples" .IX Subsection "Command blocks examples" In a given command block, you can declare the command, its input and its expected output in the order that you see fit. .PP .Vb 3 \& $ cat \& < TOTO \& > TOTO \& \& > TOTO \& $ cat \& < TOTO \& \& > TOTO \& < TOTO \& $ cat .Ve .PP You can group several commands together, provided that they don't have any input nor output. .PP .Vb 2 \& $ mkdir testdir \& $ cd testdir .Ve .SS "Enforcing the command return code" .IX Subsection "Enforcing the command return code" By default, Tesh enforces that the tested command returns 0. If not, it fails with an appropriate message and returns \fIcode+40\fR itself. .PP You specify that a given command block is expected to return another code as follows: .PP .Vb 3 \& # This command MUST return 42 \& ! expect return 42 \& $ sh \-e "exit 42" .Ve .PP The \fIexpect return\fR construct applies only to the next command block. .SS "Commands that are expected to raise signals" .IX Subsection "Commands that are expected to raise signals" By default, Tesh detects when the command is killed by a signal (such as \s-1SEGV\s0 on segfaults). This is usually unexpected and unfortunate. But if not, you can specify that a given command block is expected to fail with a signal as follows: .PP .Vb 3 \& # This command MUST raise a segfault \& ! expect signal SIGSEGV \& $ ./some_failing_code .Ve .PP The \fIexpect signal\fR construct applies only to the next command block. .SS "Timeouts" .IX Subsection "Timeouts" By default, no command is allowed to run more than 10 seconds. You can change this value as follows: .PP .Vb 3 \& # Allow some more time to the command \& ! timeout 60 \& $ ./some_longer_command .Ve .PP You can also disable the timeout completely by passing \*(L"no\*(R" as a value: .PP .Vb 3 \& # This command will never timeout \& ! timeout no \& $ ./some_very_long_but_safe_command .Ve .SS "Setting environment variables" .IX Subsection "Setting environment variables" You can modify the environment of the tested commands as follows: .PP .Vb 2 \& ! setenv PATH=/bin \& $ my_command .Ve .PP You can also set an envirmnent variable from the command line: .PP .Vb 1 \& tesh \-\-setenv bindir=/opt/bin/ .Ve .PP And then use it within the tesh file: .PP .Vb 1 \& $ ${bindir}/myprogram .Ve .PP Tesh also supports perl default value for undefined variables: .PP .Vb 1 \& $ ${bindir:=/usr/bin}/myprogram .Ve .SS "Not enforcing the expected output" .IX Subsection "Not enforcing the expected output" By default, the commands output is matched against the one expected, and an error is raised on discrepancy. Metacommands to change this: .IP "output ignore" 4 .IX Item "output ignore" The output is completely discarded. .IP "output display" 4 .IX Item "output display" The output is displayed, but no error is issued if it differs from the expected output. .IP "output sort" 4 .IX Item "output sort" The output and the expected output are sorted before comparison (see next section). .SS "Sorting output" .IX Subsection "Sorting output" If the order of the command output changes between runs, you want to sort it before enforcing that it is exactly what you expect. In SimGrid for example, this happens when parallel execution is activated: User processes are run in parallel at each timestamp, and the output is not reproducible anymore. Until you sort the lines. .PP You can sort the command output as follows: .PP .Vb 2 \& ! output sort \& $ ./some_multithreaded_command .Ve .PP Sorting lines this ways often makes the tesh output very intricate, complicating the error analysis: the process logical order is defeated by the lexicographical sort. .PP The solution is to prefix each line of your output with temporal information so that lines can be grouped by timestamps. The lexicographical sort then only applies to lines that occurred at the same timestamp. Here is a SimGrid example: .PP .Vb 3 \& # Sort only lines depending on the first 19 chars \& ! output sort 19 \& $ ./some_simgrid_simulator \-\-log=root.fmt:[%10.6r]%e(%i:%P@%h)%e%m%n .Ve .PP This approach may seem surprizing at the first glance but it does its job: .IP "Every timestamps remain separated, as it should;" 4 .IX Item "Every timestamps remain separated, as it should;" .PD 0 .IP "In each timestamp, the output order of processes become reproducible: that's the lexicographical order of their name;" 4 .IX Item "In each timestamp, the output order of processes become reproducible: that's the lexicographical order of their name;" .IP "For each process, the order of its execution is preserved: its messages within a given timestamp are not reordered." 4 .IX Item "For each process, the order of its execution is preserved: its messages within a given timestamp are not reordered." .PD .PP That way, tesh can do its job (no false positive, no false negative) despite the unpredictable order of executions of processes within a timestamp, and reported errors remain easy to analyze (execution of a given process preserved). .PP This example is very SimGrid oriented, but the feature could even be usable by others, who knows? .SS "Ignoring some output" .IX Subsection "Ignoring some output" Some outputed lines can be ignored by setting the ignore command followed by a regular expression: .PP .Vb 4 \& ! ignore .*0x[0\-9A\-F]+\e. \& $ printf \*(Aqword\enMemory address: 0x42AA42.\enanotherword\en\*(Aq \& > word \& > anotherword .Ve .SS "Colored and formatted text" .IX Subsection "Colored and formatted text" Tesh removes \s-1ANSI/VT100\s0 control sequences from outputed text to make easier the writing of tests. .PP .Vb 2 \& $ printf "I \e033[0;31mlove\e033[0m tesh\en" \& > I love tesh .Ve .SH "BUILTIN COMMANDS" .IX Header "BUILTIN COMMANDS" .SS "mkfile: creating a file" .IX Subsection "mkfile: creating a file" This command creates a file of the name provided as argument, and adds the content it gets as input. .PP .Vb 3 \& $ mkfile myFile \& > some content \& > to the file .Ve .PP It is not possible to use the cat command, as one would expect, because stream redirections are currently not implemented in Tesh. .SH "BUGS, LIMITATIONS AND POSSIBLE IMPROVEMENTS" .IX Header "BUGS, LIMITATIONS AND POSSIBLE IMPROVEMENTS" The main limitation is the lack of stream redirections in the commands (\*(L">\*(R", \*(L"<\*(R" and \*(L"|\*(R" shell constructs and friends). The \fBmkfile\fR builtin command makes this situation bearable. .PP It would be nice if we could replace the tesh file completely with command line flags when the output is not to be verified.