.\" Copyright 2013 Lars Wirzenius .\" .\" This program is free software: you can redistribute it and/or modify .\" it under the terms of the GNU General Public License as published by .\" the Free Software Foundation, either version 3 of the License, or .\" (at your option) any later version. .\" .\" This program is distributed in the hope that it will be useful, .\" but WITHOUT ANY WARRANTY; without even the implied warranty of .\" MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the .\" GNU General Public License for more details. .\" .\" You should have received a copy of the GNU General Public License .\" along with this program. If not, see . .\" .TH YARN 1 .SH NAME yarn \- scenario testing of Unix command line tools .SH SYNOPSIS .nh .B yarn .RB [ \-\-allow\-missing\-steps ] .RB [ \-\-no\-allow\-missing\-steps ] .RB [ \-\-cd\-datadir ] .RB [ \-\-no\-cd\-datadir ] .RB [ \-\-config\fR=\fIFILE ] .RB [ \-\-dump\-config ] .RB [ \-\-dump\-setting\-names ] .RB [ \-\-generate\-manpage\fR=\fITEMPLATE ] .RB [ \-h ] .RB [ \-\-help ] .RB [ \-\-help\-all ] .RB [ \-\-list\-config\-files ] .RB [ \-\-version ] .RB [ \-\-no\-default\-configs ] .RB [ \-\-dump\-memory\-profile\fR=\fIMETHOD ] .RB [ \-\-env\fR=\fINAME=VALUE ] .RB [ \-\-log\fR=\fIFILE ] .RB [ \-\-log\-keep\fR=\fIN ] .RB [ \-\-log\-level\fR=\fILEVEL ] .RB [ \-\-log\-max\fR=\fISIZE ] .RB [ \-\-log\-mode\fR=\fIMODE ] .RB [ \-\-memory\-dump\-interval\fR=\fISECONDS ] .RB [ \-\-output\fR=\fIFILE ] .RB [ \-q ] .RB [ \-\-quiet ] .RB [ \-\-no\-quiet ] .RB [ \-\-require\-assumptions ] .RB [ \-\-no\-require\-assumptions ] .RB [ \-r\fISCENARIO ] .RB [ \-\-run\fR=\fISCENARIO ] .RB [ \-\-shell\fR=\fISHELL ] .RB [ \-\-shell\-arg\fR=\fIARG ] .RB [ \-s\fISHELL\-LIBRARY ] .RB [ \-\-shell\-library\fR=\fISHELL\-LIBRARY ] .RB [ \-\-snapshot ] .RB [ \-\-no\-snapshot ] .RB [ \-\-stop\-on\-first\-fail ] .RB [ \-\-no\-stop\-on\-first\-fail ] .RB [ \-\-tempdir\fR=\fIDIR ] .RB [ \-\-timings ] .RB [ \-\-no\-timings ] .RB [ \-v ] .RB [ \-\-verbose ] .RB [ \-\-no\-verbose ] .RB [ \-n ] .RB [ \-\-no\-act ] .RB [ \-\-dry\-run ] .RB [ \-\-pretend ] .RB [ \-\-no\-no\-act ] .RB [ \-\-no\-dry\-run ] .RB [ \-\-no\-pretend ] .RI [ FILE ]... "" .hy .SH DESCRIPTION .B yarn is a scenario testing tool: you write a scenario describing how a user uses your software and what should happen, and express, using very lightweight syntax, the scenario in such a way that it can be tested automatically. The scenario has a simple, but strict structure: .IP .nf GIVEN some setup for the test WHEN thing that is to be tested happens THEN the post-conditions must be true .fi .PP As an example, consider a very short test scenario for verifying that a backup program works, at least for one simple case. .IP .nf SCENARIO backups can be restored GIVEN some live data in a directory AND an empty backup repository WHEN a backup is made THEN the data case be restored FINALLY cleanup .fi .PP Note the addition of AND: you can have multiple GIVEN, WHEN, and THEN statements. The AND keyword makes the text be more readable. SCENARIO is also necessary, and gives the title. .PP FINALLY is for cleanups. The FINALLY steps will be run regardless of whether the scenario succeeds or not. .PP Scenarios are meant to be written in somewhat human readable language. However, they are not free form text. In addition to the GIVEN/WHEN/THEN structure, the text for each of the steps needs a computer-executable implementation. This is done by using IMPLEMENTS. The backup scenario from above might be implemented as follows: .IP .nf IMPLEMENTS GIVEN some live data in a directory rm -rf "$DATADIR/data" mkdir "$DATADIR/data" echo foo > "$DATADIR/data/foo" .IP IMPLEMENTS GIVEN an empty backup repository rm -rf "$DATADIR/repo" mkdir "$DATADIR/repo" .IP IMPLEMENTS WHEN a backup is made backup-program -r "$DATADIR/repo" "$DATADIR/data" .IP IMPLEMENTS THEN the data can be restored mkdir "$DATADIR/restored" restore-program -r "$DATADIR/repo" "$DATADIR/restored" diff -rq "$DATADIR/data" "$DATADIR/restored" .IP IMPLEMENTS FINALLY cleanup echo nothing to do, actually .fi .PP Each "IMPLEMENTS GIVEN" (or WHEN, THEN, FINALLY) is followed by a regular expression on the same line, and then a shell script that gets executed to implement any step that matches the regular expression. The implementation can extract data from the match as well: for example, the regular expression might allow a file size to be specified. .PP The above example is a bit silly, of course: why go to the effort to obfuscate the various steps? The answer is that the various steps, implemented using IMPLEMENTS, can be combined in many ways, to test different aspects of the program being tested. .PP Moreover, by making the step descriptions be human language text, matched by regular expressions, most of the test can hopefully be written, and understood, by non-programmers. Someone who understands what a program should do, could write tests to verify its behaviour. The implementations of the various steps need to be implemented by a programmer, but given a well-designed set of steps, with enough flexibility in their implementation, that quite a good test suite can be written. .PP The shell commands in an IMPLEMENTS section are run in the directory in which the user ran .BR yarn . The environment variable .B SRCDIR is set to the fully qualified path to that directory. .SH OPTIONS .TP .BR \-\-allow\-missing\-steps allow scenarios to reference steps that do not exist, by warning about them, but otherwise ignoring the scenarios .TP .BR \-\-no\-allow\-missing\-steps opposite of --allow-missing-steps .TP .BR \-\-cd\-datadir change to DATADIR when running commands .TP .BR \-\-no\-cd\-datadir opposite of --cd-datadir .TP .BR \-\-env =\fINAME=VALUE add NAME=VALUE to the environment when tests are run .TP .BR \-\-generate\-manpage =\fITEMPLATE fill in manual page TEMPLATE .TP .BR \-h ", " \-\-help show this help message and exit .TP .BR \-\-output =\fIFILE write output to FILE, instead of standard output .TP .BR \-q ", " \-\-quiet be quiet, avoid progress reporting, only show errors .TP .BR \-\-no\-quiet opposite of --quiet .TP .BR \-\-require\-assumptions require ASSUMING to always pass .TP .BR \-\-no\-require\-assumptions opposite of --require-assumptions .TP .BR \-r ", " \-\-run =\fISCENARIO run only SCENARIO (this option can be repeated) .TP .BR \-\-shell =\fISHELL run IMPLEMENTS using SHELL .TP .BR \-\-shell\-arg =\fIARG use ARG when running shell .TP .BR \-s ", " \-\-shell\-library =\fISHELL-LIBRARY include a shell library for the IMPLEMENTS sections to use .TP .BR \-\-snapshot make snapshots of test working directory after each scenario step; you probably want to use this with --tempdir .TP .BR \-\-no\-snapshot opposite of --snapshot .TP .BR \-\-stop\-on\-first\-fail stop if any scenario step fails, don't run more scenarios .TP .BR \-\-no\-stop\-on\-first\-fail opposite of --stop-on-first-fail .TP .BR \-\-tempdir =\fIDIR use DIR as the temporary directory for tests; it should be empty or not exist .TP .BR \-\-timings report wall clock time for each scenario and step .TP .BR \-\-no\-timings opposite of --timings .TP .BR \-v ", " \-\-verbose make progress reporting be more verbose ("wall of text"), instead of a one-line status info; this is turned automatically if there is not terminal .TP .BR \-\-no\-verbose opposite of --verbose .TP .BR \-\-version show program's version number and exit .TP .BR \-n ", " \-\-no\-act ", " \-\-dry\-run ", " \-\-pretend do not actually run any tests, merely print what would be run .TP .BR \-\-no\-no\-act ", " \-\-no\-dry\-run ", " \-\-no\-pretend opposite of --no-act .SS "Configuration files and settings" .TP .BR \-\-config =\fIFILE add FILE to config files .TP .BR \-\-dump\-config write out the entire current configuration .TP .BR \-\-dump\-setting\-names write out all names of settings and quit .TP .BR \-\-help\-all show all options .TP .BR \-\-list\-config\-files list all possible config files .TP .BR \-\-no\-default\-configs clear list of configuration files to read .SS "Logging" .TP .BR \-\-log =\fIFILE write log entries to FILE (default is to not write log files at all); use "syslog" to log to system log, "stderr" to log to the standard error output, or "none" to disable logging .TP .BR \-\-log\-keep =\fIN keep last N logs (10) .TP .BR \-\-log\-level =\fILEVEL log at LEVEL, one of debug, info, warning, error, critical, fatal (default: debug) .TP .BR \-\-log\-max =\fISIZE rotate logs larger than SIZE, zero for never (default: 0) .TP .BR \-\-log\-mode =\fIMODE set permissions of new log files to MODE (octal; default 0600) .SS "Peformance" .TP .BR \-\-dump\-memory\-profile =\fIMETHOD make memory profiling dumps using METHOD, which is one of: none, or simple (no meliae support anymore)(default: simple) .TP .BR \-\-memory\-dump\-interval =\fISECONDS make memory profiling dumps at least SECONDS apart .SH ENVIRONMENT .TP .B DATADIR Fully qualified pathname to a temporary directory, in which the tests can use files. The temporary directory is removed at the end of the test execution, unless the user specifies otherwise with \-\-snapshot. .TP .B SRCDIR Fully qualitifed pathname to the directory in which the user ran .BR yarn . This is useful when the tests want to change the directory. .SH EXAMPLE To run .B yarn on all the scenarios in your current directory: .IP .nf yarn *.scenario .fi .PP All the files will be treated together as if they had been one file. .PP To add a shell library to be included when running any IMPLEMENTS section: .IP .nf yarn \-\-shell\-library mylib.sh *.scenario .fi .PP You can repeat .B \-\-shell\-library as many times as necessary. .SH "SEE ALSO" .BR cmdtest (1), .BR cliapp (5). .PP The README.yarn file has more details on the scenario testing language.