NAME¶
Lintian::Tutorial::TestSuite -- Quick intro to running the Lintian testsuite
SYNOPSIS¶
This guide will quickly introduce you to running the Lintian test suite and some
tricks. The Lintian test suite is fairly large and accordingly it can take a
substantial amount of time to run. To speed up development, there are various
options to limit the tests to run.
If you are looking for a guide on how to write tests, please consult
Lintian::Tutorial::WritingTests.
DESCRIPTION¶
The Lintian test suite is an extensive collection of various test cases. The
test suite is divided into 4 "sub-suites". The majority of tests are
currently located in the "tests" sub-suite.
To run the full suite in all its glory, simply invoke:
$ debian/rules runtests
OR
$ mkdir -p debian/test-out
$ t/runtests -k --dump-logs t debian/test-out
While writing a new tag (or check) you probably only want to run a particular
(subset of the) test(s). See "Running a subset of the tests" for the
available options.
When run via
debian/rules, the test suite respects
"DEB_BUILD_OPTIONS=parallel=N". When using
t/runtests
directly, use
-jN to choose the number of threads. Note that
"N" denotes the amount of "worker" threads. The test
runner will generally have 2 threads more than that. Also each
"worker" will run lintian, which runs multiple unpacking jobs in
parallel as well.
In case you are wondering about the 2 extra threads in the test runner, the
first of them is the "coordinator" thread (which will generally be
waiting when the workers are active) and the second one is the
"output" thread (which handles the fancy output).
Running a subset of the tests¶
The following options are available:
- Running a single test
- To run a single test by its name, use:
$ debian/rules runtests onlyrun=$name
OR
$ t/runtests --dump-logs t debian/test-out $name
- Running all tests for a check
- To run all tests for a given check, use:
$ debian/rules runtests onlyrun=$check
OR
$ t/runtests --dump-logs -k t debian/test-out $check
$check must be the name of a check (it will test for checks/$check.desc) or
"legacy". This will run all tests that start with
"$check-".
Note: The "changes" sub-suite in the new test suite does not
support this.
- Running all tests in a sub-suite
- To run all tests in a given sub-suite, use:
$ debian/rules runtests onlyrun=suite:$suites
OR
$ t/runtests --dump-logs -k t debian/test-out suite:$suites
$suites is a comma-separated list of names of sub-suites to run.
Note: this cannot be used to influence the order in which the sub-suites are
run.
- Running all tests designed for a specific tag
- To run all tests that have a "Test-For" or a
"Test-Against" for a given tag, use:
$ debian/rules runtests onlyrun=tag:$tag
OR
$ t/runtests --dump-logs -k t debian/test-out tag:$tag
Running tests under coverage¶
It is possible to run most of the tests under Devel::Cover. This is done by
passing
--coverage to
t/runtests. Example:
$ t/runtests --coverage --dump-logs -j1 -k t debian/test-out
Please note that Devel::Cover does not seem to handle multiple threads too well.
You may see spurious warnings/errors if you run the tests with 2 or more
active worker threads.
Caveat 1: Coverage for collections (i.e. programs in
collection/)
does not seem to work at the moment. Therefore, they often end up with (next
to) zero coverage in the generated reports.
Caveat 2: Devel::Cover sometimes changes the output of Lintian or tools
called by Lintian. Obviously, this leads to test failures. Therefore, you may
see weird test failures (or warnings) when running under coverage.
Collecting the coverage you want in a reasonable time
Collecting coverage is excruciatingly slow. This is not helped by the fact that
it becomes unreliable when run under 2 or more threads.
Fortunately, Devel::Cover "appends" to its cover database. This allows
you to "slowly" build up the coverage database over multiple runs.
Example:
$ t/runtests --coverage --dump-logs -j1 -k t debian/test-out suite:scripts
$ t/runtests --coverage --dump-logs -j1 -k t debian/test-out suite:debs
$ t/runtests --coverage --dump-logs -j1 -k t debian/test-out suite:source
...
Or:
$ t/runtests --coverage --dump-logs -j1 -k t debian/test-out $check
$ t/runtests --coverage --dump-logs -j1 -k t debian/test-out legacy
SEE ALSO¶
Lintian::Tutorial::WritingTests