NAME¶
Test::LectroTest - Easy, automatic, specification-based tests
SYNOPSIS¶
#!/usr/bin/perl -w
use MyModule; # contains code we want to test
use Test::LectroTest;
Property {
##[ x <- Int, y <- Int ]##
MyModule::my_function( $x, $y ) >= 0;
}, name => "my_function output is non-negative" ;
Property { ... }, name => "yet another property" ;
# more properties to check here
DESCRIPTION¶
This module provides a simple (yet full featured) interface to LectroTest, an
automated, specification-based testing system for Perl. To use it, declare
properties that specify the expected behavior of your software. LectroTest
then checks your software to see whether those properties hold.
Declare properties using the "Property" function, which takes a block
of code and promotes it to a Test::LectroTest::Property:
Property {
##[ x <- Int, y <- Int ]##
MyModule::my_function( $x, $y ) >= 0;
}, name => "my_function output is non-negative" ;
The first part of the block must contain a generator-binding declaration. For
example:
##[ x <- Int, y <- Int ]##
(Note the special bracketing, which is required.) This particular binding says,
"For all integers
x and
y." (By the way, you aren't
limited to integers. LectroTest also gives you booleans, strings, lists,
hashes, and more, and it lets you define your own generator types. See
Test::LectroTest::Generator for more.)
The second part of the block is simply a snippet of code that makes use of the
variables we bound earlier to test whether a property holds for the piece of
software we are testing:
MyModule::my_function( $x, $y ) >= 0;
In this case, it asserts that "MyModule::my_function($x,$y)" returns a
non-negative result. (Yes, $x and $y refer to the same
x and
y
that we bound to the generators earlier. LectroTest automagically loads these
lexically bound Perl variables with values behind the scenes.)
Note: If you want to use testing assertions like "ok" from
Test::Simple or "is", "like", or "cmp_ok" from
Test::More (and the related family of Test::Builder-based testing modules),
see Test::LectroTest::Compat, which lets you mix and match LectroTest with
these modules.
Finally, we give the whole Property a name, in this case "my_function
output is non-negative." It's a good idea to use a meaningful name
because LectroTest refers to properties by name in its output.
Let's take a look at the finished property specification:
Property {
##[ x <- Int, y <- Int ]##
MyModule::my_function( $x, $y ) >= 0;
}, name => "my_function output is non-negative" ;
It says, "For all integers
x and
y, we assert that
my_function's output is non-negative."
To check whether this property holds, simply put it in a Perl program that uses
the Test::LectroTest module. (See the "SYNOPSIS" for an example.)
When you run the program, LectroTest will load the property (and any others in
the file) and check it by running random trials against the software you're
testing.
Note: If you want to place LectroTest property checks into a test plan
managed by Test::Builder-based modules such as Test::Simple or Test::More, see
Test::LectroTest::Compat.
If LectroTest is able to "break" your software during the property
check, it will emit a counterexample to your property's assertions and stop.
You can plug the counterexample back into your software to debug the problem.
(You might also want to add the counterexample to a list of regression tests.)
A successful LectroTest looks like this:
1..1
ok 1 - 'my_function output is non-negative' (1000 attempts)
On the other hand, if you're not so lucky:
1..1
not ok 1 - 'my_function output is non-negative' falsified \
in 324 attempts
# Counterexample:
# $x = -34
# $y = 0
EXIT CODE¶
The exit code returned by running a suite of property checks is the number of
failed checks. The code is 0 if all properties passed their checks or
N
if
N properties failed. (If more than 254 properties failed, the exit
code will be 254.)
ADJUSTING THE TESTING PARAMETERS¶
There is one testing parameter (among others) that you might wish to change from
time to time: the number of trials to run for each property checked. By
default it is 1,000. If you want to try more or fewer trials, pass the
"trials=>"
N flag:
use Test::LectroTest trials => 10_000;
TESTING FOR REGRESSIONS AND CORNER CASES¶
LectroTest can record failure-causing test cases to a file, and it can play
those test cases back as part of its normal testing strategy. The easiest way
to take advantage of this feature is to set the
regressions parameter
when you "use" this module:
use Test::LectroTest
regressions => "regressions.txt";
This tells LectroTest to use the file "regressions.txt" for both
recording and playing back failures. If you want to record and play back from
separate files, or want only to record
or play back, use the
record_failures and/or
playback_failures options:
use Test::LectroTest
playback_failures => "regression_suite_for_my_module.txt",
record_failures => "failures_in_the_field.txt";
See Test::LectroTest::RegressionTesting for more.
CAVEATS¶
When you use this module, it imports all of the generator-building functions
from Test::LectroTest::Generator into the your code's namespace. This is
almost always what you want, but I figured I ought to say something about it
here to reduce the possibility of surprise.
A Property specification must appear in the first column, i.e., without any
indentation, in order for it to be automatically loaded and checked. If this
poses a problem, let me know, and this restriction can be lifted.
SEE ALSO¶
For a gentle introduction to LectroTest, see Test::LectroTest::Tutorial. Also,
the slides from my LectroTest talk for the Pittsburgh Perl Mongers make for a
great introduction. Download a copy from the LectroTest home (see below).
Test::LectroTest::RegressionTesting explains how to test for regressions and
corner cases using LectroTest.
Test::LectroTest::Compat lets you mix LectroTest with the popular family of
Test::Builder-based modules such as Test::Simple and Test::More.
Test::LectroTest::Property explains in detail what you can put inside of your
property specifications.
Test::LectroTest::Generator describes the many generators and generator
combinators that you can use to define the test or condition space that you
want LectroTest to search for bugs.
Test::LectroTest::TestRunner describes the objects that check your properties
and tells you how to turn their control knobs. You'll want to look here if
you're interested in customizing the testing procedure.
LECTROTEST HOME¶
The LectroTest home is
http://community.moertel.com/LectroTest. There you will
find more documentation, presentations, mailing-list archives, a wiki, and
other helpful LectroTest-related resources. It's also the best place to ask
questions.
AUTHOR¶
Tom Moertel (tom@moertel.com)
INSPIRATION¶
The LectroTest project was inspired by Haskell's QuickCheck module by Koen
Claessen and John Hughes:
http://www.cs.chalmers.se/~rjmh/QuickCheck/.
COPYRIGHT and LICENSE¶
Copyright (c) 2004-05 by Thomas G Moertel. All rights reserved.
This program is free software; you can redistribute it and/or modify it under
the same terms as Perl itself.