NAME¶
aegis test - run tests
SYNOPSIS¶
aegis -Test [
option... ][
name=value
][
file-name... ]
aegis -Test -INDependent [
option... ][
name=value ][
file-name... ]
aegis -Test -List [
option... ]
aegis -Test -Help
DESCRIPTION¶
The
aegis -Test command is used to run tests. If no files are
named, all relevant tests are run. By default both automatic and manual tests
are run.
You may name directories on the command line, and all relevant tests in that
directory tree in the change will be run. It is an error if there are no
relevant tests.
Each architecture must be tested separately. This is because there may be subtle
problems that are only revealed on some architectures. Some projects may also
have different code for different architectures.
The status of the last test run is remembered so that tests are not run if there
is no need. (This does not apply to
-REGression tests, unfortunately.)
Tests must be re-run if the test previously failed, if the test file has
changed, if there has been a build, and for each architecture.
name=value¶
You can add
name=value pairs to the command line, these
will be passed unchanged to the test command. Usually on the end of the
command line, but this can be changed in the project configuration file.
The
-force option results in an implicit force=1 variable being added to
the list of variable assignments, and thus added to the end of the command.
This is of most use when using the
batch_test_command filed of the
project configuration file.
This may initially look like a development process end-run, allowing test
scripts to be written so that they give all the right answers without actually
doing anything. You have always been able to do this with environment
variables, so this isn't anything new.
It is possible to get all of the variable assignments to turn into environment
variables by putting $var at the
start of the command, before the name
of the shell, rather than at the default location at the end of the command.
File Name Interpretation¶
The aegis program will attempt to determine the project file names from the file
names given on the command line. All file names are stored within aegis
projects as relative to the root of the baseline directory tree. The
development directory and the integration directory are shadows of this
baseline directory, and so these relative names apply here, too. Files named
on the command line are first converted to absolute paths if necessary. They
are then compared with the baseline path, the development directory path, and
the integration directory path, to determine a baseline-relative name. It is
an error if the file named is outside one of these directory trees.
The
-BAse_RElative option may be used to cause relative filenames to be
interpreted as relative to the baseline path; absolute filenames will still be
compared with the various paths in order to determine a baseline-relative
name.
The
relative_filename_preference in the user configuration file may be
used to modify this default behavior. See
aeuconf(5) for more
information.
TEST PROCESS¶
Each change is required to be accompanied by tests, and those tests are required
to be run against the built development directory, and they must pass. This
ensures that new functionality is accompanied by tests to verify its
correctness, and bug fixes are accompanied by tests which confirm that the bug
has been fixed.
Regression Tests¶
Tests are treated as any other source file, and are maintained in the baseline
and history with all other source files. The tests which must accompany every
change accumulate in the project baseline, providing a definition of correct
function for the baseline. These accumulated tests may be executed using an
“aegis -REGression” command, to verify that the project will not
“regress” as a result of a change.
Baseline Tests¶
Bug fixes are required to have their tests
fail against the project
baseline (in contrast to the development directory). This ensures that the
test actually demonstrates the bug in the baseline, as well as demonstrating
that it is fixed by the change. New functionality trivially fails against the
baseline, and so aegis does not attempt to guess if a test is a bug fix test
or new functionality test, it simply requires tests to fail against the
baseline.
This requirement applies both to new tests being created by a change and also to
tests which have been copied into a change for modification.
Reviewing Tests¶
Reviewers may be confident that aegis has enforced the test requirements; that a
change must have tests, that the change must build, that the tests pass
against the development directory, and that the tests fail against the
baseline. These conditions are enforced by
aede(1) and the change will
not be advanced to the
being reviewed state until these conditions are
met. Reviewers should thus review tests for
completeness of coverage of
the code in the change, and insensitivity to changes in the execution
environment (e.g. not date sensitive). Reviewers should also use “aegis
-list change_details” to verify that a change does or does not have
testing exemptions.
Exemptions¶
Various test exemptions may be granted by project administrators, see
aepa(1) and
aepattr(5) for more information. Copying tests into
a change, or adding new tests to a change, may cancel those exemptions.
TEST COMMAND CONFIGURATION¶
The command used to execute tests is defined by the
test_command field in
the project configuration file (see
aepconf(5) for more information),
this defaults to using the Bourne shell if not set. The current directory will
be the top of the appropriate directory tree. If tests require temporary
files, they should create them in
/tmp, as a test cannot expect to have
write permission in the current directory.
If you want to use a more sophisticated test engine, rather than a simple shell
script, but this test engine does not return result codes suitable for use
with aegis, you could wrap it in a shell script which re-writes the exit
status into the values aegis expects. You could also achieve the same results
by writing a more complex
test_command in the project
config
file.
It is also possible to write test commands which are able to test more than one
file at once. This is controlled by the
batch_test_command field of the
project
config file. In this case, the ${output} substitution indicates
the name of a file the test command must create, in
aetest(5) format,
to contain the results of the tests run. This is often used on systems with
multiple CPUs or the ability to distribute jobs across several computers on a
network.
Substitutions¶
All of the
aesub(5) substitutions are available in the test commands.
Some of them are of particular note:
- ARCHitecture
- This substitution is replaced by the name of the
architecture to be tested.
- Search_Path
- This substitution is replaced by a colon separated list of
absolute paths to search when looking for test support files.
- Search_Path_Executable
- This substitution is replaced by a colon separated list of
absolute paths to search when looking for executable support files
(library files and sub-commands).
Most of the time
$Search_Path_Executable are exactly the same. However,
during “aegis -t -bl” they will be different, with
$Seach_Path starting at the development directory (the test being run)
and
$Seach_Path_Executable starting at the baseline (the executable
being run).
Test Result Codes¶
As each test is run (via the
test_command field in the project
config file), aegis determines whether the test succeeded or failed by
looking at its exit status. This exit status is mostly as expected for UNIX
commands.
- Success
- A test should exit 0 to indicate success, i.e. that the
specific function under test worked as expected.
- Failure
- A test should exit 1 to indicate failure, i.e. that the
specific function under test did not work as expected.
- No Result
- A test should exit 2 to indicate no result, i.e.
that the specific function under test could not be exercised because
something else went wrong. For example, running out of disk space when
creating the test input files in the /tmp directory.
- Skipped
- A test should exit 77 to indicate that it was skipped. This
is usually to do with the current architecture not being meaningful.
Whenever possible, use “No Result” instead. (The value was
chosen for compatibility with other test systems.)
Actually, any exit code other than 0, 1 or 77 will be interpreted as “no
result”. However, always using 0, 1, 2 or 77 means that if a new result
code is required by a later release of Aegis your existing tests will continue
to work.
TEST CORRELATIONS¶
The “aegis -Test -SUGgest” command may be used to have aegis suggest
suitable regression tests for your change, based on the source files in your
change. This automatically focuses testing effort to relevant tests, reducing
the number of regression tests necessary to be confident that you have not
introduced a bug.
The test correlations are generated by the “aegis -Integrate_Pass”
command, which associates each test in the change with each source file in the
change. Thus, each source file accumulates a list of tests which have been
associated with it in the past. This is not as exact as code coverage
analysis, but is a reasonable approximation in practice.
The
aecp(1) and
aenf(1) commands are used to associate files with
a change. While they do not actively perform the association, these are the
files used by
aeipass(1) and
aet(1) to determine which source
files are associated with which tests.
Test Correlation Accuracy¶
Assuming that the testing correlations are accurate and that the tests are
evenly distributed across the function space, there will be a less than
1/number chance that a relevant test has not been run by the
“aegis -Test -SUGgest
number” command. A small amount of
noise is added to the test weighting, so that unexpected things are sometimes
tested, and the same tests are not run every time.
Test correlation accuracy can be improved by ensuring that:
- •
- Each change should be strongly focused, with no gratuitous
file inclusions. This avoids spurious correlations.
- •
- Each item of new functionality should be added in an
individual change, rather than several together. This strongly correlates
tests with functionality.
- •
- Each bug should be fixed in an individual change, rather
than several together. This strongly correlates tests with
functionality.
- •
- Test correlations will be lost if files are moved. This is
because correlations are by name.
The best way for tests to correlate accurately with source files is when a
change contains a test and exactly those files relating to the functionality
under test. Too many spurious files will weaken the usefulness of the testing
correlations.
OPTIONS¶
The following options are understood:
- -AUTOmatic
- This option may be used to specify automatic tests.
Automatic tests require no human assistance.
- -BaseLine
- This option may be used to specify that the project
baseline is the subject of the command.
- -BAse_RElative
- This option may be used to cause relative filenames to be
considered relative to the base of the source tree. See aeuconf(5)
for the corresponding user preference.
- -CUrrent_RElative
- This option may be used to cause relative filenames to be
considered relative to the current directory. This is usually the default.
See aeuconf(5) for the corresponding user preference.
- -Change number
- This option may be used to specify a particular change
within a project. See aegis(1) for a complete description of this
option.
- -FOrce
- This option may be used to specify that all tests should be
run, even if the status of the last test run indicates that there is no
need to run a specific test.
- -Help
-
This option may be used to obtain more information about how to use the
aegis program.
- -INDependent
- This option is used to specify that the test is to be run
independent of any particular change. If no tests are named, all tests in
the baseline will be run.
- -List
-
This option may be used to obtain a list of suitable subjects for this
command. The list may be more general than expected.
- -MANual
- This option may be used to specify manual tests. Manual
tests require some human intervention, e.g.: confirmation of some screen
behavior (X11, for instance), or some user action, "unplug ethernet
cable now".
- -Not_Logging
-
This option may be used to disable the automatic logging of output and
errors to a file. This is often useful when several aegis commands are
combined in a shell script.
- -PErsevere
- This option may be used to specify that all tests should be
run, even if some fail. Defaults to the user's persevere_preference
if not specified, see aeuconf(5) for more information.
- -No_PErsevere
- This option may be used to specify that the test run should
stop after the first failure. Defaults to the user's
persevere_preference if not specified, see aeuconf(5) for
more information.
- -Project name
- This option may be used to select the project of interest.
When no -Project option is specified, the AEGIS_PROJECT
environment variable is consulted. If that does not exist, the user's
$HOME/.aegisrc file is examined for a default project field (see
aeuconf(5) for more information). If that does not exist, when the
user is only working on changes within a single project, the project name
defaults to that project. Otherwise, it is an error.
- -PROGress
- This option may be used to specify that progress messages
should be issued before each test run or before each batch test run in
case batch_test_command field specified in project config
file (see aeuconf(5) for more information).
- -No_PROGress
- This option may be used to specify that progress messages
should be suppressed. This is the default.
- -REGression
- This option is used to specify that the regression test
suite is to be run. The regression test suite consists of all tests in the
baseline which do not appear in the change. It is an error if there are no
regression tests. You may not name tests on the command line when using
the -REGression option. You may name individual tests to be run on the
command line, without using the -REGression option; if they are not part
of the change, the tests of the same name in the baseline will be
run.
- -SUGgest [ number ]
-
The “ aegis -Integrate_Pass” command collects test
correlation statistics when changes are integrated. This option may be
used to request that aegis suggest which tests should be run, using these
testing correlations. If no number is specified, 10 tests will be
suggested. This option implies the -REGression option.
- -SUGgest_Limit minutes
-
This option may be used to limit the number of tests to a certain number of
minutes. They will be run from most relevant to least relevant.
- -SUGgest_Noise number
-
This option may be used to control the amount of noise injected into the
test selection performed by the -SUGgest option. The number is a
percentage of noise to be injected. Defaults to 10 if not specified. The
injection of noise ensures that a variety of tests are run on subsequent
runs, and also some from left-field as a sanity check.
- -TERse
-
This option may be used to cause listings to produce the bare minimum of
information. It is usually useful for shell scripts.
- -Verbose
- This option may be used to cause aegis to produce more
output. By default aegis only produces output on errors. When used with
the -List option this option causes column headings to be
added.
- -Wait
- This option may be used to require Aegis commands to wait
for access locks, if they cannot be obtained immediately. Defaults to the
user's lock_wait_preference if not specified, see aeuconf(5)
for more information.
- -No_Wait
- This option may be used to require Aegis commands to emit a
fatal error if access locks cannot be obtained immediately. Defaults to
the user's lock_wait_preference if not specified, see
aeuconf(5) for more information.
See also
aegis(1) for options common to all aegis commands.
All options may be abbreviated; the abbreviation is documented as the upper case
letters, all lower case letters and underscores (_) are optional. You must use
consecutive sequences of optional letters.
All options are case insensitive, you may type them in upper case or lower case
or a combination of both, case is not important.
For example: the arguments "-project, "-PROJ" and "-p"
are all interpreted to mean the
-Project option. The argument
"-prj" will not be understood, because consecutive optional
characters were not supplied.
Options and other command line arguments may be mixed arbitrarily on the command
line, after the function selectors.
The GNU long option names are understood. Since all option names for
aegis are long, this means ignoring the extra leading '-'. The "
--option=value" convention is also
understood.
RECOMMENDED ALIAS¶
The recommended alias for this command is
csh% alias aet 'aegis -t \!* -v'
sh$ aet(){aegis -t "$@" -v}
ERRORS¶
It is an error if the change is not in one of the
being developed or
being integrated states.
It is an error if the change is not assigned to the current user.
It is an error if your have no relevant tests and no relevant exemption.
EXIT STATUS¶
The
aegis command will exit with a status of 1 on any error. The
aegis command will only exit with a status of 0 if there are no errors.
ENVIRONMENT VARIABLES¶
See
aegis(1) for a list of environment variables which may affect this
command. See
aepconf(5) for the project configuration file's
project_specific field for how to set environment variables for all
commands executed by Aegis.
SEE ALSO¶
- aeb(1)
- build a change
- aeca(1)
- modify the attributes of a change
- aedb(1)
- begin development of a change
- aeib(1)
- begin integration of a change
- aent(1)
- add a new test to a change
- aecp(1)
- copy an existing test into a change
- aepconf(5)
- project configuration file format
- aeuconf(5)
- user configuration file format
COPYRIGHT¶
aegis version 4.24.3.D001
Copyright (C) 1991, 1992, 1993, 1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001,
2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010 Peter Miller
The aegis program comes with ABSOLUTELY NO WARRANTY; for details use the '
aegis -VERSion License' command. This is free software and you are
welcome to redistribute it under certain conditions; for details use the '
aegis -VERSion License' command.
AUTHOR¶