.\" Copyright 2012 The Kyua Authors. .\" All rights reserved. .\" .\" Redistribution and use in source and binary forms, with or without .\" modification, are permitted provided that the following conditions are .\" met: .\" .\" * Redistributions of source code must retain the above copyright .\" notice, this list of conditions and the following disclaimer. .\" * Redistributions in binary form must reproduce the above copyright .\" notice, this list of conditions and the following disclaimer in the .\" documentation and/or other materials provided with the distribution. .\" * Neither the name of Google Inc. nor the names of its contributors .\" may be used to endorse or promote products derived from this software .\" without specific prior written permission. .\" .\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS .\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT .\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR .\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT .\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, .\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT .\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, .\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY .\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT .\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE .\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. .Dd October 13, 2014 .Dt KYUA-DEBUG 1 .Os .Sh NAME .Nm "kyua debug" .Nd Executes a single test case with facilities for debugging .Sh SYNOPSIS .Nm .Op Fl -build-root Ar path .Op Fl -kyuafile Ar file .Op Fl -stdout Ar path .Op Fl -stderr Ar path .Ar test_case .Sh DESCRIPTION The .Nm command provides a mechanism to execute a single test case bypassing some of the Kyua infrastructure and allowing the user to poke into the execution behavior of the test. .Pp The test case to run is selected by providing a test filter, described below in .Sx Test filters , that matches a single test case. The test case is executed and its result is printed as the last line of the output of the tool. .Pp The test executed by .Nm is run under a controlled environment as described in .Sx Test isolation . .Pp At the moment, the .Nm command allows the following aspects of a test case execution to be tweaked: .Bl -bullet .It Redirection of the test case's stdout and stderr to the console (the default) or to arbitraty files. See the .Fl -stdout and .Fl -stderr options below. .El .Pp The following subcommand options are recognized: .Bl -tag -width XX .It Fl -build-root Ar path Specifies the build root in which to find the test programs referenced by the Kyuafile, if different from the Kyuafile's directory. See .Sx Build directories below for more information. .It Fl -kyuafile Ar file , Fl k Ar file Specifies the Kyuafile to process. Defaults to .Pa Kyuafile file in the current directory. .It Fl -stderr Ar path Specifies the file to which to send the standard error of the test program's body. The default is .Pa /dev/stderr , which is a special that redirects the output to the console. .It Fl -stdout Ar path Specifies the file to which to send the standard output of the test program's body. The default is .Pa /dev/stdout , which is a special that redirects the output to the console. .El .Pp For example, consider the following Kyua session: .Bd -literal -offset indent $ kyua test kernel/fs:mkdir -> passed kernel/fs:rmdir -> failed: Invalid argument 1/2 passed (1 failed) .Ed .Pp At this point, we do not have a lot of information regarding the failure of the .Sq kernel/fs:rmdir test. We can run this test through the .Nm command to inspect its output a bit closer, hoping that the test case is kind enough to log its progress: .Bd -literal -offset indent $ kyua debug kernel/fs:rmdir Trying rmdir('foo') Trying rmdir(NULL) kernel/fs:rmdir -> failed: Invalid argument .Ed .Pp Luckily, the offending test case was printing status lines as it progressed, so we could see the last attempted call and we can know match the failure message to the problem. .Ss Build directories .\" Copyright 2012 The Kyua Authors. .\" All rights reserved. .\" .\" Redistribution and use in source and binary forms, with or without .\" modification, are permitted provided that the following conditions are .\" met: .\" .\" * Redistributions of source code must retain the above copyright .\" notice, this list of conditions and the following disclaimer. .\" * Redistributions in binary form must reproduce the above copyright .\" notice, this list of conditions and the following disclaimer in the .\" documentation and/or other materials provided with the distribution. .\" * Neither the name of Google Inc. nor the names of its contributors .\" may be used to endorse or promote products derived from this software .\" without specific prior written permission. .\" .\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS .\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT .\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR .\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT .\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, .\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT .\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, .\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY .\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT .\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE .\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. .Em Build directories (or object directories, target directories, product directories, etc.) is the concept that allows a developer to keep the source tree clean from build products by asking the build system to place such build products under a separate subtree. .Pp Most build systems today support build directories. For example, the GNU Automake/Autoconf build system exposes such concept when invoked as follows: .Bd -literal -offset indent $ cd my-project-1.0 $ mkdir build $ cd build $ ../configure $ make .Ed .Pp Under such invocation, all the results of the build are left in the .Pa my-project-1.0/build/ subdirectory while maintaining the contents of .Pa my-project-1.0/ intact. .Pp Because build directories are an integral part of most build systems, and because they are a tool that developers use frequently, .Nm supports build directories too. This manifests in the form of .Nm being able to run tests from build directories while reading the (often immutable) test suite definition from the source tree. .Pp One important property of build directories is that they follow (or need to follow) the exact same layout as the source tree. For example, consider the following directory listings: .Bd -literal -offset indent src/Kyuafile src/bin/ls/ src/bin/ls/Kyuafile src/bin/ls/ls.c src/bin/ls/ls_test.c src/sbin/su/ src/sbin/su/Kyuafile src/sbin/su/su.c src/sbin/su/su_test.c obj/bin/ls/ obj/bin/ls/ls* obj/bin/ls/ls_test* obj/sbin/su/ obj/sbin/su/su* obj/sbin/su/su_test* .Ed .Pp Note how the directory layout within .Pa src/ matches that of .Pa obj/ . The .Pa src/ directory contains only source files and the definition of the test suite (the Kyuafiles), while the .Pa obj/ directory contains only the binaries generated during a build. .Pp All commands that deal with the workspace support the .Fl -build-root Ar path option. When this option is provided, the directory specified by the option is considered to be the root of the build directory. For example, considering our previous fake tree layout, we could invoke .Nm as any of the following: .Bd -literal -offset indent $ kyua debug --kyuafile=src/Kyuafile --build-root=obj $ cd src && kyua debug --build-root=../obj .Ed .Ss Test filters .\" Copyright 2012 The Kyua Authors. .\" All rights reserved. .\" .\" Redistribution and use in source and binary forms, with or without .\" modification, are permitted provided that the following conditions are .\" met: .\" .\" * Redistributions of source code must retain the above copyright .\" notice, this list of conditions and the following disclaimer. .\" * Redistributions in binary form must reproduce the above copyright .\" notice, this list of conditions and the following disclaimer in the .\" documentation and/or other materials provided with the distribution. .\" * Neither the name of Google Inc. nor the names of its contributors .\" may be used to endorse or promote products derived from this software .\" without specific prior written permission. .\" .\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS .\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT .\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR .\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT .\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, .\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT .\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, .\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY .\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT .\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE .\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. A .Em test filter is a string that is used to match test cases or test programs in a test suite. Filters have the following form: .Bd -literal -offset indent test_program_name[:test_case_name] .Ed .Pp Where .Sq test_program_name is the name of a test program or a subdirectory in the test suite, and .Sq test_case_name is the name of a test case. .Ss Test isolation .\" Copyright 2014 The Kyua Authors. .\" All rights reserved. .\" .\" Redistribution and use in source and binary forms, with or without .\" modification, are permitted provided that the following conditions are .\" met: .\" .\" * Redistributions of source code must retain the above copyright .\" notice, this list of conditions and the following disclaimer. .\" * Redistributions in binary form must reproduce the above copyright .\" notice, this list of conditions and the following disclaimer in the .\" documentation and/or other materials provided with the distribution. .\" * Neither the name of Google Inc. nor the names of its contributors .\" may be used to endorse or promote products derived from this software .\" without specific prior written permission. .\" .\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS .\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT .\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR .\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT .\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, .\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT .\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, .\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY .\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT .\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE .\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. The test programs and test cases run by .Nm are all executed in a deterministic environment. This known, clean environment serves to make the test execution as reproducible as possible and also to prevent clashes between tests that may, for example, create auxiliary files with overlapping names. .Pp For plain test programs and for TAP test programs, the whole test program is run under a single instance of the environment described in this page. For ATF test programs (see .Xr atf 7 ) , each individual test case .Em and test cleanup routine are executed in separate environments. .Bl -tag -width XX .It Process space Each test is executed in an independent processes. Corollary: the test can do whatever it wants to the current process (such as modify global variables) without having to undo such changes. .It Session and process group The test is executed in its own session and its own process group. There is no controlling terminal attached to the session. .Pp Should the test spawn any children, the children should maintain the same session and process group. Modifying any of these settings prevents .Nm from being able to kill any stray subprocess as part of the cleanup phase. If modifying these settings is necessary, or if any subprocess started by the test decides to use a different process group or session, it is the responsibility of the test to ensure those subprocesses are forcibly terminated during cleanup. .It Work directory The test is executed in a temporary directory automatically created by the runtime engine. Corollary: the test can write to its current directory without needing to clean any files and/or directories it creates. The runtime engine takes care to recursively delete the temporary directories after the execution of a test case. Any file systems mounted within the temporary directory are also unmounted. .It Home directory The .Va HOME environment variable is set to the absolute path of the work directory. .It Umask The value of the umask is set to 0022. .It Environment The .Va LANG , .Va LC_ALL , .Va LC_COLLATE , .Va LC_CTYPE , .Va LC_MESSAGES , .Va LC_MONETARY , .Va LC_NUMERIC and .Va LC_TIME variables are unset. .Pp The .Va TZ variable is set to .Sq UTC . .Pp The .Va TMPDIR variable is set to the absolute path of the work directory. This is to prevent the test from mistakenly using a temporary directory outside of the automatically-managed work directory, should the test use the .Xr mktemp 3 familiy of functions. .It Process limits The maximum soft core size limit is raised to its corresponding hard limit. This is a simple, best-effort attempt at allowing tests to dump core for further diagnostic purposes. .It Configuration varibles The test engine may pass run-time configuration variables to the test program via the environment. The name of the configuration variable is prefixed with .Sq TEST_ENV_ so that a configuration variable of the form .Sq foo=bar becomes accessible in the environment as .Sq TEST_ENV_foo=bar . .El .Sh EXIT STATUS The .Nm command returns 0 if the test case passes or 1 if the test case fails. .Pp Additional exit codes may be returned as described in .Xr kyua 1 . .Sh SEE ALSO .Xr kyua 1 , .Xr kyuafile 5