summaryrefslogtreecommitdiff
path: root/tools/testing/selftests/kselftest.h
AgeCommit message (Collapse)Author
2020-08-05Merge tag 'linux-kselftest-5.9-rc1' of ↵Linus Torvalds
git://git.kernel.org/pub/scm/linux/kernel/git/shuah/linux-kselftest Pull kselftest updates form Shuah Khan: - TAP output reporting related fixes from Paolo Bonzini and Kees Cook. These fixes make it skip reporting consistent with TAP format. - Cleanup fixes to framework run_tests from Yauheni Kaliuta * tag 'linux-kselftest-5.9-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/shuah/linux-kselftest: (23 commits) selftests/harness: Limit step counter reporting selftests/seccomp: Check ENOSYS under tracing selftests/seccomp: Refactor to use fixture variants selftests/harness: Clean up kern-doc for fixtures selftests: kmod: Add module address visibility test Replace HTTP links with HTTPS ones: KMOD KERNEL MODULE LOADER - USERMODE HELPER selftests: fix condition in run_tests selftests: do not use .ONESHELL selftests: pidfd: skip test if unshare fails with EPERM selftests: pidfd: do not use ksft_exit_skip after ksft_set_plan selftests/harness: Report skip reason selftests/harness: Display signed values correctly selftests/harness: Refactor XFAIL into SKIP selftests/harness: Switch to TAP output selftests: Add header documentation and helpers selftests/binderfs: Fix harness API usage selftests: Remove unneeded selftest API headers selftests/clone3: Reorder reporting output selftests: sync_test: do not use ksft_exit_skip after ksft_set_plan selftests: sigaltstack: do not use ksft_exit_skip after ksft_set_plan ...
2020-07-06selftests/harness: Switch to TAP outputKees Cook
Using the kselftest_harness.h would result in non-TAP test reporting, which didn't make much sense given that all the requirements for using the low-level API were met. Switch to using ksft_*() helpers while retaining as much of a human-readability as possible. Signed-off-by: Kees Cook <keescook@chromium.org> Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
2020-07-06selftests: Add header documentation and helpersKees Cook
Add "how to use this API" documentation to kselftest.h, and include some addition helpers and notes to make things easier to use. Additionally removes the incorrect "Bail out!" line from the standard exit path. The TAP13 specification says that "Bail out!" should be used when giving up before all tests have been run. For a "normal" execution run, the selftests should not report "Bail out!". Signed-off-by: Kees Cook <keescook@chromium.org> Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
2020-07-06kselftest: fix TAP output for skipped testsPaolo Bonzini
According to the TAP specification, a skipped test must be marked as "ok" and annotated with the SKIP directive, for example ok 23 # skip Insufficient flogiston pressure. (https://testanything.org/tap-specification.html) Fix the kselftest infrastructure to match this. For ksft_exit_skip, it is preferrable to emit a dummy plan line that indicates the whole test was skipped, but this is not always possible because of ksft_exit_skip being used as a "shortcut" by the tests. In that case, print the test counts and a normal "ok" line. The format is now the same independent of whether msg is NULL or not (but it is never NULL in any caller right now). Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
2020-07-06kselftest: ksft_test_num return type should be unsignedPaolo Bonzini
Fixes a compiler warning: In file included from sync_test.c:37: ../kselftest.h: In function ‘ksft_print_cnts’: ../kselftest.h:78:16: warning: comparison of integer expressions of different signedness: ‘unsigned int’ and ‘int’ [-Wsign-compare] if (ksft_plan != ksft_test_num()) ^~ Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
2019-07-30kselftest: save-and-restore errno to allow for %m formattingAleksa Sarai
Previously, using "%m" in a ksft_* format string can result in strange output because the errno value wasn't saved before calling other libc functions. The solution is to simply save and restore the errno before we format the user-supplied format string. Signed-off-by: Aleksa Sarai <cyphar@cyphar.com> Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
2019-04-25selftests: Add test plan API to kselftest.h and adjust callersKees Cook
The test plan for TAP needs to be declared immediately after the header. This adds the test plan API to kselftest.h and updates all callers to declare their expected test counts. Signed-off-by: Kees Cook <keescook@chromium.org> Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
2019-04-25selftests: Move test output to diagnostic linesKees Cook
This changes the selftest output so that each test's output is prefixed with "# " as a TAP "diagnostic line". This creates a bit of a kernel-specific TAP dialect where the diagnostics precede the results. The TAP spec isn't entirely clear about this, though, so I think it's the correct solution so as to keep interactive runs making sense. If the output _followed_ the result line in the spec-suggested YAML form, each test would dump all of its output at once instead of as it went, making debugging harder. This does, however, solve the recursive TAP output problem, as sub-tests will simply be prefixed by "# ". Parsing sub-tests becomes a simple problem of just removing the first two characters of a given top-level test's diagnostic output, and parsing the results. Note that the shell construct needed to both get an exit code from the first command in a pipe and still filter the pipe (to add the "# " prefix) uses a POSIX solution rather than the bash "pipefail" option which is not supported by dash. Since some test environments may have a very minimal set of utilities available, the new prefixing code will fall back to doing line-at-a-time prefixing if perl and/or stdbuf are not available. Signed-off-by: Kees Cook <keescook@chromium.org> Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
2019-04-25selftests: Add plan line and fix result line syntaxKees Cook
The TAP version 13 spec requires a "plan" line, which has been missing. Since we always know how many tests we're going to run, emit the count on the plan line. This also fixes the result lines to remove the "1.." prefix which is against spec, and to mark skips with the correct "# SKIP" suffix. Signed-off-by: Kees Cook <keescook@chromium.org> Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
2019-02-13selftest: include stdio.h in kselftest.hTycho Andersen
While playing around with a way to skip the seccomp get_metadata test, I noticed that this header uses printf() without defining it, leading to, ../kselftest.h: In function ‘ksft_print_header’: ../kselftest.h:61:3: warning: implicit declaration of function ‘printf’ [-Wimplicit-function-declaration] printf("TAP version 13\n"); ^~~~~~ ../kselftest.h:61:3: warning: incompatible implicit declaration of built-in function ‘printf’ ../kselftest.h:61:3: note: include ‘<stdio.h>’ or provide a declaration of ‘printf’ if user code doesn't also use printf. Signed-off-by: Tycho Andersen <tycho@tycho.ws> Acked-by: Kees Cook <keescook@chromium.org> Signed-off-by: Shuah Khan <shuah@kernel.org>
2018-08-27selftests: kselftest: Remove outdated commentThiago Jung Bauermann
Commit 3c07aaef6598 ("selftests: kselftest: change KSFT_SKIP=4 instead of KSFT_PASS") reverted commit 11867a77eb85 ("selftests: kselftest framework: change skip exit code to 0") but missed removing the comment which that commit added, so do that now. Signed-off-by: Thiago Jung Bauermann <bauerman@linux.ibm.com> Signed-off-by: Shuah Khan (Samsung OSG) <shuah@kernel.org>
2018-05-30selftests: kselftest: change KSFT_SKIP=4 instead of KSFT_PASSShuah Khan (Samsung OSG)
KSFT_SKIP points to KSFT_PASS resulting in reporting skipped tests as Passed, when test programs exit with KSFT_SKIP or call ksft_exit_skip(). If tests are skipped because of unmet dependencies and/or unsupported configuration, reporting them as passed leads to too many false positives. Fix it to return a skip code of 4 to clearly differentiate the skipped tests. Signed-off-by: Shuah Khan (Samsung OSG) <shuah@kernel.org>
2018-03-05selftests: kselftest framework: add handling for TAP header levelShuah Khan
Introduce environment variable KSFT_TAP_LEVEL to avoid printing nested TAP headers for each test. lib.mk run_tests target prints TAP header before invoking the test program or test script. Tests need a way to suppress TAP headers if it is already printed out. This new environment variable adds a way for ksft_print_header() print TAP header only when KSFT_TAP_LEVEL isn't set. lib.mk run_tests and test program should print TAP header and set KSFT_TAP_LEVEL to avoid a second TAP header to be printed. selftests Makefile should export KSFT_TAP_LEVEL and add TAP Header echo to the run_kselftest.sh script from emit_tests target handling. Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>
2018-01-12selftests: kselftest.h: Add SPDX license identifierShuah Khan
Replace GPL license statement with SPDX GPL-2.0 license identifier. Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>
2017-08-25selftests: kselftest framework: change skip exit code to 0Shuah Khan
When a test is skipped, instead of using a special exit code of 4, treat it as pass condition and use exit code of 0. It makes sense to treat skip as pass since the test couldn't be run as opposed to a failed test. Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>
2017-08-09selftests: kselftest framework: add error counterShuah Khan
Some tests track errors in addition to test failures. Add ksft_error counter, ksft_get_error_cnt(), and ksft_test_result_error() API to get the counter value and print error message. Update ksft_print_cnts(), and ksft_test_num() to include error counter. Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>
2017-07-28selftests: kselftest framework: add API to return pass/fail/* countsShuah Khan
Some tests print final pass/fail message based on fail count. Add ksft_get_*_cnt() API to kselftest framework to return counts. Update ksft_print_cnts() to print the test results summary message with individual pass, fail, ... counters. Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com> Reviewed-by: Gustavo Padovan <gustavo.padovan@collabora.com>
2017-06-30kselftest: add ksft_print_msg() function to output general informationPaul Elder
Add a generic information output function: ksft_print_msg() Signed-off-by: Paul Elder <paul.elder@pitt.edu> Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>
2017-06-30kselftest: make ksft_* output functions variadicPaul Elder
Make the ksft_* output functions variadic to allow string formatting directly in these functions. Signed-off-by: Paul Elder <paul.elder@pitt.edu> Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>
2017-06-15kselftest: make ksft_exit_skip() output a reason for skippingPaul Elder
Make ksft_exit_skip() input an optional message string as the reason for skipping all the tests and outputs it prior to exiting. Signed-off-by: Paul Elder <paul.elder@pitt.edu> Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>
2017-06-13kselftest: add TAP13 conformant versions of ksft_* functionsPaul Elder
Add TAP13 conformat output functions to kselftest.h. Also add exit functions that output TAP13 exiting text, as well as functions to keep track of testing progress. Signed-off-by: Paul Elder <paul.elder@pitt.edu> Signed-off-by: Alice Ferrazzi <alice.ferrazzi@gmail.com> Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>
2015-05-26kselftest: Add exit code definesDarren Hart
Define the exit codes with KSFT_PASS and similar so tests can use these directly if they choose. Also enable harnesses and other tooling to use the defines instead of hardcoding the return codes. Cc: Shuah Khan <shuahkh@osg.samsung.com> Cc: linux-api@vger.kernel.org Cc: Ingo Molnar <mingo@elte.hu> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Davidlohr Bueso <dave@stgolabs.net> Cc: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com> Signed-off-by: Darren Hart <dvhart@linux.intel.com> Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>
2014-11-17selftests: add kselftest framework for uniform test reportingShuah Khan
Add kselftest framework for tests to use. This is a light weight framework provides a set of interfaces to report test results. Tests can use these interfaces to report pass, and fail cases as well as when failure is due to configuration problems such as missing modules, or when a test that is should fail, fails as expected, and a test that should fail, passes. The framework uses POSIX standard return codes for reporting results to address the needs of users that want to run the kernel selftests from their user-space test suites and want to know why a test failed. In addition, the framework includes interfaces to use to report test statistics on number of tests passed and failed. Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>