Add mutation testing to the Gitea workflow
A separate mutation_test job runs scripts/mutation_test.py over src/stdlib.c with a --threshold gate and publishes a JUnit report, mirroring libakerror's CI. Checkout needs submodules: recursive -- the harness copies the repo and builds the copy top-level, so deps/libakerror has to be there. Measured score on the section 1.0 suite is 46.8% (81/173 killed). The gate is set at 40: a regression ratchet, not a quality bar. Survivors cluster in the wrappers that have no tests yet (printf family, ato* family, realpath, the memory and stream wrappers); the list and tree code that section 1.0 does cover leaves only 5 survivors between them. Also caps every test with TIMEOUT 30. Measuring the score turned this up: a mutant deleting `fast = fast->next->next` makes aksl_list_iterate spin forever, so ctest never returned, the harness killed ctest at its own 120s timeout, and the orphaned test binary kept burning a core while holding the pipe open -- the run wedged and needed killing by hand. With a per-test timeout ctest reaps its own child, those mutants are killed in 30s, and nothing is left behind. It guards the real suite against the same shape of bug too. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -156,12 +156,25 @@ if(AKSL_WILL_FAIL_TESTS OR AKSL_KNOWN_FAILING_TESTS)
|
||||
)
|
||||
endif()
|
||||
|
||||
# Cap every test. The list and tree code is full of loops whose termination
|
||||
# depends on a single condition, so a plausible bug -- or a mutant from the
|
||||
# target below -- turns a test into an infinite loop. Without this, ctest waits
|
||||
# forever; the mutation harness then kills ctest at its own timeout and the
|
||||
# spinning test binary is left orphaned, holding the pipe open and burning a
|
||||
# core. The whole suite runs in well under a second, so 30s is pure headroom.
|
||||
set_tests_properties(
|
||||
${AKSL_TESTS} ${AKSL_WILL_FAIL_TESTS} ${AKSL_KNOWN_FAILING_TESTS}
|
||||
PROPERTIES TIMEOUT 30
|
||||
)
|
||||
|
||||
# Mutation testing: break the library in small ways and confirm the test suite
|
||||
# notices. This is a meta-check on the tests themselves, so it is a manual
|
||||
# target (it rebuilds and re-runs the whole suite once per mutant), not a CTest
|
||||
# test and not yet a CI gate -- the per-function tests in TODO.md sections
|
||||
# 1.1-1.9 need to exist before a threshold would mean anything.
|
||||
# notices. This is a meta-check on the tests themselves, and it rebuilds and
|
||||
# re-runs the whole suite once per mutant, so it is a manual target rather than
|
||||
# a CTest test:
|
||||
# cmake --build build --target mutation
|
||||
# This target covers both src/stdlib.c and include/akstdlib.h. CI runs the
|
||||
# narrower, faster src/stdlib.c set with a --threshold gate; see
|
||||
# .gitea/workflows/ci.yaml.
|
||||
find_package(Python3 COMPONENTS Interpreter)
|
||||
if(Python3_FOUND)
|
||||
add_custom_target(mutation
|
||||
|
||||
Reference in New Issue
Block a user