Add mutation testing to validate the test suite

Introduce a self-contained mutation testing harness that verifies the unit
tests actually catch bugs: it makes small deliberate breakages to the library
(flip comparisons, delete statements, swap true/false, etc.), rebuilds, and
runs the whole CTest suite against each mutant. Tests that still pass reveal a
gap; tests that fail "kill" the mutant.

- scripts/mutation_test.py: the engine (stdlib only, no LLVM/clang deps).
  Operators ROR/LCR/BCR/AOR/ICR/SDL over src/error.c and the macro header.
  Mutates a scratch copy, never the working tree. Supports --target, --list,
  --max-mutants sampling, --threshold gating, --timeout.
- CMakeLists.txt: 'mutation' custom target (cmake --build build --target mutation).
- .gitea/workflows/ci.yaml: gated mutation job on src/error.c (threshold 65%).
- tests/MUTATION.md: how to run, interpret survivors, and known equivalents.

Close the real gaps the harness found in src/error.c (score 53% -> 71%):
- err_error_names: the AKERR_* codes have their names registered by akerr_init
- err_release_clears: releasing a context wipes it before reuse
- err_pool_exhaust: akerr_next_error returns NULL when the pool is full and
  always hands back the lowest free slot

Also surfaced (documented, not fixed): AKERR_MAX_ERR_VALUE (+15) is below
AKERR_NOT_IMPLEMENTED (+16) and AKERR_BADEXC (+17), so those codes can never
have a name registered.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-07-27 17:03:53 -04:00
parent e5f761662c
commit 43f46dca64
7 changed files with 651 additions and 0 deletions

View File

@@ -63,6 +63,9 @@ set(AKERR_TESTS
err_errno
err_break_variants
err_custom_handler
err_error_names
err_release_clears
err_pool_exhaust
)
set(AKERR_WILL_FAIL_TESTS
@@ -82,6 +85,22 @@ set_tests_properties(
PROPERTIES WILL_FAIL TRUE
)
# Mutation testing: break the library in small ways and confirm the test suite
# notices. This is a meta-check on the tests themselves, so it is a manual
# target (it rebuilds and re-runs the whole suite many times), not a CTest test.
# cmake --build build --target mutation
find_package(Python3 COMPONENTS Interpreter)
if(Python3_FOUND)
add_custom_target(mutation
COMMAND ${Python3_EXECUTABLE}
${CMAKE_CURRENT_SOURCE_DIR}/scripts/mutation_test.py
--source-root ${CMAKE_CURRENT_SOURCE_DIR}
WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}
USES_TERMINAL
COMMENT "Running mutation tests (breaks the library, expects tests to fail)"
)
endif()
set(main_lib_dest "lib/my_library-${MY_LIBRARY_VERSION}")
install(TARGETS akerror
EXPORT akerrorTargets