Add mutation testing to validate the test suite
Introduce a self-contained mutation testing harness that verifies the unit tests actually catch bugs: it makes small deliberate breakages to the library (flip comparisons, delete statements, swap true/false, etc.), rebuilds, and runs the whole CTest suite against each mutant. Tests that still pass reveal a gap; tests that fail "kill" the mutant. - scripts/mutation_test.py: the engine (stdlib only, no LLVM/clang deps). Operators ROR/LCR/BCR/AOR/ICR/SDL over src/error.c and the macro header. Mutates a scratch copy, never the working tree. Supports --target, --list, --max-mutants sampling, --threshold gating, --timeout. - CMakeLists.txt: 'mutation' custom target (cmake --build build --target mutation). - .gitea/workflows/ci.yaml: gated mutation job on src/error.c (threshold 65%). - tests/MUTATION.md: how to run, interpret survivors, and known equivalents. Close the real gaps the harness found in src/error.c (score 53% -> 71%): - err_error_names: the AKERR_* codes have their names registered by akerr_init - err_release_clears: releasing a context wipes it before reuse - err_pool_exhaust: akerr_next_error returns NULL when the pool is full and always hands back the lowest free slot Also surfaced (documented, not fixed): AKERR_MAX_ERR_VALUE (+15) is below AKERR_NOT_IMPLEMENTED (+16) and AKERR_BADEXC (+17), so those codes can never have a name registered. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -21,3 +21,22 @@ jobs:
|
||||
cmake --install build
|
||||
cmake --build build --target test
|
||||
- run: echo "🍏 This job's status is ${{ job.status }}."
|
||||
|
||||
mutation_test:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Check out repository code
|
||||
uses: actions/checkout@v4
|
||||
- name: dependencies
|
||||
run: |
|
||||
sudo apt-get update -y
|
||||
sudo apt-get install -y cmake gcc moreutils python3
|
||||
# Verify the tests actually catch bugs: break the library many ways and
|
||||
# confirm the suite fails. Gated on src/error.c (fast, deterministic).
|
||||
# The threshold keeps headroom below the current score for equivalent
|
||||
# mutants; see tests/MUTATION.md. Run the full default target locally for
|
||||
# deeper (slower) coverage including the macro header.
|
||||
- name: mutation testing
|
||||
run: |
|
||||
python3 scripts/mutation_test.py --target src/error.c --threshold 65
|
||||
- run: echo "🍏 This job's status is ${{ job.status }}."
|
||||
|
||||
Reference in New Issue
Block a user