Add mutation testing to validate the test suite

Introduce a self-contained mutation testing harness that verifies the unit
tests actually catch bugs: it makes small deliberate breakages to the library
(flip comparisons, delete statements, swap true/false, etc.), rebuilds, and
runs the whole CTest suite against each mutant. Tests that still pass reveal a
gap; tests that fail "kill" the mutant.

- scripts/mutation_test.py: the engine (stdlib only, no LLVM/clang deps).
  Operators ROR/LCR/BCR/AOR/ICR/SDL over src/error.c and the macro header.
  Mutates a scratch copy, never the working tree. Supports --target, --list,
  --max-mutants sampling, --threshold gating, --timeout.
- CMakeLists.txt: 'mutation' custom target (cmake --build build --target mutation).
- .gitea/workflows/ci.yaml: gated mutation job on src/error.c (threshold 65%).
- tests/MUTATION.md: how to run, interpret survivors, and known equivalents.

Close the real gaps the harness found in src/error.c (score 53% -> 71%):
- err_error_names: the AKERR_* codes have their names registered by akerr_init
- err_release_clears: releasing a context wipes it before reuse
- err_pool_exhaust: akerr_next_error returns NULL when the pool is full and
  always hands back the lowest free slot

Also surfaced (documented, not fixed): AKERR_MAX_ERR_VALUE (+15) is below
AKERR_NOT_IMPLEMENTED (+16) and AKERR_BADEXC (+17), so those codes can never
have a name registered.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-07-27 17:03:53 -04:00
parent e5f761662c
commit 43f46dca64
7 changed files with 651 additions and 0 deletions

100
tests/MUTATION.md Normal file
View File

@@ -0,0 +1,100 @@
# Mutation testing
The unit tests tell us the library works. **Mutation testing tells us the tests
work** — that they would actually fail if the library were broken.
`scripts/mutation_test.py` deliberately breaks the library in small ways
("mutants"), one at a time, and runs the whole CTest suite against each broken
copy:
* if the tests **fail**, the mutant is **killed** — good, the suite caught it;
* if the tests still **pass**, the mutant **survived** — a bug of that shape
would slip through, so it points at a missing test.
The **mutation score** is `killed / (killed + survived)`. A surviving mutant is
a to-do item: write a test that distinguishes the mutant from the original.
## Running
No third-party tools are required — just Python 3 and the normal
cmake/ctest toolchain. The harness never touches your working tree; it copies
the repo to a scratch directory and mutates the copy.
```sh
# Default: mutate src/error.c and include/akerror.tmpl.h
scripts/mutation_test.py
# Faster: just the C source
scripts/mutation_test.py --target src/error.c
# See what would run without building anything
scripts/mutation_test.py --target src/error.c --list
# Gate CI: exit non-zero if the score drops below 90%
scripts/mutation_test.py --threshold 90
```
Via CMake (configures a build first if needed):
```sh
cmake --build build --target mutation
```
Useful flags: `--timeout SECONDS` (per-suite build+test cap; a mutant that
hangs is counted as killed), `--keep` (retain the scratch copy for debugging),
`--work DIR` (use a specific scratch directory).
## Mutation operators
Each mutant changes exactly one location by one of:
| Tag | Operator | Example |
|-----|--------------------------------|----------------------------------|
| ROR | relational operator | `==``!=`, `<``<=`, `>=``>` |
| LCR | logical connector | `&&``\|\|` |
| BCR | boolean constant | `true``false` |
| AOR | arithmetic / compound assign | `+``-`, `+=``-=` |
| ICR | integer literal | `0``1`, `1``0` |
| SDL | statement deletion | `err->refcount += 1;`*(removed)* |
Preprocessor control lines, comments, and the block of error-code / buffer-size
`#define`s are skipped: mutating those produces equivalent or uninteresting
mutants that only add noise.
## Interpreting survivors
Not every survivor is a test gap — some mutants are **equivalent** (they don't
change observable behaviour, e.g. resizing an internal scratch buffer). For each
survivor, decide:
1. **Real gap** → add or strengthen a test in `tests/` so the mutant is killed,
then re-run.
2. **Equivalent mutant** → no test can catch it; leave a note. If a specific
line is a persistent source of equivalents, narrow the target with
`--target` or extend the skip rules in `scripts/mutation_test.py`.
Re-run after adding tests and confirm the score went up.
## Current status
`src/error.c` scores ~71% (the CI gate is set to 65% for headroom). The
remaining survivors are dominated by:
* **Equivalent mutants** in `akerr_init`: deleting the `memset`/`NULL` setup of
file-scope statics (`AKERR_ARRAY_ERROR`, `__akerr_last_ditch`,
`__akerr_last_ignored`) changes nothing, because C already zero-initializes
objects with static storage duration. `int oldid = 0;``1` is likewise
dead: it is overwritten before use.
* **Default logger / handler internals** (`vfprintf`, `va_end`, the
`errctx == NULL` branch, `exit(1)`): killing these needs a subprocess-based
test that captures a child's stderr and exit code, rather than the in-process
capturing logger the other tests use.
Findings worth noting (surfaced by mutation testing, not yet fixed):
* `AKERR_MAX_ERR_VALUE` is `AKERR_LAST_ERRNO_VALUE + 15`, but `AKERR_NOT_IMPLEMENTED`
(+16) and `AKERR_BADEXC` (+17) exceed it. `akerr_name_for_status` rejects any
status `> AKERR_MAX_ERR_VALUE`, so those two codes can never store or return a
name — the `akerr_name_for_status(AKERR_BADEXC, ...)` call in `akerr_init` is
dead code (which is why deleting it survives). Bumping `AKERR_MAX_ERR_VALUE` to
`+ 17` would fix it.