Add mutation testing to the Gitea workflow

A separate mutation_test job runs scripts/mutation_test.py over src/stdlib.c
with a --threshold gate and publishes a JUnit report, mirroring libakerror's
CI. Checkout needs submodules: recursive -- the harness copies the repo and
builds the copy top-level, so deps/libakerror has to be there.

Measured score on the section 1.0 suite is 46.8% (81/173 killed). The gate is
set at 40: a regression ratchet, not a quality bar. Survivors cluster in the
wrappers that have no tests yet (printf family, ato* family, realpath, the
memory and stream wrappers); the list and tree code that section 1.0 does
cover leaves only 5 survivors between them.

Also caps every test with TIMEOUT 30. Measuring the score turned this up: a
mutant deleting `fast = fast->next->next` makes aksl_list_iterate spin
forever, so ctest never returned, the harness killed ctest at its own 120s
timeout, and the orphaned test binary kept burning a core while holding the
pipe open -- the run wedged and needed killing by hand. With a per-test
timeout ctest reaps its own child, those mutants are killed in 30s, and
nothing is left behind. It guards the real suite against the same shape of
bug too.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-07-29 10:35:46 -04:00
parent 3e296c3bff
commit 68009ea0e3
3 changed files with 86 additions and 8 deletions

28
TODO.md
View File

@@ -56,10 +56,30 @@ against `tests/aksl_capture.h` and added to the `AKSL_TESTS` list.
`aksl_realpath`, unbounded `vsprintf`, missing `va_end`).
- [x] **Port `scripts/mutation_test.py` from libakerror.** Retargeted at `src/stdlib.c` and
`include/akstdlib.h` (178 mutants) and exposed as
`cmake --build build --target mutation`. Deliberately *not* a CI gate yet, and no
`--threshold` is set: a mutation score only means something once §1.11.9 exist, and
against today's tests it would do little but confirm that most of the library is
untested.
`cmake --build build --target mutation`. CI runs the narrower `src/stdlib.c` set (173
mutants) as its own `mutation_test` job with `--threshold 40` and a JUnit report.
**Current score: 46.8%** — 81 killed, 92 survived. The survivors are concentrated
exactly where §1.11.9 have yet to be written:
| surviving mutants | function |
|---|---|
| 10 | `aksl_tree_iterate` |
| 6 each | `aksl_fread`, `aksl_fwrite`, `aksl_fprintf`, `aksl_sprintf` |
| 5 | `aksl_printf` |
| 4 each | `aksl_memcpy`, `aksl_realpath`, `aksl_strhash_djb2`, `aksl_list_append` |
| 3 each | `aksl_malloc`, `aksl_free`, `aksl_memset`, `aksl_fopen`, `aksl_fclose`, `aksl_atoi`, `aksl_atol`, `aksl_atoll`, `aksl_atof` |
| 1 | `aksl_list_iterate` |
The threshold is a regression ratchet, not a quality bar; raise it as sections below
land. Each surviving mutant is a concrete missing test — `mutation-junit.xml` lists
them with `file:line` and the exact edit.
- [x] **Cap every test with a CTest `TIMEOUT`.** Found while measuring the above: a mutant
that deletes `fast = fast->next->next` turns `aksl_list_iterate` into an infinite
loop, so `ctest` waited forever, the mutation harness killed `ctest` at its own
120 s timeout, and the orphaned test binary kept spinning at 100% CPU while holding
the pipe open — the run wedged and had to be killed by hand. `TIMEOUT 30` on every
test makes `ctest` reap its own child, so those mutants are now killed in 30 s and
leave nothing behind. It also guards the real suite against the same class of bug.
- [x] **CI could not have run any of this.** `actions/checkout@v4` was not fetching
submodules, so `add_subdirectory(deps/libakerror)` had nothing to descend into and the
configure step failed before ever reaching the tests. Added `submodules: recursive`,