Record the mutation score and the concurrency mutants it misses
Some checks failed
libakerror CI Build / cmake_build (push) Successful in 2m48s
libakerror CI Build / coverage (push) Successful in 2m47s
libakerror CI Build / thread_sanitizer (push) Failing after 2m49s
libakerror CI Build / mutation_test (push) Successful in 38m22s

src/error.c now scores 81.2%: 238 of 293 mutants killed, 204 by a
failing test, 24 by failing to compile, and 10 by hanging the suite --
deleting akerr_mutex_init() or the akerr_initializing re-entry guard
deadlocks the first test, which is the right answer for a broken lock.

Lock deletions are the one survivor category where surviving does not
mean harmless, so measure it rather than assume: rebuilt, the surviving
"delete the pool lock" mutant fails tests/err_threads_pool.c in 4 of 10
runs and fails under scripts/thread_test.sh in 5 of 5, with no false
positive on the unmutated library. The property assertions alone are a
coin flip on a missing lock; the sanitizer run is what holds that line.
The harness builds mutants with default CMake options and so never sees
it -- TODO item 8.

Also warn that a sanitized test binary run by hand does not inherit the
halt_on_error CTest gives it, and will print a race and still exit 0.
That is how the 5-of-5 above first read as 2 of 5.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-07-31 08:45:20 -04:00
parent be24f80022
commit 756933c600
3 changed files with 51 additions and 5 deletions

View File

@@ -94,14 +94,35 @@ Re-run after adding tests and confirm the score went up.
## Current status
`src/error.c` scores ~77% (the CI gate is set to 65% for headroom). The
remaining survivors are dominated by:
`src/error.c` scores 81.2% — 238 of 293 mutants killed (204 by a failing test,
24 by failing to compile, 10 by hanging the suite), 55 surviving. The CI gate is
set to 65% for headroom.
The ten timeout kills are all in the locking: deleting `akerr_mutex_init()` or
the `akerr_initializing` re-entry guard deadlocks the very first test, which is
the correct behaviour for a broken lock and is why the harness counts a hang as
a kill.
The remaining survivors are dominated by:
* **Equivalent mutants** in `akerr_init`: deleting the `memset`/`NULL` setup of
file-scope statics (`AKERR_ARRAY_ERROR`, `__akerr_last_ditch`,
`__akerr_last_ignored`) changes nothing, because C already zero-initializes
objects with static storage duration. `int oldid = 0;``1` is likewise
dead: it is overwritten before use.
dead: it is overwritten before use, and so is clearing `akerr_initializing`
at the end of initialization — nothing reads that flag once the once-routine
has returned.
* **Lock acquisition** (`akerr_mutex_lock`/`unlock` deletions, and the
`akerr_init()` call at the head of an entry point). These are the one category
where a survivor does *not* mean the mutant is harmless. Removing a lock
leaves a real race, and the assertions in `tests/err_threads_pool.c` only fire
when the race actually loses: rebuilding the surviving mutant and running that
test ten times caught it **four** times. The same mutant under
`scripts/thread_test.sh` failed **five of five**, with no false positive on
the unmutated library — but the mutation harness builds without sanitizers, so
it never sees that. Deleting an `akerr_init()` call survives for a duller
reason: something else has always initialized the library by the time that
line runs.
* **Default logger / handler internals** (`vfprintf`, `va_end`, the
`errctx == NULL` branch, `exit(1)`): killing these needs a subprocess-based
test that captures a child's stderr and exit code, rather than the in-process
@@ -119,6 +140,13 @@ remaining survivors are dominated by:
Findings surfaced by mutation testing:
* **Open:** the harness builds every mutant with the default CMake options, so a
mutant that only breaks under concurrency is judged by a suite running without
ThreadSanitizer. Mutating under `-DAKERR_SANITIZE=thread` would close that,
and needs a way to pass CMake options through to the mutant build. See
"Mutation testing judges concurrency mutants without a sanitizer" in
`TODO.md`.
* **Superseded:** status names now use a private sparse registry, so the old
public `AKERR_MAX_ERR_VALUE` ceiling and its consumer ABI mismatch no longer
exist. `tests/err_maxval.c` covers arbitrary `int` values and registry