Record the mutation score and the concurrency mutants it misses
src/error.c now scores 81.2%: 238 of 293 mutants killed, 204 by a failing test, 24 by failing to compile, and 10 by hanging the suite -- deleting akerr_mutex_init() or the akerr_initializing re-entry guard deadlocks the first test, which is the right answer for a broken lock. Lock deletions are the one survivor category where surviving does not mean harmless, so measure it rather than assume: rebuilt, the surviving "delete the pool lock" mutant fails tests/err_threads_pool.c in 4 of 10 runs and fails under scripts/thread_test.sh in 5 of 5, with no false positive on the unmutated library. The property assertions alone are a coin flip on a missing lock; the sanitizer run is what holds that line. The harness builds mutants with default CMake options and so never sees it -- TODO item 8. Also warn that a sanitized test binary run by hand does not inherit the halt_on_error CTest gives it, and will print a race and still exit 0. That is how the 5-of-5 above first read as 2 of 5. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -94,14 +94,35 @@ Re-run after adding tests and confirm the score went up.
|
||||
|
||||
## Current status
|
||||
|
||||
`src/error.c` scores ~77% (the CI gate is set to 65% for headroom). The
|
||||
remaining survivors are dominated by:
|
||||
`src/error.c` scores 81.2% — 238 of 293 mutants killed (204 by a failing test,
|
||||
24 by failing to compile, 10 by hanging the suite), 55 surviving. The CI gate is
|
||||
set to 65% for headroom.
|
||||
|
||||
The ten timeout kills are all in the locking: deleting `akerr_mutex_init()` or
|
||||
the `akerr_initializing` re-entry guard deadlocks the very first test, which is
|
||||
the correct behaviour for a broken lock and is why the harness counts a hang as
|
||||
a kill.
|
||||
|
||||
The remaining survivors are dominated by:
|
||||
|
||||
* **Equivalent mutants** in `akerr_init`: deleting the `memset`/`NULL` setup of
|
||||
file-scope statics (`AKERR_ARRAY_ERROR`, `__akerr_last_ditch`,
|
||||
`__akerr_last_ignored`) changes nothing, because C already zero-initializes
|
||||
objects with static storage duration. `int oldid = 0;` → `1` is likewise
|
||||
dead: it is overwritten before use.
|
||||
dead: it is overwritten before use, and so is clearing `akerr_initializing`
|
||||
at the end of initialization — nothing reads that flag once the once-routine
|
||||
has returned.
|
||||
* **Lock acquisition** (`akerr_mutex_lock`/`unlock` deletions, and the
|
||||
`akerr_init()` call at the head of an entry point). These are the one category
|
||||
where a survivor does *not* mean the mutant is harmless. Removing a lock
|
||||
leaves a real race, and the assertions in `tests/err_threads_pool.c` only fire
|
||||
when the race actually loses: rebuilding the surviving mutant and running that
|
||||
test ten times caught it **four** times. The same mutant under
|
||||
`scripts/thread_test.sh` failed **five of five**, with no false positive on
|
||||
the unmutated library — but the mutation harness builds without sanitizers, so
|
||||
it never sees that. Deleting an `akerr_init()` call survives for a duller
|
||||
reason: something else has always initialized the library by the time that
|
||||
line runs.
|
||||
* **Default logger / handler internals** (`vfprintf`, `va_end`, the
|
||||
`errctx == NULL` branch, `exit(1)`): killing these needs a subprocess-based
|
||||
test that captures a child's stderr and exit code, rather than the in-process
|
||||
@@ -119,6 +140,13 @@ remaining survivors are dominated by:
|
||||
|
||||
Findings surfaced by mutation testing:
|
||||
|
||||
* **Open:** the harness builds every mutant with the default CMake options, so a
|
||||
mutant that only breaks under concurrency is judged by a suite running without
|
||||
ThreadSanitizer. Mutating under `-DAKERR_SANITIZE=thread` would close that,
|
||||
and needs a way to pass CMake options through to the mutant build. See
|
||||
"Mutation testing judges concurrency mutants without a sanitizer" in
|
||||
`TODO.md`.
|
||||
|
||||
* **Superseded:** status names now use a private sparse registry, so the old
|
||||
public `AKERR_MAX_ERR_VALUE` ceiling and its consumer ABI mismatch no longer
|
||||
exist. `tests/err_maxval.c` covers arbitrary `int` values and registry
|
||||
|
||||
Reference in New Issue
Block a user