diff --git a/AGENTS.md b/AGENTS.md index 735a651..369e3a9 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -40,8 +40,11 @@ which configures `build/tsan` with `-DAKERR_SANITIZE=thread`, builds the library *and* the tests with it, and runs CTest with ASLR disabled (TSan aborts with an "unexpected memory mapping" on kernels with `vm.mmap_rnd_bits` above 28). `AKERR_SANITIZE` takes any sanitizer list, so `-DAKERR_SANITIZE=address,undefined` -works the same way. Under a sanitized build a report fails the test rather than -being printed and passed over. +works the same way. CTest gives each test `halt_on_error=1`, so a report fails +the test rather than being printed and passed over — running a sanitized test +binary by hand does **not** inherit that. Set it yourself +(`TSAN_OPTIONS=halt_on_error=1 ./build/tsan/test_err_threads_pool`) or the +binary can print a race and still exit 0. Mutation testing is available through: diff --git a/TODO.md b/TODO.md index 19fb819..0c61eb0 100644 --- a/TODO.md +++ b/TODO.md @@ -97,6 +97,21 @@ fail carries about twenty-five. Validating more inputs therefore lowers the ratio by construction. Before adding defensive checks, expect to add a test that drives them, as `tests/err_copy_string.c` does. +## 8. Mutation testing judges concurrency mutants without a sanitizer + +`scripts/mutation_test.py` configures each mutant build with the default CMake +options, so a mutant that only breaks under concurrency is judged by a suite +running without ThreadSanitizer. Deleting the pool's `akerr_mutex_lock()` call +survives the run even though it is a real race: rebuilt and run directly, that +mutant fails `tests/err_threads_pool.c` in 4 of 10 runs, and fails under +`scripts/thread_test.sh` in 5 of 5. So 81.2% is a floor for that category, not a +verdict. + +Closing it means a `--cmake-arg` passthrough on the harness so the mutant build +can be configured with `-DAKERR_SANITIZE=thread`. The whole run then costs a +TSan-instrumented suite per mutant (roughly 6s instead of 0.4s), so it belongs +behind a flag rather than in the default target or in CI. + ## Unrelated pre-existing issues - The `AKERR_USE_STDLIB=OFF` build does not compile at all: `bool`, `PATH_MAX` diff --git a/tests/MUTATION.md b/tests/MUTATION.md index 05defb4..963ca75 100644 --- a/tests/MUTATION.md +++ b/tests/MUTATION.md @@ -94,14 +94,35 @@ Re-run after adding tests and confirm the score went up. ## Current status -`src/error.c` scores ~77% (the CI gate is set to 65% for headroom). The -remaining survivors are dominated by: +`src/error.c` scores 81.2% — 238 of 293 mutants killed (204 by a failing test, +24 by failing to compile, 10 by hanging the suite), 55 surviving. The CI gate is +set to 65% for headroom. + +The ten timeout kills are all in the locking: deleting `akerr_mutex_init()` or +the `akerr_initializing` re-entry guard deadlocks the very first test, which is +the correct behaviour for a broken lock and is why the harness counts a hang as +a kill. + +The remaining survivors are dominated by: * **Equivalent mutants** in `akerr_init`: deleting the `memset`/`NULL` setup of file-scope statics (`AKERR_ARRAY_ERROR`, `__akerr_last_ditch`, `__akerr_last_ignored`) changes nothing, because C already zero-initializes objects with static storage duration. `int oldid = 0;` → `1` is likewise - dead: it is overwritten before use. + dead: it is overwritten before use, and so is clearing `akerr_initializing` + at the end of initialization — nothing reads that flag once the once-routine + has returned. +* **Lock acquisition** (`akerr_mutex_lock`/`unlock` deletions, and the + `akerr_init()` call at the head of an entry point). These are the one category + where a survivor does *not* mean the mutant is harmless. Removing a lock + leaves a real race, and the assertions in `tests/err_threads_pool.c` only fire + when the race actually loses: rebuilding the surviving mutant and running that + test ten times caught it **four** times. The same mutant under + `scripts/thread_test.sh` failed **five of five**, with no false positive on + the unmutated library — but the mutation harness builds without sanitizers, so + it never sees that. Deleting an `akerr_init()` call survives for a duller + reason: something else has always initialized the library by the time that + line runs. * **Default logger / handler internals** (`vfprintf`, `va_end`, the `errctx == NULL` branch, `exit(1)`): killing these needs a subprocess-based test that captures a child's stderr and exit code, rather than the in-process @@ -119,6 +140,13 @@ remaining survivors are dominated by: Findings surfaced by mutation testing: +* **Open:** the harness builds every mutant with the default CMake options, so a + mutant that only breaks under concurrency is judged by a suite running without + ThreadSanitizer. Mutating under `-DAKERR_SANITIZE=thread` would close that, + and needs a way to pass CMake options through to the mutant build. See + "Mutation testing judges concurrency mutants without a sanitizer" in + `TODO.md`. + * **Superseded:** status names now use a private sparse registry, so the old public `AKERR_MAX_ERR_VALUE` ceiling and its consumer ABI mismatch no longer exist. `tests/err_maxval.c` covers arbitrary `int` values and registry