Gate mutation testing on a measured score, not an inherited one
All checks were successful
libakstdlib CI Build / cmake_build (push) Successful in 2m53s
libakstdlib CI Build / sanitizers (push) Successful in 2m51s
libakstdlib CI Build / coverage (push) Successful in 2m43s
libakstdlib CI Build / mutation_test (push) Successful in 12m37s

The threshold had been 80 against src/stdlib.c alone. There are three
more sources now and nobody had measured them, so that number was a
guess carried forward.

Measured: 72.3%, 188 of a 260-mutant sample from the 1701 the four
sources generate. Gate set to 65 -- a ratchet with headroom for the
runner and for the sample shifting as sources change, not a target.

A sample rather than the whole set, because 1701 rebuilds and test runs
is hours. --max-mutants samples by even index rather than at random, so
the same 260 run every time and the gate stays reproducible; sampling
all four files beats exhausting one of them, which is what this job did
before.

72.3% against the 89.6% reported at 0.1.0 is a change in denominator,
not a regression in the tests. That figure covered one 561-line file;
this covers four totalling 1716 lines, and most of the added surface is
argument validation whose mutants are frequently *equivalent* -- 12 of
the 72 survivors are `errno = 0` deleted from a wrapper whose libc call
always sets errno, which no test that could be written would catch. The
README breaks all 72 down and says which are worth acting on; TODO.md
2.4 carries the three clusters that are.

Two of them were real and are fixed here and in the previous commit: the
right child's `depth + 1` in the depth-first walk, and aksl_tree_remove
on an empty tree, which without its guard dereferences NULL. Neither had
a test; both do now. That is what the harness is for.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-07-31 08:15:08 -04:00
parent cc5e7899bb
commit f8425b8729
5 changed files with 120 additions and 26 deletions

View File

@@ -424,23 +424,49 @@ run prints every survivor with `file:line` and the exact edit. The harness never
touches your working tree — it copies the repo to a scratch directory and mutates
the copy.
CI runs the `src/stdlib.c` set with `--threshold 80`. That is a regression ratchet
rather than a quality bar: the current score is **89.6% (155/173 killed)**, up
from 46.8% before the wrapper tests landed. Raise the threshold as the remaining
survivors are turned into assertions.
CI runs a 260-mutant sample across all four sources with `--threshold 65`.
A sample, because 1701 mutants each needing a full rebuild and test run is
hours — and `--max-mutants` samples by *even index*, not at random, so the same
260 run every time and the gate is reproducible. Sampling all four files beats
exhausting one of them, which is what this job used to do.
The 18 survivors cluster in three places, and each names a real gap rather than a
test-harness artifact:
**Where it stands: 72.3% (188/260 killed).** That is a ratchet with headroom,
not a target.
- **Statements whose absence nothing observes** — deleting `free(ptr)`,
`obj->next = NULL`, or a `SUCCEED_RETURN` leaves behaviour the suite does not
look at (a leak, a stale pointer, a success that was already NULL).
- **The `aksl_list_append` cycle/tail walk** (`tail = slow`, `slow = slow->next`,
`tail = fast`) — the function is broken in exactly this area (`TODO.md` §2.1.1),
so its known-failing test cannot pin the internals yet.
- **`lalloc`/`lfree` defaulting in `aksl_tree_iterate`** — dead parameters
(§2.2.8): they are defaulted and then never called, so inverting the guard
changes nothing observable.
It is well below the 89.6% this project reported at 0.1.0, and the difference is
denominator rather than tests. That figure covered one 561-line file; this covers
four totalling 1716 lines, and most of the new surface is argument validation
whose mutants are frequently *equivalent* — a mutation that cannot change
observable behaviour, so no test could ever kill it. The clearest example:
```c
errno = 0; /* delete this line */
*dst = malloc(size);
FAIL_ZERO_RETURN(e, *dst, AKSL_ERRNO_OR(ENOMEM), "%zu bytes", size);
```
Deleting the `errno = 0` is undetectable, because `malloc` always sets `errno`
when it fails. The line is still right to have — it is what makes
`AKSL_ERRNO_OR`'s contract sound, and it matters for the calls that *don't* set
`errno` — but no test distinguishes the two versions. Twelve of the 72 survivors
are that line in twelve different wrappers.
The survivors do break down usefully:
| Survivors | What | Verdict |
|---|---|---|
| 12 | `errno = 0` deleted before a call that always sets `errno` | Equivalent. Not a missing test. |
| 14 | `FAIL_*` guards deleted or their constants shifted | Mixed — the constant shifts are undetectable where the test names the same constant symbolically; the deletions are real. |
| 7 | `SUCCEED_RETURN` deleted | The function falls off the end and returns whatever is in the return register, which is often NULL by luck. Needs an assertion on a side effect, not on the status. |
| 3 | `va_end` deleted | Undetectable on x86-64 SysV, where `va_end` is a no-op. Real UB, invisible here. |
| 3 | `FINISH(e, true)``FINISH(e, false)` | Real: an error swallowed instead of propagated. Worth a test. |
| 33 | the rest | Individually listed with `file:line` and the exact edit in the published report. |
Two of them were real gaps and are now fixed: the depth-first walk's `depth + 1`
on the *right* child (nothing had ever recursed right more than three deep, so a
right-leaning tree would have blown the stack the depth cap exists to protect),
and `aksl_tree_remove` on an empty tree, which without its guard dereferences
NULL. Both are in the suite now — which is what the harness is for.
## The pre-push hook