Gate mutation testing on a measured score, not an inherited one
All checks were successful
All checks were successful
The threshold had been 80 against src/stdlib.c alone. There are three more sources now and nobody had measured them, so that number was a guess carried forward. Measured: 72.3%, 188 of a 260-mutant sample from the 1701 the four sources generate. Gate set to 65 -- a ratchet with headroom for the runner and for the sample shifting as sources change, not a target. A sample rather than the whole set, because 1701 rebuilds and test runs is hours. --max-mutants samples by even index rather than at random, so the same 260 run every time and the gate stays reproducible; sampling all four files beats exhausting one of them, which is what this job did before. 72.3% against the 89.6% reported at 0.1.0 is a change in denominator, not a regression in the tests. That figure covered one 561-line file; this covers four totalling 1716 lines, and most of the added surface is argument validation whose mutants are frequently *equivalent* -- 12 of the 72 survivors are `errno = 0` deleted from a wrapper whose libc call always sets errno, which no test that could be written would catch. The README breaks all 72 down and says which are worth acting on; TODO.md 2.4 carries the three clusters that are. Two of them were real and are fixed here and in the previous commit: the right child's `depth + 1` in the depth-first walk, and aksl_tree_remove on an empty tree, which without its guard dereferences NULL. Neither had a test; both do now. That is what the harness is for. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
56
README.md
56
README.md
@@ -424,23 +424,49 @@ run prints every survivor with `file:line` and the exact edit. The harness never
|
||||
touches your working tree — it copies the repo to a scratch directory and mutates
|
||||
the copy.
|
||||
|
||||
CI runs the `src/stdlib.c` set with `--threshold 80`. That is a regression ratchet
|
||||
rather than a quality bar: the current score is **89.6% (155/173 killed)**, up
|
||||
from 46.8% before the wrapper tests landed. Raise the threshold as the remaining
|
||||
survivors are turned into assertions.
|
||||
CI runs a 260-mutant sample across all four sources with `--threshold 65`.
|
||||
A sample, because 1701 mutants each needing a full rebuild and test run is
|
||||
hours — and `--max-mutants` samples by *even index*, not at random, so the same
|
||||
260 run every time and the gate is reproducible. Sampling all four files beats
|
||||
exhausting one of them, which is what this job used to do.
|
||||
|
||||
The 18 survivors cluster in three places, and each names a real gap rather than a
|
||||
test-harness artifact:
|
||||
**Where it stands: 72.3% (188/260 killed).** That is a ratchet with headroom,
|
||||
not a target.
|
||||
|
||||
- **Statements whose absence nothing observes** — deleting `free(ptr)`,
|
||||
`obj->next = NULL`, or a `SUCCEED_RETURN` leaves behaviour the suite does not
|
||||
look at (a leak, a stale pointer, a success that was already NULL).
|
||||
- **The `aksl_list_append` cycle/tail walk** (`tail = slow`, `slow = slow->next`,
|
||||
`tail = fast`) — the function is broken in exactly this area (`TODO.md` §2.1.1),
|
||||
so its known-failing test cannot pin the internals yet.
|
||||
- **`lalloc`/`lfree` defaulting in `aksl_tree_iterate`** — dead parameters
|
||||
(§2.2.8): they are defaulted and then never called, so inverting the guard
|
||||
changes nothing observable.
|
||||
It is well below the 89.6% this project reported at 0.1.0, and the difference is
|
||||
denominator rather than tests. That figure covered one 561-line file; this covers
|
||||
four totalling 1716 lines, and most of the new surface is argument validation
|
||||
whose mutants are frequently *equivalent* — a mutation that cannot change
|
||||
observable behaviour, so no test could ever kill it. The clearest example:
|
||||
|
||||
```c
|
||||
errno = 0; /* delete this line */
|
||||
*dst = malloc(size);
|
||||
FAIL_ZERO_RETURN(e, *dst, AKSL_ERRNO_OR(ENOMEM), "%zu bytes", size);
|
||||
```
|
||||
|
||||
Deleting the `errno = 0` is undetectable, because `malloc` always sets `errno`
|
||||
when it fails. The line is still right to have — it is what makes
|
||||
`AKSL_ERRNO_OR`'s contract sound, and it matters for the calls that *don't* set
|
||||
`errno` — but no test distinguishes the two versions. Twelve of the 72 survivors
|
||||
are that line in twelve different wrappers.
|
||||
|
||||
The survivors do break down usefully:
|
||||
|
||||
| Survivors | What | Verdict |
|
||||
|---|---|---|
|
||||
| 12 | `errno = 0` deleted before a call that always sets `errno` | Equivalent. Not a missing test. |
|
||||
| 14 | `FAIL_*` guards deleted or their constants shifted | Mixed — the constant shifts are undetectable where the test names the same constant symbolically; the deletions are real. |
|
||||
| 7 | `SUCCEED_RETURN` deleted | The function falls off the end and returns whatever is in the return register, which is often NULL by luck. Needs an assertion on a side effect, not on the status. |
|
||||
| 3 | `va_end` deleted | Undetectable on x86-64 SysV, where `va_end` is a no-op. Real UB, invisible here. |
|
||||
| 3 | `FINISH(e, true)` → `FINISH(e, false)` | Real: an error swallowed instead of propagated. Worth a test. |
|
||||
| 33 | the rest | Individually listed with `file:line` and the exact edit in the published report. |
|
||||
|
||||
Two of them were real gaps and are now fixed: the depth-first walk's `depth + 1`
|
||||
on the *right* child (nothing had ever recursed right more than three deep, so a
|
||||
right-leaning tree would have blown the stack the depth cap exists to protect),
|
||||
and `aksl_tree_remove` on an empty tree, which without its guard dereferences
|
||||
NULL. Both are in the suite now — which is what the harness is for.
|
||||
|
||||
## The pre-push hook
|
||||
|
||||
|
||||
Reference in New Issue
Block a user