Gate mutation testing on a measured score, not an inherited one
All checks were successful
All checks were successful
The threshold had been 80 against src/stdlib.c alone. There are three more sources now and nobody had measured them, so that number was a guess carried forward. Measured: 72.3%, 188 of a 260-mutant sample from the 1701 the four sources generate. Gate set to 65 -- a ratchet with headroom for the runner and for the sample shifting as sources change, not a target. A sample rather than the whole set, because 1701 rebuilds and test runs is hours. --max-mutants samples by even index rather than at random, so the same 260 run every time and the gate stays reproducible; sampling all four files beats exhausting one of them, which is what this job did before. 72.3% against the 89.6% reported at 0.1.0 is a change in denominator, not a regression in the tests. That figure covered one 561-line file; this covers four totalling 1716 lines, and most of the added surface is argument validation whose mutants are frequently *equivalent* -- 12 of the 72 survivors are `errno = 0` deleted from a wrapper whose libc call always sets errno, which no test that could be written would catch. The README breaks all 72 down and says which are worth acting on; TODO.md 2.4 carries the three clusters that are. Two of them were real and are fixed here and in the previous commit: the right child's `depth + 1` in the depth-first walk, and aksl_tree_remove on an empty tree, which without its guard dereferences NULL. Neither had a test; both do now. That is what the harness is for. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -30,7 +30,7 @@ set -u
|
||||
|
||||
# Keep in sync with the --threshold in .gitea/workflows/ci.yaml, so a push that
|
||||
# would fail CI fails here first.
|
||||
MUTATION_THRESHOLD="${AKSL_MUTATION_THRESHOLD:-80}"
|
||||
MUTATION_THRESHOLD="${AKSL_MUTATION_THRESHOLD:-65}"
|
||||
|
||||
ZERO_SHA=0000000000000000000000000000000000000000
|
||||
|
||||
@@ -102,11 +102,15 @@ if [ "${AKSL_HOOK_MUTATION:-0}" = "1" ]; then
|
||||
echo "pre-push: mutation testing, threshold ${MUTATION_THRESHOLD}% (this takes a while)"
|
||||
# Deliberately not wrapped in run(): this one is slow enough that you want
|
||||
# to watch it make progress.
|
||||
# Sampled, like CI: the full set is 1701 mutants and each is a rebuild plus
|
||||
# a full test run. --max-mutants samples by even index rather than at
|
||||
# random, so this is the same 260 CI gates on.
|
||||
if ! python3 scripts/mutation_test.py \
|
||||
--target src/stdlib.c \
|
||||
--target src/string.c \
|
||||
--target src/stream.c \
|
||||
--target src/collections.c \
|
||||
--max-mutants "${AKSL_HOOK_MUTANTS:-260}" \
|
||||
--threshold "$MUTATION_THRESHOLD"; then
|
||||
echo >&2
|
||||
echo "pre-push: mutation score below ${MUTATION_THRESHOLD}%. Push aborted." >&2
|
||||
|
||||
Reference in New Issue
Block a user