Gate mutation testing on a measured score, not an inherited one
All checks were successful
All checks were successful
The threshold had been 80 against src/stdlib.c alone. There are three more sources now and nobody had measured them, so that number was a guess carried forward. Measured: 72.3%, 188 of a 260-mutant sample from the 1701 the four sources generate. Gate set to 65 -- a ratchet with headroom for the runner and for the sample shifting as sources change, not a target. A sample rather than the whole set, because 1701 rebuilds and test runs is hours. --max-mutants samples by even index rather than at random, so the same 260 run every time and the gate stays reproducible; sampling all four files beats exhausting one of them, which is what this job did before. 72.3% against the 89.6% reported at 0.1.0 is a change in denominator, not a regression in the tests. That figure covered one 561-line file; this covers four totalling 1716 lines, and most of the added surface is argument validation whose mutants are frequently *equivalent* -- 12 of the 72 survivors are `errno = 0` deleted from a wrapper whose libc call always sets errno, which no test that could be written would catch. The README breaks all 72 down and says which are worth acting on; TODO.md 2.4 carries the three clusters that are. Two of them were real and are fixed here and in the previous commit: the right child's `depth + 1` in the depth-first walk, and aksl_tree_remove on an empty tree, which without its guard dereferences NULL. Neither had a test; both do now. That is what the harness is for. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -136,21 +136,37 @@ jobs:
|
||||
sudo apt-get update -y
|
||||
sudo apt-get install -y cmake gcc moreutils python3
|
||||
# Verify the tests actually catch bugs: break the library many ways and
|
||||
# confirm the suite fails. Gated on src/stdlib.c (fast, deterministic);
|
||||
# run the full default target locally for the macro header as well.
|
||||
# confirm the suite fails.
|
||||
#
|
||||
# The threshold is a ratchet, not a quality bar. The score is 89.6%
|
||||
# (155/173 killed) now that TODO.md sections 1.2-1.6 have tests; it was
|
||||
# 46.8% when only the list and tree functions were covered. 80 leaves
|
||||
# headroom for the runner while still failing on a real regression (tests
|
||||
# deleted, or new untested code added). The 18 survivors are listed in the
|
||||
# published report -- each one is a missing assertion.
|
||||
# A sample rather than the whole set. The four sources generate 1701
|
||||
# mutants and each one is a full rebuild and re-run of the suite, which is
|
||||
# hours. --max-mutants samples by even index, not at random, so the same
|
||||
# 260 run every time and the gate is reproducible -- and sampling all four
|
||||
# files beats exhausting one of them, which is what this job used to do.
|
||||
#
|
||||
# The threshold is a ratchet, not a quality bar. The measured score is
|
||||
# 72.3% (188/260). 65 leaves headroom for the runner and for the sample
|
||||
# shifting as sources change, while still failing on a real regression --
|
||||
# a test deleted, or new untested code added.
|
||||
#
|
||||
# It is well below the 89.6% this job reported at 0.1.0, and that is a
|
||||
# change in denominator rather than in the tests: that figure covered one
|
||||
# 561-line file, this one covers four totalling 1716 lines, most of the
|
||||
# new surface being argument validation whose mutants are frequently
|
||||
# equivalent. `errno = 0` deleted from a wrapper whose libc call always
|
||||
# sets errno cannot be distinguished by any test that could be written.
|
||||
# The survivors worth acting on are named in TODO.md; raise the gate as
|
||||
# they become assertions.
|
||||
- name: mutation testing
|
||||
run: |
|
||||
python3 scripts/mutation_test.py \
|
||||
--target src/stdlib.c \
|
||||
--target src/string.c \
|
||||
--target src/stream.c \
|
||||
--target src/collections.c \
|
||||
--max-mutants 260 \
|
||||
--junit mutation-junit.xml \
|
||||
--threshold 80
|
||||
--threshold 65
|
||||
# Publish even when the threshold gate fails, so survivors are visible --
|
||||
# each one is a missing test. Display-only (fail_on_failure: false); the
|
||||
# --threshold above is the gate. annotate_only avoids the Checks API 404
|
||||
|
||||
Reference in New Issue
Block a user