Gate mutation testing on a measured score, not an inherited one
All checks were successful
libakstdlib CI Build / cmake_build (push) Successful in 2m53s
libakstdlib CI Build / sanitizers (push) Successful in 2m51s
libakstdlib CI Build / coverage (push) Successful in 2m43s
libakstdlib CI Build / mutation_test (push) Successful in 12m37s

The threshold had been 80 against src/stdlib.c alone. There are three
more sources now and nobody had measured them, so that number was a
guess carried forward.

Measured: 72.3%, 188 of a 260-mutant sample from the 1701 the four
sources generate. Gate set to 65 -- a ratchet with headroom for the
runner and for the sample shifting as sources change, not a target.

A sample rather than the whole set, because 1701 rebuilds and test runs
is hours. --max-mutants samples by even index rather than at random, so
the same 260 run every time and the gate stays reproducible; sampling
all four files beats exhausting one of them, which is what this job did
before.

72.3% against the 89.6% reported at 0.1.0 is a change in denominator,
not a regression in the tests. That figure covered one 561-line file;
this covers four totalling 1716 lines, and most of the added surface is
argument validation whose mutants are frequently *equivalent* -- 12 of
the 72 survivors are `errno = 0` deleted from a wrapper whose libc call
always sets errno, which no test that could be written would catch. The
README breaks all 72 down and says which are worth acting on; TODO.md
2.4 carries the three clusters that are.

Two of them were real and are fixed here and in the previous commit: the
right child's `depth + 1` in the depth-first walk, and aksl_tree_remove
on an empty tree, which without its guard dereferences NULL. Neither had
a test; both do now. That is what the harness is for.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-07-31 08:15:08 -04:00
parent cc5e7899bb
commit f8425b8729
5 changed files with 120 additions and 26 deletions

View File

@@ -136,21 +136,37 @@ jobs:
sudo apt-get update -y
sudo apt-get install -y cmake gcc moreutils python3
# Verify the tests actually catch bugs: break the library many ways and
# confirm the suite fails. Gated on src/stdlib.c (fast, deterministic);
# run the full default target locally for the macro header as well.
# confirm the suite fails.
#
# The threshold is a ratchet, not a quality bar. The score is 89.6%
# (155/173 killed) now that TODO.md sections 1.2-1.6 have tests; it was
# 46.8% when only the list and tree functions were covered. 80 leaves
# headroom for the runner while still failing on a real regression (tests
# deleted, or new untested code added). The 18 survivors are listed in the
# published report -- each one is a missing assertion.
# A sample rather than the whole set. The four sources generate 1701
# mutants and each one is a full rebuild and re-run of the suite, which is
# hours. --max-mutants samples by even index, not at random, so the same
# 260 run every time and the gate is reproducible -- and sampling all four
# files beats exhausting one of them, which is what this job used to do.
#
# The threshold is a ratchet, not a quality bar. The measured score is
# 72.3% (188/260). 65 leaves headroom for the runner and for the sample
# shifting as sources change, while still failing on a real regression --
# a test deleted, or new untested code added.
#
# It is well below the 89.6% this job reported at 0.1.0, and that is a
# change in denominator rather than in the tests: that figure covered one
# 561-line file, this one covers four totalling 1716 lines, most of the
# new surface being argument validation whose mutants are frequently
# equivalent. `errno = 0` deleted from a wrapper whose libc call always
# sets errno cannot be distinguished by any test that could be written.
# The survivors worth acting on are named in TODO.md; raise the gate as
# they become assertions.
- name: mutation testing
run: |
python3 scripts/mutation_test.py \
--target src/stdlib.c \
--target src/string.c \
--target src/stream.c \
--target src/collections.c \
--max-mutants 260 \
--junit mutation-junit.xml \
--threshold 80
--threshold 65
# Publish even when the threshold gate fails, so survivors are visible --
# each one is a missing test. Display-only (fail_on_failure: false); the
# --threshold above is the gate. annotate_only avoids the Checks API 404