Add mutation testing, and split the release gates into their own workflow

Ports libakstdlib's mutation harness (the newest of the three) to scripts/
mutation_test.py, adds the namespaced `mutation` CMake target the sibling
libraries have, and splits CI in two.

.gitea/workflows/ci.yaml keeps the push path: suite, ASan+UBSan, coverage, and a
mutation run bounded to src/convert.c and src/symtab.c -- about four minutes,
scoring 77.8% against a gate of 65.

.gitea/workflows/release.yaml is new and manual (workflow_dispatch). It carries
the doxygen gate, moved out of ci.yaml, and a whole-tree mutation run: 3675
mutants and hours of runner time, which is a release cost rather than a
per-commit one. Both artifacts a release wants -- api-documentation and
mutation-report -- come out of it. Two optional inputs narrow the run or change
the threshold; they arrive through the environment rather than being
interpolated into the shell, because ${{ }} substitution happens before the
shell sees the line.

The harness paid for itself immediately, which is the point of it. Three real
gaps, each checked to be a genuine bug rather than an equivalent mutant:

  - errno was never asserted clear before a strtoll. Confirmed with a standalone
    probe that strtoll leaves a stale errno untouched on success, so without the
    `errno = 0` a valid conversion raises ERANGE.
  - Nothing exercised a maximum-length symbol-table key, so every MAX_KEY - 1
    off-by-one in a strncpy and its NUL terminator survived. The same hole exists
    for strings in src/value.c and is filed.
  - Nothing asserted a freshly initialised table was actually zeroed.

Closing the first two took the measured score from 73.1% to 77.8%.

I set the push-path threshold to 75 first, on an estimate. Measuring gave 73.1%
and the job would have failed on its first run -- the earlier per-file figure was
too high because the captured output had been truncated to its last lines and I
counted fewer survivors than there were. It is 65 now, and the gate was run as
written and confirmed to exit 0.

src/value.c is the file most worth mutating and is deliberately off the push
path: 368 mutants at ~11s each is about 70 minutes, because almost everything
links against it. A partial run over it found the same maximum-length-string
hole plus two genuinely equivalent mutants that only exist because of reference
defect section 6 item 5 -- adding both of the right operand's numeric fields
works only while the unused one is zero, so + and - are interchangeable there.
Recorded in TODO.md.

ctest 61/61; doxygen exits 0; both workflows' steps were executed locally, both
input paths included.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-07-31 07:46:53 -04:00
parent 2ddee571a1
commit 506af093a1
8 changed files with 828 additions and 48 deletions

View File

@@ -18,9 +18,31 @@ ctest --test-dir build --output-on-failure
doxygen Doxyfile
```
CI is `.gitea/workflows/ci.yaml` and runs all four of those: the suite, the doxygen gate,
ASan+UBSan, and coverage gated at 90% of lines. The generated API documentation and the
coverage report are uploaded as build artifacts (`api-documentation` and `code-coverage`).
There are two workflows. **`.gitea/workflows/ci.yaml`** runs on every push: the suite,
ASan+UBSan, coverage gated at 90% of lines, and mutation testing over two files. The coverage
report is uploaded as a `code-coverage` artifact.
**`.gitea/workflows/release.yaml`** is manual (`workflow_dispatch`) and is what a release runs.
It builds the API documentation and uploads it as `api-documentation`, and it mutates the
*whole* `src/` tree — 3675 mutants, hours of runner time, which is why it is not on the push
path. It takes two optional inputs: a mutation threshold, and a space-separated file list to
narrow the run.
Mutation testing is worth a word, because it is the only gate that checks the error-handling
control flow at all. `libakerror`'s `ATTEMPT`/`CATCH`/`PASS` macros expand at their call sites,
so gcov attributes them to the caller and line coverage cannot see them. `scripts/mutation_test.py`
breaks the library many small ways and checks that the suite notices:
```sh
cmake --build build --target mutation # the whole src/ tree; slow, hours
python3 scripts/mutation_test.py --target src/value.c --list
python3 scripts/mutation_test.py --target src/value.c --threshold 70
```
It earns its keep. Writing this suite, it found that nothing exercised a maximum-length string
or symbol-table key, so every `MAX - 1` off-by-one in a `strncpy` would have gone unnoticed;
and that `errno` was never asserted to be cleared before a `strtoll`, which is what stops a
stale `ERANGE` from failing a perfectly valid conversion.
The `Doxyfile` is configured the way `libakgl`'s is, including
`WARN_AS_ERROR = FAIL_ON_WARNINGS` — a doc block that documents some of a function's