Add mutation testing, and split the release gates into their own workflow
Ports libakstdlib's mutation harness (the newest of the three) to scripts/
mutation_test.py, adds the namespaced `mutation` CMake target the sibling
libraries have, and splits CI in two.
.gitea/workflows/ci.yaml keeps the push path: suite, ASan+UBSan, coverage, and a
mutation run bounded to src/convert.c and src/symtab.c -- about four minutes,
scoring 77.8% against a gate of 65.
.gitea/workflows/release.yaml is new and manual (workflow_dispatch). It carries
the doxygen gate, moved out of ci.yaml, and a whole-tree mutation run: 3675
mutants and hours of runner time, which is a release cost rather than a
per-commit one. Both artifacts a release wants -- api-documentation and
mutation-report -- come out of it. Two optional inputs narrow the run or change
the threshold; they arrive through the environment rather than being
interpolated into the shell, because ${{ }} substitution happens before the
shell sees the line.
The harness paid for itself immediately, which is the point of it. Three real
gaps, each checked to be a genuine bug rather than an equivalent mutant:
- errno was never asserted clear before a strtoll. Confirmed with a standalone
probe that strtoll leaves a stale errno untouched on success, so without the
`errno = 0` a valid conversion raises ERANGE.
- Nothing exercised a maximum-length symbol-table key, so every MAX_KEY - 1
off-by-one in a strncpy and its NUL terminator survived. The same hole exists
for strings in src/value.c and is filed.
- Nothing asserted a freshly initialised table was actually zeroed.
Closing the first two took the measured score from 73.1% to 77.8%.
I set the push-path threshold to 75 first, on an estimate. Measuring gave 73.1%
and the job would have failed on its first run -- the earlier per-file figure was
too high because the captured output had been truncated to its last lines and I
counted fewer survivors than there were. It is 65 now, and the gate was run as
written and confirmed to exit 0.
src/value.c is the file most worth mutating and is deliberately off the push
path: 368 mutants at ~11s each is about 70 minutes, because almost everything
links against it. A partial run over it found the same maximum-length-string
hole plus two genuinely equivalent mutants that only exist because of reference
defect section 6 item 5 -- adding both of the right operand's numeric fields
works only while the unused one is zero, so + and - are interchangeable there.
Recorded in TODO.md.
ctest 61/61; doxygen exits 0; both workflows' steps were executed locally, both
input paths included.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
49
TODO.md
49
TODO.md
@@ -621,6 +621,7 @@ entire test corpus and passes clean under ASan and UBSan.
|
||||
| Function coverage | 96.9% (221/228) |
|
||||
| Warnings | none under `-Wall -Wextra` |
|
||||
| `doxygen Doxyfile` | clean; 114/114 public declarations documented |
|
||||
| Mutation (`src/convert.c`, `src/symtab.c`) | see `.gitea/workflows/ci.yaml` for the current score and gate |
|
||||
|
||||
Branch coverage reads 17.3% and is not a target, for the reason `libakgl/TODO.md` and
|
||||
`libakstdlib/TODO.md` both give: the akerror control-flow macros expand into large branch trees
|
||||
@@ -635,17 +636,27 @@ Dependency baseline:
|
||||
| `deps/libakstdlib` | 0.1.0 | soname `libakstdlib.so.0.1`. `AKSL_VERSION_CHECK()` asserted in `tests/version_check.c`. Its `aksl_ato*` family is banned here — see §1.9. |
|
||||
| `deps/libakgl` | 0.1.0 | Migrated to the 1.0.0 registry and owns 256–260. Not yet linked: `AKBASIC_WITH_AKGL` defaults OFF and `src/sink_akgl.c` does not exist. |
|
||||
|
||||
CI is `.gitea/workflows/ci.yaml`, in the shape the sibling libraries use: four jobs covering
|
||||
the suite, the doxygen gate, ASan+UBSan and coverage. The generated API documentation and the
|
||||
coverage report are both uploaded as artifacts. Two notes on it worth keeping:
|
||||
CI is split in two. `.gitea/workflows/ci.yaml` runs on push -- the suite, ASan+UBSan, coverage
|
||||
and a two-file mutation run -- and `.gitea/workflows/release.yaml` is manual
|
||||
(`workflow_dispatch`), carrying the doxygen gate and the whole-tree mutation run. The split is
|
||||
about cost: the full mutation set is 3675 mutants and hours of runner time, which is a release
|
||||
gate rather than a per-commit one, and the docs are wanted at release time. Three notes worth
|
||||
keeping:
|
||||
|
||||
- The checkout is `submodules: true`, **not** `recursive`. The build needs `libakerror`,
|
||||
`libakstdlib` and `basicinterpret`; it does not need `libakgl`, which is guarded behind
|
||||
`AKBASIC_WITH_AKGL` and defaults OFF — and recursing into it would clone SDL, SDL_image,
|
||||
SDL_mixer, SDL_ttf and jansson for a target that is never configured. The `docs` job needs
|
||||
no submodules at all.
|
||||
- There is no mutation-testing job, unlike `libakerror`, `libakstdlib` and `libakgl`. This
|
||||
repository has no `scripts/mutation_test.py`; see the item below.
|
||||
- The push-path `mutation_test` job is bounded to `src/convert.c` and `src/symtab.c`, about
|
||||
four minutes between them, scoring 77.8% against a gate of 65. `src/value.c` is the file most
|
||||
worth mutating and is deliberately not on that path: 368 mutants at roughly 11 seconds each
|
||||
is about 70 minutes, because almost everything links against it. It is covered by
|
||||
`release.yaml`, or locally with `cmake --build build --target mutation`.
|
||||
- `release.yaml`'s threshold defaults to 65, the same as the push path, rather than something
|
||||
stricter. Only two files have ever been measured; a tighter whole-tree number would be a
|
||||
guess until a full run establishes a baseline. Raise it through the workflow input once one
|
||||
has.
|
||||
|
||||
What remains, in priority order:
|
||||
|
||||
@@ -656,9 +667,25 @@ What remains, in priority order:
|
||||
and nothing consumes it) should come first, because `DO`/`LOOP` reads badly without it.
|
||||
3. **§6 items 12–17** — the defects the port uncovered. Item 12 is the one that produces a
|
||||
*wrong answer* rather than a refused one, and should be fixed first.
|
||||
4. **Mutation testing.** All three sibling libraries gate on it and this one does not, because
|
||||
there is no `scripts/mutation_test.py` here to run. It matters more than usual for this
|
||||
codebase: the akerror macros expand at their call sites, so line coverage cannot see them
|
||||
and mutation testing is the only thing that checks the `ATTEMPT`/`CATCH`/`PASS` control
|
||||
flow at all. `deps/libakerror/scripts/mutation_test.py` is the one to port; `src/value.c`
|
||||
or `src/scanner.c` would be the right first target.
|
||||
4. **Mutation survivors in `src/value.c`.** The harness is in place (`scripts/mutation_test.py`,
|
||||
the `mutation` CMake target, and a CI job on `src/convert.c`), and a partial run over
|
||||
`src/value.c` turned up test gaps worth closing. These are *not* equivalent mutants; each
|
||||
is a real bug of that shape the suite would not notice:
|
||||
|
||||
- `AKBASIC_MAX_STRING_LENGTH - 1` → `+ 1` and → `- 0` survive at `src/value.c:47-48`, in
|
||||
`set_string`'s `strncpy` and its NUL terminator. Nothing in the suite writes a
|
||||
*maximum-length* string, so the off-by-one that would truncate or overrun goes unseen.
|
||||
One test that round-trips a 255-character string kills all five.
|
||||
- `memset(obj, 0, ...)` → `memset(obj, 1, ...)` and its deletion survive in
|
||||
`akbasic_valuepool_init`, as does `obj->next = 0` → `= 1`. Nothing asserts a freshly
|
||||
initialised pool is actually empty and zeroed.
|
||||
|
||||
Two survivors there are genuinely equivalent and cannot be killed: `rval_as_int` and
|
||||
`rval_as_float` (`src/value.c:30,35`) tolerate `+` → `-` because the reference's habit of
|
||||
adding *both* of the right operand's numeric fields only works when the unused one is zero
|
||||
— §6 item 5. Fixing that defect would make these two mutants killable, which is a small
|
||||
argument for doing it.
|
||||
|
||||
`src/value.c` takes about 70 minutes to mutate in full (368 mutants, ~11s each, because
|
||||
almost everything links against it), which is why CI runs `src/convert.c` instead and the
|
||||
whole-tree run is a local `cmake --build build --target mutation`.
|
||||
|
||||
Reference in New Issue
Block a user