Add mutation testing, and split the release gates into their own workflow
Ports libakstdlib's mutation harness (the newest of the three) to scripts/
mutation_test.py, adds the namespaced `mutation` CMake target the sibling
libraries have, and splits CI in two.
.gitea/workflows/ci.yaml keeps the push path: suite, ASan+UBSan, coverage, and a
mutation run bounded to src/convert.c and src/symtab.c -- about four minutes,
scoring 77.8% against a gate of 65.
.gitea/workflows/release.yaml is new and manual (workflow_dispatch). It carries
the doxygen gate, moved out of ci.yaml, and a whole-tree mutation run: 3675
mutants and hours of runner time, which is a release cost rather than a
per-commit one. Both artifacts a release wants -- api-documentation and
mutation-report -- come out of it. Two optional inputs narrow the run or change
the threshold; they arrive through the environment rather than being
interpolated into the shell, because ${{ }} substitution happens before the
shell sees the line.
The harness paid for itself immediately, which is the point of it. Three real
gaps, each checked to be a genuine bug rather than an equivalent mutant:
- errno was never asserted clear before a strtoll. Confirmed with a standalone
probe that strtoll leaves a stale errno untouched on success, so without the
`errno = 0` a valid conversion raises ERANGE.
- Nothing exercised a maximum-length symbol-table key, so every MAX_KEY - 1
off-by-one in a strncpy and its NUL terminator survived. The same hole exists
for strings in src/value.c and is filed.
- Nothing asserted a freshly initialised table was actually zeroed.
Closing the first two took the measured score from 73.1% to 77.8%.
I set the push-path threshold to 75 first, on an estimate. Measuring gave 73.1%
and the job would have failed on its first run -- the earlier per-file figure was
too high because the captured output had been truncated to its last lines and I
counted fewer survivors than there were. It is 65 now, and the gate was run as
written and confirmed to exit 0.
src/value.c is the file most worth mutating and is deliberately off the push
path: 368 mutants at ~11s each is about 70 minutes, because almost everything
links against it. A partial run over it found the same maximum-length-string
hole plus two genuinely equivalent mutants that only exist because of reference
defect section 6 item 5 -- adding both of the right operand's numeric fields
works only while the unused one is zero, so + and - are interchangeable there.
Recorded in TODO.md.
ctest 61/61; doxygen exits 0; both workflows' steps were executed locally, both
input paths included.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -267,6 +267,38 @@ if(AKBASIC_GOLDEN_CASES)
|
||||
_set_tests_properties(${AKBASIC_GOLDEN_NAMES} PROPERTIES TIMEOUT 30)
|
||||
endif()
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Mutation testing.
|
||||
#
|
||||
# Coverage says which lines ran; mutation testing says whether anything would
|
||||
# have noticed if they were wrong. That distinction matters more here than in an
|
||||
# ordinary C library: the akerror control-flow macros expand at their call sites,
|
||||
# so gcov attributes ATTEMPT/CATCH/PASS to the caller and cannot really see them.
|
||||
# Mutation testing is the only thing that checks them at all.
|
||||
#
|
||||
# This target runs the whole akbasic-owned src/ tree and is slow -- upwards of an
|
||||
# hour. CI runs a narrower, faster set with a --threshold gate; see
|
||||
# .gitea/workflows/ci.yaml.
|
||||
#
|
||||
# Namespaced when embedded in another project, for the same reason the coverage
|
||||
# targets in the dependencies are: a sibling may well ship a `mutation` target.
|
||||
find_package(Python3 COMPONENTS Interpreter)
|
||||
if(Python3_FOUND)
|
||||
if(CMAKE_SOURCE_DIR STREQUAL CMAKE_CURRENT_SOURCE_DIR)
|
||||
set(AKBASIC_MUTATION_TARGET mutation)
|
||||
else()
|
||||
set(AKBASIC_MUTATION_TARGET akbasic_mutation)
|
||||
endif()
|
||||
add_custom_target(${AKBASIC_MUTATION_TARGET}
|
||||
COMMAND ${Python3_EXECUTABLE}
|
||||
${CMAKE_CURRENT_SOURCE_DIR}/scripts/mutation_test.py
|
||||
--source-root ${CMAKE_CURRENT_SOURCE_DIR}
|
||||
WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}
|
||||
USES_TERMINAL
|
||||
COMMENT "Running mutation tests (breaks the library, expects tests to fail)"
|
||||
)
|
||||
endif()
|
||||
|
||||
install(TARGETS akbasic basic
|
||||
ARCHIVE DESTINATION ${CMAKE_INSTALL_LIBDIR}
|
||||
LIBRARY DESTINATION ${CMAKE_INSTALL_LIBDIR}
|
||||
|
||||
Reference in New Issue
Block a user