libakerror/tests/MUTATION.md

Mutation testing

The unit tests tell us the library works. Mutation testing tells us the tests work — that they would actually fail if the library were broken.

scripts/mutation_test.py deliberately breaks the library in small ways ("mutants"), one at a time, and runs the whole CTest suite against each broken copy:

• if the tests fail, the mutant is killed — good, the suite caught it;
• if the tests still pass, the mutant survived — a bug of that shape would slip through, so it points at a missing test.

The mutation score is killed / (killed + survived). A surviving mutant is a to-do item: write a test that distinguishes the mutant from the original.

Running

No third-party tools are required — just Python 3 and the normal cmake/ctest toolchain. The harness never touches your working tree; it copies the repo to a scratch directory and mutates the copy.

# Default: mutate src/error.c and include/akerror.tmpl.h
scripts/mutation_test.py

# Faster: just the C source
scripts/mutation_test.py --target src/error.c

# See what would run without building anything
scripts/mutation_test.py --target src/error.c --list

# Gate CI: exit non-zero if the score drops below 90%
scripts/mutation_test.py --threshold 90

Via CMake (configures a build first if needed):

cmake --build build --target mutation

Useful flags: --timeout SECONDS (per-suite build+test cap; a mutant that hangs is counted as killed), --keep (retain the scratch copy for debugging), --work DIR (use a specific scratch directory), --junit FILE (write a JUnit XML report — surviving mutants appear as failing test cases).

CI reporting

Both the unit tests and the mutation run emit JUnit XML that CI consumes:

• ctest --test-dir build --output-junit "$(pwd)/ctest-junit.xml" — note the absolute path; --output-junit otherwise resolves relative to the test dir.
• scripts/mutation_test.py --junit mutation-junit.xml

.gitea/workflows/ci.yaml runs both and feeds the XML to mikepenz/action-junit-report (with if: always(), so results publish even when a gate fails). The reporter runs with annotate_only: true: Gitea does not implement the Checks API the action uses to create a check run, so creating one 404s (mikepenz/action-junit-report#23). annotate_only skips that call and the results surface via the job summary (detailed_summary: true) instead. The generated *-junit.xml files are git-ignored.

Mutation operators

Each mutant changes exactly one location by one of:

Tag	Operator	Example
ROR	relational operator	== → !=, < → <=, >= → >
LCR	logical connector	&& → ||
BCR	boolean constant	true → false
AOR	arithmetic / compound assign	+ → -, += → -=
ICR	integer literal	0 → 1, 1 → 0
SDL	statement deletion	err->refcount += 1; → (removed)

Preprocessor control lines, comments, and the block of error-code / buffer-size #defines are skipped: mutating those produces equivalent or uninteresting mutants that only add noise.

Interpreting survivors

Not every survivor is a test gap — some mutants are equivalent (they don't change observable behaviour, e.g. resizing an internal scratch buffer). For each survivor, decide:

1. Real gap → add or strengthen a test in tests/ so the mutant is killed, then re-run.
2. Equivalent mutant → no test can catch it; leave a note. If a specific line is a persistent source of equivalents, narrow the target with --target or extend the skip rules in scripts/mutation_test.py.

Re-run after adding tests and confirm the score went up.

Current status

src/error.c scores ~74% (the CI gate is set to 65% for headroom). The remaining survivors are dominated by:

• Equivalent mutants in akerr_init: deleting the memset/NULL setup of file-scope statics (AKERR_ARRAY_ERROR, __akerr_last_ditch, __akerr_last_ignored) changes nothing, because C already zero-initializes objects with static storage duration. int oldid = 0; → 1 is likewise dead: it is overwritten before use.
• Default logger / handler internals (vfprintf, va_end, the errctx == NULL branch, exit(1)): killing these needs a subprocess-based test that captures a child's stderr and exit code, rather than the in-process capturing logger the other tests use.

Findings surfaced by mutation testing:

• Fixed: AKERR_MAX_ERR_VALUE was AKERR_LAST_ERRNO_VALUE + 15, below AKERR_NOT_IMPLEMENTED (+16) and AKERR_BADEXC (+17). akerr_name_for_status rejects any status > AKERR_MAX_ERR_VALUE, so those codes could never store or return a name and the akerr_name_for_status(AKERR_BADEXC, ...) call in akerr_init was dead code (which is why deleting it survived). The max is now + 17, and tests/err_maxval.c guards the invariant so it can't regress.