Andrew Kesterson
f1283e21a3
Reservations were advisory bookkeeping: any component could name any status, so the registry only detected declared-range overlap between components that both opted in. Naming a status now requires a reservation. akerr_register_status_name() checks that the range belongs to the caller, and the legacy two-argument akerr_name_for_status() set path, which cannot identify its caller, requires that some reservation covers the status. Every refusal is logged and names the real owner, because a name that fails to register degrades that code to "Unknown Error" in every later stack trace. Fix a reservation made before the first PREPARE_ERROR being silently discarded. akerr_init() clears the tables, so whichever component first triggered it wiped an earlier reservation and the next component to claim the same range was told it was free, producing exactly the undetected aliasing the registry exists to prevent. Every registry entry point now calls akerr_init(), which sets its guard before doing any work so those calls do not recurse. Replace the linear-scan name array with an open-addressed hash table, taking lookup from O(n) to O(1) and raising usable capacity from 512 entries (366 free to consumers after errno registration) to 3072 (~2900 free). Both table sizes are build-time overridable and applied PRIVATE: they live entirely in src/error.c, so raising them cannot desynchronize a library from its consumers the way AKERR_MAX_ERR_VALUE could. Exhausting either table is now logged and returned to the caller rather than silently dropping the entry. No dynamic allocation is introduced; both tables remain file-scope arrays, and the library's undefined-symbol set gains only strcmp and strlen. Register names for AKERR_EOF, AKERR_ITERATOR_BREAK and AKERR_NOT_IMPLEMENTED, which had none and rendered as "Unknown Error" in every stack trace carrying them. err_error_names.c now sweeps the whole AKERR_* offset span so a code added without a name fails there instead of in production traces. Add static assertions that the slot count is a power of two and that AKERR_BADEXC stays inside the library's own 0-255 band, the latter guarding against a host errno space large enough to push library codes into the range consumers are told to allocate from. Set a project version and soname (1.0.0 / libakerror.so.1) so a stale installed library can no longer be silently paired with newer headers, and so akerror.pc ships a real Version field instead of an empty one. Mutation testing surfaced an out-of-bounds probe in the new table that the suite did not catch: masking with SLOTS rather than SLOTS-1 indexes past the array, and err_maxval.c asserted only that some names registered before the table filled, which a collapsed probe sequence still satisfies. It now requires a substantial entry count and reads every entry back by its own distinct name. Tests: 28/28 pass. Coverage 99.4% line / 86.8% branch. Mutation score for src/error.c 74% -> 77.3%. Compatibility: source and ABI break. AKERR_MAX_ERR_VALUE and the __AKERR_ERROR_NAMES data symbol are gone, custom codes must move out of 0-255, and names must be registered against a reserved range. README.md carries the migration steps. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
133 lines
6.1 KiB
Markdown
133 lines
6.1 KiB
Markdown
# Mutation testing
|
|
|
|
The unit tests tell us the library works. **Mutation testing tells us the tests
|
|
work** — that they would actually fail if the library were broken.
|
|
|
|
`scripts/mutation_test.py` deliberately breaks the library in small ways
|
|
("mutants"), one at a time, and runs the whole CTest suite against each broken
|
|
copy:
|
|
|
|
* if the tests **fail**, the mutant is **killed** — good, the suite caught it;
|
|
* if the tests still **pass**, the mutant **survived** — a bug of that shape
|
|
would slip through, so it points at a missing test.
|
|
|
|
The **mutation score** is `killed / (killed + survived)`. A surviving mutant is
|
|
a to-do item: write a test that distinguishes the mutant from the original.
|
|
|
|
## Running
|
|
|
|
No third-party tools are required — just Python 3 and the normal
|
|
cmake/ctest toolchain. The harness never touches your working tree; it copies
|
|
the repo to a scratch directory and mutates the copy.
|
|
|
|
```sh
|
|
# Default: mutate src/error.c and include/akerror.tmpl.h
|
|
scripts/mutation_test.py
|
|
|
|
# Faster: just the C source
|
|
scripts/mutation_test.py --target src/error.c
|
|
|
|
# See what would run without building anything
|
|
scripts/mutation_test.py --target src/error.c --list
|
|
|
|
# Gate CI: exit non-zero if the score drops below 90%
|
|
scripts/mutation_test.py --threshold 90
|
|
```
|
|
|
|
Via CMake (configures a build first if needed):
|
|
|
|
```sh
|
|
cmake --build build --target mutation
|
|
```
|
|
|
|
Useful flags: `--timeout SECONDS` (per-suite build+test cap; a mutant that
|
|
hangs is counted as killed), `--keep` (retain the scratch copy for debugging),
|
|
`--work DIR` (use a specific scratch directory), `--junit FILE` (write a JUnit
|
|
XML report — surviving mutants appear as failing test cases).
|
|
|
|
## CI reporting
|
|
|
|
Both the unit tests and the mutation run emit JUnit XML that CI consumes:
|
|
|
|
* `ctest --test-dir build --output-junit "$(pwd)/ctest-junit.xml"` — note the
|
|
absolute path; `--output-junit` otherwise resolves relative to the test dir.
|
|
* `scripts/mutation_test.py --junit mutation-junit.xml`
|
|
|
|
`.gitea/workflows/ci.yaml` runs both and feeds the XML to
|
|
`mikepenz/action-junit-report` (with `if: always()`, so results publish even
|
|
when a gate fails). The reporter runs with `annotate_only: true`: Gitea does not
|
|
implement the Checks API the action uses to create a check run, so creating one
|
|
404s (mikepenz/action-junit-report#23). `annotate_only` skips that call and the
|
|
results surface via the job summary (`detailed_summary: true`) instead. The
|
|
generated `*-junit.xml` files are git-ignored.
|
|
|
|
## Mutation operators
|
|
|
|
Each mutant changes exactly one location by one of:
|
|
|
|
| Tag | Operator | Example |
|
|
|-----|--------------------------------|----------------------------------|
|
|
| ROR | relational operator | `==` → `!=`, `<` → `<=`, `>=` → `>` |
|
|
| LCR | logical connector | `&&` → `\|\|` |
|
|
| BCR | boolean constant | `true` → `false` |
|
|
| AOR | arithmetic / compound assign | `+` → `-`, `+=` → `-=` |
|
|
| ICR | integer literal | `0` → `1`, `1` → `0` |
|
|
| SDL | statement deletion | `err->refcount += 1;` → *(removed)* |
|
|
|
|
Preprocessor control lines, comments, and the block of error-code / buffer-size
|
|
`#define`s are skipped: mutating those produces equivalent or uninteresting
|
|
mutants that only add noise.
|
|
|
|
## Interpreting survivors
|
|
|
|
Not every survivor is a test gap — some mutants are **equivalent** (they don't
|
|
change observable behaviour, e.g. resizing an internal scratch buffer). For each
|
|
survivor, decide:
|
|
|
|
1. **Real gap** → add or strengthen a test in `tests/` so the mutant is killed,
|
|
then re-run.
|
|
2. **Equivalent mutant** → no test can catch it; leave a note. If a specific
|
|
line is a persistent source of equivalents, narrow the target with
|
|
`--target` or extend the skip rules in `scripts/mutation_test.py`.
|
|
|
|
Re-run after adding tests and confirm the score went up.
|
|
|
|
## Current status
|
|
|
|
`src/error.c` scores ~77% (the CI gate is set to 65% for headroom). The
|
|
remaining survivors are dominated by:
|
|
|
|
* **Equivalent mutants** in `akerr_init`: deleting the `memset`/`NULL` setup of
|
|
file-scope statics (`AKERR_ARRAY_ERROR`, `__akerr_last_ditch`,
|
|
`__akerr_last_ignored`) changes nothing, because C already zero-initializes
|
|
objects with static storage duration. `int oldid = 0;` → `1` is likewise
|
|
dead: it is overwritten before use.
|
|
* **Default logger / handler internals** (`vfprintf`, `va_end`, the
|
|
`errctx == NULL` branch, `exit(1)`): killing these needs a subprocess-based
|
|
test that captures a child's stderr and exit code, rather than the in-process
|
|
capturing logger the other tests use.
|
|
* **Static assertions** (`akerr_assert_name_slots_pow2` and the occupancy cap
|
|
it guards): a mutated compile-time assertion that still compiles has no
|
|
runtime behavior to observe. Unkillable by construction — the assertion is
|
|
itself the test, and `tests/err_maxval.c` covers the runtime consequence.
|
|
* **Hash and probe details** in `akerr_status_slot`: dropping one of the
|
|
multiply steps in `akerr_status_hash` leaves a worse but still correct hash,
|
|
and probing backwards (`slot - 1u`) is an equally valid sequence over a
|
|
power-of-two table. Both are behaviorally equivalent.
|
|
* **The `capacity <= 0` guard** in `akerr_copy_string`, which is defensive: both
|
|
call sites pass a positive constant.
|
|
|
|
Findings surfaced by mutation testing:
|
|
|
|
* **Superseded:** status names now use a private sparse registry, so the old
|
|
public `AKERR_MAX_ERR_VALUE` ceiling and its consumer ABI mismatch no longer
|
|
exist. `tests/err_maxval.c` covers arbitrary `int` values and registry
|
|
exhaustion.
|
|
* **Fixed:** the open-addressing probe mask (`& (AKERR_STATUS_NAME_SLOTS - 1)`)
|
|
could be mutated to `- 0` or `+ 1` — both of which index past the end of the
|
|
table — without any test noticing. `tests/err_maxval.c` only asserted that
|
|
*some* names registered before the table filled, which a collapsed probe
|
|
sequence still satisfies. It now requires a substantial number of entries and
|
|
reads every one of them back by its own distinct name, so a probe that
|
|
revisits slots fails on both counts.
|