Add mutation testing to validate the test suite
Introduce a self-contained mutation testing harness that verifies the unit tests actually catch bugs: it makes small deliberate breakages to the library (flip comparisons, delete statements, swap true/false, etc.), rebuilds, and runs the whole CTest suite against each mutant. Tests that still pass reveal a gap; tests that fail "kill" the mutant. - scripts/mutation_test.py: the engine (stdlib only, no LLVM/clang deps). Operators ROR/LCR/BCR/AOR/ICR/SDL over src/error.c and the macro header. Mutates a scratch copy, never the working tree. Supports --target, --list, --max-mutants sampling, --threshold gating, --timeout. - CMakeLists.txt: 'mutation' custom target (cmake --build build --target mutation). - .gitea/workflows/ci.yaml: gated mutation job on src/error.c (threshold 65%). - tests/MUTATION.md: how to run, interpret survivors, and known equivalents. Close the real gaps the harness found in src/error.c (score 53% -> 71%): - err_error_names: the AKERR_* codes have their names registered by akerr_init - err_release_clears: releasing a context wipes it before reuse - err_pool_exhaust: akerr_next_error returns NULL when the pool is full and always hands back the lowest free slot Also surfaced (documented, not fixed): AKERR_MAX_ERR_VALUE (+15) is below AKERR_NOT_IMPLEMENTED (+16) and AKERR_BADEXC (+17), so those codes can never have a name registered. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
100
tests/MUTATION.md
Normal file
100
tests/MUTATION.md
Normal file
@@ -0,0 +1,100 @@
|
||||
# Mutation testing
|
||||
|
||||
The unit tests tell us the library works. **Mutation testing tells us the tests
|
||||
work** — that they would actually fail if the library were broken.
|
||||
|
||||
`scripts/mutation_test.py` deliberately breaks the library in small ways
|
||||
("mutants"), one at a time, and runs the whole CTest suite against each broken
|
||||
copy:
|
||||
|
||||
* if the tests **fail**, the mutant is **killed** — good, the suite caught it;
|
||||
* if the tests still **pass**, the mutant **survived** — a bug of that shape
|
||||
would slip through, so it points at a missing test.
|
||||
|
||||
The **mutation score** is `killed / (killed + survived)`. A surviving mutant is
|
||||
a to-do item: write a test that distinguishes the mutant from the original.
|
||||
|
||||
## Running
|
||||
|
||||
No third-party tools are required — just Python 3 and the normal
|
||||
cmake/ctest toolchain. The harness never touches your working tree; it copies
|
||||
the repo to a scratch directory and mutates the copy.
|
||||
|
||||
```sh
|
||||
# Default: mutate src/error.c and include/akerror.tmpl.h
|
||||
scripts/mutation_test.py
|
||||
|
||||
# Faster: just the C source
|
||||
scripts/mutation_test.py --target src/error.c
|
||||
|
||||
# See what would run without building anything
|
||||
scripts/mutation_test.py --target src/error.c --list
|
||||
|
||||
# Gate CI: exit non-zero if the score drops below 90%
|
||||
scripts/mutation_test.py --threshold 90
|
||||
```
|
||||
|
||||
Via CMake (configures a build first if needed):
|
||||
|
||||
```sh
|
||||
cmake --build build --target mutation
|
||||
```
|
||||
|
||||
Useful flags: `--timeout SECONDS` (per-suite build+test cap; a mutant that
|
||||
hangs is counted as killed), `--keep` (retain the scratch copy for debugging),
|
||||
`--work DIR` (use a specific scratch directory).
|
||||
|
||||
## Mutation operators
|
||||
|
||||
Each mutant changes exactly one location by one of:
|
||||
|
||||
| Tag | Operator | Example |
|
||||
|-----|--------------------------------|----------------------------------|
|
||||
| ROR | relational operator | `==` → `!=`, `<` → `<=`, `>=` → `>` |
|
||||
| LCR | logical connector | `&&` → `\|\|` |
|
||||
| BCR | boolean constant | `true` → `false` |
|
||||
| AOR | arithmetic / compound assign | `+` → `-`, `+=` → `-=` |
|
||||
| ICR | integer literal | `0` → `1`, `1` → `0` |
|
||||
| SDL | statement deletion | `err->refcount += 1;` → *(removed)* |
|
||||
|
||||
Preprocessor control lines, comments, and the block of error-code / buffer-size
|
||||
`#define`s are skipped: mutating those produces equivalent or uninteresting
|
||||
mutants that only add noise.
|
||||
|
||||
## Interpreting survivors
|
||||
|
||||
Not every survivor is a test gap — some mutants are **equivalent** (they don't
|
||||
change observable behaviour, e.g. resizing an internal scratch buffer). For each
|
||||
survivor, decide:
|
||||
|
||||
1. **Real gap** → add or strengthen a test in `tests/` so the mutant is killed,
|
||||
then re-run.
|
||||
2. **Equivalent mutant** → no test can catch it; leave a note. If a specific
|
||||
line is a persistent source of equivalents, narrow the target with
|
||||
`--target` or extend the skip rules in `scripts/mutation_test.py`.
|
||||
|
||||
Re-run after adding tests and confirm the score went up.
|
||||
|
||||
## Current status
|
||||
|
||||
`src/error.c` scores ~71% (the CI gate is set to 65% for headroom). The
|
||||
remaining survivors are dominated by:
|
||||
|
||||
* **Equivalent mutants** in `akerr_init`: deleting the `memset`/`NULL` setup of
|
||||
file-scope statics (`AKERR_ARRAY_ERROR`, `__akerr_last_ditch`,
|
||||
`__akerr_last_ignored`) changes nothing, because C already zero-initializes
|
||||
objects with static storage duration. `int oldid = 0;` → `1` is likewise
|
||||
dead: it is overwritten before use.
|
||||
* **Default logger / handler internals** (`vfprintf`, `va_end`, the
|
||||
`errctx == NULL` branch, `exit(1)`): killing these needs a subprocess-based
|
||||
test that captures a child's stderr and exit code, rather than the in-process
|
||||
capturing logger the other tests use.
|
||||
|
||||
Findings worth noting (surfaced by mutation testing, not yet fixed):
|
||||
|
||||
* `AKERR_MAX_ERR_VALUE` is `AKERR_LAST_ERRNO_VALUE + 15`, but `AKERR_NOT_IMPLEMENTED`
|
||||
(+16) and `AKERR_BADEXC` (+17) exceed it. `akerr_name_for_status` rejects any
|
||||
status `> AKERR_MAX_ERR_VALUE`, so those two codes can never store or return a
|
||||
name — the `akerr_name_for_status(AKERR_BADEXC, ...)` call in `akerr_init` is
|
||||
dead code (which is why deleting it survives). Bumping `AKERR_MAX_ERR_VALUE` to
|
||||
`+ 17` would fix it.
|
||||
45
tests/err_error_names.c
Normal file
45
tests/err_error_names.c
Normal file
@@ -0,0 +1,45 @@
|
||||
#include "akerror.h"
|
||||
#include "err_capture.h"
|
||||
#include <string.h>
|
||||
|
||||
/*
|
||||
* akerr_init() registers a human-readable name for each library error code.
|
||||
* Verify the names are actually installed (mutation testing showed the
|
||||
* registration calls could be deleted without any test noticing).
|
||||
*
|
||||
* Note: AKERR_NOT_IMPLEMENTED and AKERR_BADEXC are intentionally omitted -- they
|
||||
* exceed AKERR_MAX_ERR_VALUE, so akerr_name_for_status cannot store or return
|
||||
* their names (see tests/MUTATION.md).
|
||||
*/
|
||||
|
||||
static const struct {
|
||||
int code;
|
||||
const char *name;
|
||||
} expected[] = {
|
||||
{ AKERR_NULLPOINTER, "Null Pointer Error" },
|
||||
{ AKERR_OUTOFBOUNDS, "Out Of Bounds Error" },
|
||||
{ AKERR_API, "API Error" },
|
||||
{ AKERR_ATTRIBUTE, "Attribute Error" },
|
||||
{ AKERR_TYPE, "Type Error" },
|
||||
{ AKERR_KEY, "Key Error" },
|
||||
{ AKERR_INDEX, "Index Error" },
|
||||
{ AKERR_FORMAT, "Format Error" },
|
||||
{ AKERR_IO, "Input Output Error" },
|
||||
{ AKERR_VALUE, "Value Error" },
|
||||
{ AKERR_RELATIONSHIP, "Relationship Error" },
|
||||
{ AKERR_CIRCULAR_REFERENCE, "Circular Reference Error" },
|
||||
};
|
||||
|
||||
int main(void)
|
||||
{
|
||||
akerr_init();
|
||||
|
||||
for ( unsigned i = 0; i < sizeof(expected) / sizeof(expected[0]); i++ ) {
|
||||
char *nm = akerr_name_for_status(expected[i].code, NULL);
|
||||
AKERR_CHECK(nm != NULL);
|
||||
AKERR_CHECK(strcmp(nm, expected[i].name) == 0);
|
||||
}
|
||||
|
||||
fprintf(stderr, "err_error_names ok\n");
|
||||
return 0;
|
||||
}
|
||||
39
tests/err_pool_exhaust.c
Normal file
39
tests/err_pool_exhaust.c
Normal file
@@ -0,0 +1,39 @@
|
||||
#include "akerror.h"
|
||||
#include "err_capture.h"
|
||||
|
||||
/*
|
||||
* The error pool is a fixed array of AKERR_MAX_ARRAY_ERROR slots. When every
|
||||
* slot is checked out, akerr_next_error() must return NULL rather than run off
|
||||
* the end of the array; and it must always hand back the lowest free slot.
|
||||
* Mutation testing showed both the terminating "return NULL" and the scan
|
||||
* bounds could be broken without any test noticing.
|
||||
*/
|
||||
|
||||
int main(void)
|
||||
{
|
||||
akerr_init();
|
||||
|
||||
akerr_ErrorContext *slots[AKERR_MAX_ARRAY_ERROR];
|
||||
|
||||
/* Check out every slot. */
|
||||
for ( int i = 0; i < AKERR_MAX_ARRAY_ERROR; i++ ) {
|
||||
slots[i] = akerr_next_error();
|
||||
AKERR_CHECK(slots[i] != NULL);
|
||||
slots[i]->refcount = 1;
|
||||
}
|
||||
|
||||
/* Pool is fully exhausted: the next request must fail cleanly. */
|
||||
AKERR_CHECK(akerr_next_error() == NULL);
|
||||
|
||||
/* Free exactly the first slot; the scan must find and return it. */
|
||||
slots[0]->refcount = 0;
|
||||
AKERR_CHECK(akerr_next_error() == slots[0]);
|
||||
|
||||
/* Tidy up. */
|
||||
for ( int i = 0; i < AKERR_MAX_ARRAY_ERROR; i++ ) {
|
||||
slots[i]->refcount = 0;
|
||||
}
|
||||
|
||||
fprintf(stderr, "err_pool_exhaust ok\n");
|
||||
return 0;
|
||||
}
|
||||
44
tests/err_release_clears.c
Normal file
44
tests/err_release_clears.c
Normal file
@@ -0,0 +1,44 @@
|
||||
#include "akerror.h"
|
||||
#include "err_capture.h"
|
||||
#include <string.h>
|
||||
|
||||
/*
|
||||
* Releasing an error context back to the pool must wipe it, so the next caller
|
||||
* that checks it out never sees stale status/message/stacktrace from a previous
|
||||
* error. Mutation testing showed the clearing memset in akerr_release_error
|
||||
* could be deleted without any test noticing.
|
||||
*/
|
||||
|
||||
akerr_ErrorContext *boom(void)
|
||||
{
|
||||
PREPARE_ERROR(e);
|
||||
FAIL_RETURN(e, AKERR_VALUE, "stale dirty message that must not survive");
|
||||
}
|
||||
|
||||
int main(void)
|
||||
{
|
||||
akerr_capture_install();
|
||||
akerr_init();
|
||||
|
||||
/* Raise and fully handle an error; FINISH_NORETURN releases it to the pool. */
|
||||
PREPARE_ERROR(e);
|
||||
ATTEMPT {
|
||||
CATCH(e, boom());
|
||||
} CLEANUP {
|
||||
} PROCESS(e) {
|
||||
} HANDLE(e, AKERR_VALUE) {
|
||||
} FINISH_NORETURN(e);
|
||||
|
||||
AKERR_CHECK(e == NULL);
|
||||
|
||||
/* The next context handed out is the slot we just released: it must be clean. */
|
||||
akerr_ErrorContext *slot = akerr_next_error();
|
||||
AKERR_CHECK(slot != NULL);
|
||||
AKERR_CHECK(slot->status == 0);
|
||||
AKERR_CHECK(slot->message[0] == '\0');
|
||||
AKERR_CHECK(slot->stacktracebuf[0] == '\0');
|
||||
AKERR_CHECK(strstr(slot->message, "stale dirty message") == NULL);
|
||||
|
||||
fprintf(stderr, "err_release_clears ok\n");
|
||||
return 0;
|
||||
}
|
||||
Reference in New Issue
Block a user