Add mutation testing to validate the test suite
Introduce a self-contained mutation testing harness that verifies the unit
tests actually catch bugs: it makes small deliberate breakages to the library
(flip comparisons, delete statements, swap true/false, etc.), rebuilds, and
runs the whole CTest suite against each mutant. Tests that still pass reveal a
gap; tests that fail "kill" the mutant.
- scripts/mutation_test.py: the engine (stdlib only, no LLVM/clang deps).
Operators ROR/LCR/BCR/AOR/ICR/SDL over src/error.c and the macro header.
Mutates a scratch copy, never the working tree. Supports --target, --list,
--max-mutants sampling, --threshold gating, --timeout.
- CMakeLists.txt: 'mutation' custom target (cmake --build build --target mutation).
- .gitea/workflows/ci.yaml: gated mutation job on src/error.c (threshold 65%).
- tests/MUTATION.md: how to run, interpret survivors, and known equivalents.
Close the real gaps the harness found in src/error.c (score 53% -> 71%):
- err_error_names: the AKERR_* codes have their names registered by akerr_init
- err_release_clears: releasing a context wipes it before reuse
- err_pool_exhaust: akerr_next_error returns NULL when the pool is full and
always hands back the lowest free slot
Also surfaced (documented, not fixed): AKERR_MAX_ERR_VALUE (+15) is below
AKERR_NOT_IMPLEMENTED (+16) and AKERR_BADEXC (+17), so those codes can never
have a name registered.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-27 17:03:53 -04:00
|
|
|
#include "akerror.h"
|
|
|
|
|
#include "err_capture.h"
|
|
|
|
|
#include <string.h>
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* akerr_init() registers a human-readable name for each library error code.
|
|
|
|
|
* Verify the names are actually installed (mutation testing showed the
|
|
|
|
|
* registration calls could be deleted without any test noticing).
|
|
|
|
|
*
|
2026-07-27 17:17:26 -04:00
|
|
|
* Note: AKERR_EOF, AKERR_ITERATOR_BREAK and AKERR_NOT_IMPLEMENTED are omitted --
|
|
|
|
|
* they are valid codes but akerr_init does not register a display name for them,
|
|
|
|
|
* so akerr_name_for_status returns an empty string rather than a known name.
|
Add mutation testing to validate the test suite
Introduce a self-contained mutation testing harness that verifies the unit
tests actually catch bugs: it makes small deliberate breakages to the library
(flip comparisons, delete statements, swap true/false, etc.), rebuilds, and
runs the whole CTest suite against each mutant. Tests that still pass reveal a
gap; tests that fail "kill" the mutant.
- scripts/mutation_test.py: the engine (stdlib only, no LLVM/clang deps).
Operators ROR/LCR/BCR/AOR/ICR/SDL over src/error.c and the macro header.
Mutates a scratch copy, never the working tree. Supports --target, --list,
--max-mutants sampling, --threshold gating, --timeout.
- CMakeLists.txt: 'mutation' custom target (cmake --build build --target mutation).
- .gitea/workflows/ci.yaml: gated mutation job on src/error.c (threshold 65%).
- tests/MUTATION.md: how to run, interpret survivors, and known equivalents.
Close the real gaps the harness found in src/error.c (score 53% -> 71%):
- err_error_names: the AKERR_* codes have their names registered by akerr_init
- err_release_clears: releasing a context wipes it before reuse
- err_pool_exhaust: akerr_next_error returns NULL when the pool is full and
always hands back the lowest free slot
Also surfaced (documented, not fixed): AKERR_MAX_ERR_VALUE (+15) is below
AKERR_NOT_IMPLEMENTED (+16) and AKERR_BADEXC (+17), so those codes can never
have a name registered.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-27 17:03:53 -04:00
|
|
|
*/
|
|
|
|
|
|
|
|
|
|
static const struct {
|
|
|
|
|
int code;
|
|
|
|
|
const char *name;
|
|
|
|
|
} expected[] = {
|
|
|
|
|
{ AKERR_NULLPOINTER, "Null Pointer Error" },
|
|
|
|
|
{ AKERR_OUTOFBOUNDS, "Out Of Bounds Error" },
|
|
|
|
|
{ AKERR_API, "API Error" },
|
|
|
|
|
{ AKERR_ATTRIBUTE, "Attribute Error" },
|
|
|
|
|
{ AKERR_TYPE, "Type Error" },
|
|
|
|
|
{ AKERR_KEY, "Key Error" },
|
|
|
|
|
{ AKERR_INDEX, "Index Error" },
|
|
|
|
|
{ AKERR_FORMAT, "Format Error" },
|
|
|
|
|
{ AKERR_IO, "Input Output Error" },
|
|
|
|
|
{ AKERR_VALUE, "Value Error" },
|
|
|
|
|
{ AKERR_RELATIONSHIP, "Relationship Error" },
|
|
|
|
|
{ AKERR_CIRCULAR_REFERENCE, "Circular Reference Error" },
|
2026-07-27 17:17:26 -04:00
|
|
|
{ AKERR_BADEXC, "Invalid akerr_ErrorContext" },
|
Add mutation testing to validate the test suite
Introduce a self-contained mutation testing harness that verifies the unit
tests actually catch bugs: it makes small deliberate breakages to the library
(flip comparisons, delete statements, swap true/false, etc.), rebuilds, and
runs the whole CTest suite against each mutant. Tests that still pass reveal a
gap; tests that fail "kill" the mutant.
- scripts/mutation_test.py: the engine (stdlib only, no LLVM/clang deps).
Operators ROR/LCR/BCR/AOR/ICR/SDL over src/error.c and the macro header.
Mutates a scratch copy, never the working tree. Supports --target, --list,
--max-mutants sampling, --threshold gating, --timeout.
- CMakeLists.txt: 'mutation' custom target (cmake --build build --target mutation).
- .gitea/workflows/ci.yaml: gated mutation job on src/error.c (threshold 65%).
- tests/MUTATION.md: how to run, interpret survivors, and known equivalents.
Close the real gaps the harness found in src/error.c (score 53% -> 71%):
- err_error_names: the AKERR_* codes have their names registered by akerr_init
- err_release_clears: releasing a context wipes it before reuse
- err_pool_exhaust: akerr_next_error returns NULL when the pool is full and
always hands back the lowest free slot
Also surfaced (documented, not fixed): AKERR_MAX_ERR_VALUE (+15) is below
AKERR_NOT_IMPLEMENTED (+16) and AKERR_BADEXC (+17), so those codes can never
have a name registered.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-27 17:03:53 -04:00
|
|
|
};
|
|
|
|
|
|
|
|
|
|
int main(void)
|
|
|
|
|
{
|
|
|
|
|
akerr_init();
|
|
|
|
|
|
|
|
|
|
for ( unsigned i = 0; i < sizeof(expected) / sizeof(expected[0]); i++ ) {
|
|
|
|
|
char *nm = akerr_name_for_status(expected[i].code, NULL);
|
|
|
|
|
AKERR_CHECK(nm != NULL);
|
|
|
|
|
AKERR_CHECK(strcmp(nm, expected[i].name) == 0);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
fprintf(stderr, "err_error_names ok\n");
|
|
|
|
|
return 0;
|
|
|
|
|
}
|