Files
libakstdlib/README.md

273 lines
13 KiB
Markdown
Raw Normal View History

2026-06-27 10:09:38 -04:00
# README
![build badge](https://source.starfort.tech/andrew/libakstdlib/actions/workflows/ci.yaml/badge.svg?branch=main)
`libakstdlib` wraps C standard library functions so that they report failures
through [libakerror](https://source.starfort.tech/andrew/libakerror)'s
`ATTEMPT { ... } HANDLE { ... }` error contexts instead of through return codes
and `errno`. It also provides a few data structures built on the same
convention (a doubly-linked list and a binary tree).
Every entry point returns `akerr_ErrorContext *` and is marked `AKERR_NOIGNORE`.
See `TODO.md` for the current state of the library: what is covered by tests,
which corner cases are still open, and which libc functions are not yet wrapped.
## Building
```sh
git submodule update --init --recursive # deps/libakerror
cmake -S . -B build
cmake --build build
cmake --install build
```
A top-level build compiles the vendored `deps/libakerror`. When `libakstdlib` is
consumed as a subproject, it uses whatever `akerror::akerror` target or installed
package the parent provides instead.
Upgrade to libakerror 1.0.0 Bump deps/libakerror 22 commits to 5ff8790 (1.0.0), which makes the status-name table private, moves consumer status codes to a band starting at AKERR_FIRST_CONSUMER_STATUS, enforces range ownership rather than treating it as advisory, and gives the library an soname. See deps/libakerror/UPGRADING.md. src/stdlib.c needed no changes. This library defines no status codes of its own -- it raises libakerror's AKERR_* codes and propagates errno, both inside libakerror's reserved 0-255 band -- and it never referenced AKERR_MAX_ERR_VALUE, __AKERR_ERROR_NAMES, AKERR_STATUS_RANGE_OK or AKERR_STATUS_NAME_OK. What moved was everything around the code: A -DAKSL_COVERAGE=ON build stopped configuring at all. libakerror namespaces its `mutation` target when embedded but not its `coverage` target, so it collided with ours. Shadow add_custom_target for the duration of the add_subdirectory() call and rename the dependency's to akerror_coverage, alongside the existing add_test shadow. Fix upstream and delete the workaround; recorded in TODO.md. Pin the 1.0.0 floor three ways, since no single one covers every consumption path: an #error in akstdlib.h feature-testing AKERR_FIRST_CONSUMER_STATUS, because libakerror publishes no version macro; Requires: akerror >= 1.0.0 in akstdlib.pc, which also gets consumers -lakerror transitively; and find_dependency(akerror) in akstdlibConfig.cmake. The last was already broken before this bump -- the template still carried its MyLibraryConfig placeholder with the dependency commented out, so any external find_package(akstdlib) failed with a bare "akerror::akerror not found" out of the generated targets file. Branch coverage of src/stdlib.c fell from 51.0% to 44.3% with no source or test change: the 1.0.0 PREPARE_ERROR/FAIL_* macros expand to more branches at every call site, so 337/661 became 481/1087 -- 144 more branches covered, 426 more counted. Line coverage held at 99.0% (200/202) and function coverage at 100% (21/21). Re-ratchet the CI branch gate 45 -> 40 rather than chase branches that belong to libakerror's own suite. tests/test_status_registry.c pins the contract that made the status-code migration a no-op: libakstdlib reserves no consumer range, so an application may allocate from AKERR_FIRST_CONSUMER_STATUS without coordinating with it, and every status this library raises is inside the reserved band with a name actually registered -- an unnamed one degrades to "Unknown Error" in every later stack trace, which nothing else would notice. It exercises the new ownership enforcement too, so the "reserves nothing" assertion cannot pass vacuously. ctest 13/13, ASan+UBSan 13/13, coverage 15/15 at 90/40, mutation 89.6% (155/173, unchanged). Also verified out of tree: the #error fires as the first diagnostic against a stale akerror.h, pkg-config refuses akerror 0.9.0, and an external find_package(akstdlib) consumer builds and runs against a temp-prefix install. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-30 22:20:54 -04:00
### The libakerror version floor
**libakerror 1.0.0 or newer is required.** That release made the status-name
table private, moved consumer status codes into a band starting at
`AKERR_FIRST_CONSUMER_STATUS` (256), made range ownership enforced rather than
advisory, and gave the library an soname — see
`deps/libakerror/UPGRADING.md`. It is a source *and* ABI break, so pairing this
header with an older `akerror.h` is not a compile problem you can work around;
the pairing is simply invalid.
Three things enforce the floor, because no single one covers every way the
library gets consumed:
| Mechanism | Where | Catches |
| --- | --- | --- |
| `#error` on a missing `AKERR_FIRST_CONSUMER_STATUS` | `include/akstdlib.h` | a stale `akerror.h` earlier on the include path, at the first diagnostic rather than as a pile of errors inside `src/stdlib.c` |
| `Requires: akerror >= 1.0.0` | `akstdlib.pc.in` | a pkg-config consumer, which also now gets `-lakerror` transitively |
| `find_dependency(akerror)` | `cmake/akstdlib.cmake.in` | a `find_package(akstdlib)` consumer, which previously failed with a bare *"akerror::akerror not found"* out of the generated targets file |
The header guard feature-tests rather than version-tests because libakerror
publishes no version macro; `AKERR_FIRST_CONSUMER_STATUS` is the symbol 1.0.0
introduced, so its absence is what "older than 1.0.0" actually looks like. The
CMake path requests no version for the same kind of reason: libakerror installs
no `akerrorConfigVersion.cmake`, so `find_dependency(akerror 1.0.0)` would be
refused for want of a version file no matter which akerror is installed.
**libakstdlib defines no status codes of its own.** It raises libakerror's
`AKERR_*` codes and propagates the host's `errno` values, all of which live in
libakerror's reserved `0``255` band, so it reserves no range and an application
is free to allocate from `AKERR_FIRST_CONSUMER_STATUS` without coordinating with
it. `tests/test_status_registry.c` pins that, along with the requirement that
every status this library raises actually has a name registered — an unnamed one
degrades to `"Unknown Error"` in every later stack trace, which nothing else
would notice.
## Testing
Add code coverage to the CTest suite New AKSL_COVERAGE option instruments the library and its tests with --coverage -O0 and wires the report into the suite itself, so a plain ctest --test-dir build-coverage both runs the tests and produces coverage. Two CTest entries do the work, held in order by a CTest fixture rather than by declaration order so they also hold under ctest -j: coverage_reset (FIXTURES_SETUP) clears the .gcda counters before any test, since gcov counts are cumulative and would otherwise fold in earlier runs; coverage_report (FIXTURES_CLEANUP) aggregates gcov output afterwards. AKSL_COVERAGE_THRESHOLD / AKSL_COVERAGE_BRANCH_THRESHOLD gate the report, the same regression-ratchet idea as the mutation score. The `coverage` target builds, runs and prints in one step. scripts/coverage.py parses gcov's JSON output, aggregates line, branch and function counts across translation units, and lists every uncovered line and never-called function -- the actionable half, as with surviving mutants. Python stdlib plus gcc's own gcov only: no lcov, gcovr or genhtml. It also writes coverage-summary.txt (CTest hides the output of a passing test) and a Cobertura coverage.xml for CI publishers. Instrumentation is per target, so deps/libakerror stays out of the report. The mutation harness now ignores build*/ and gcov artifacts when copying the tree, so a coverage build does not slow it down. Baseline on src/stdlib.c: 52.0% of lines, 23.6% of branches, 8 of 21 functions. The uncovered functions are the untested wrappers the mutation survivors already point at (printf, ato*, stream, realpath, strhash). Verified: cmake -S . -B build-coverage -DAKSL_COVERAGE=ON cmake --build build-coverage --target coverage # 8/8, report printed ctest --test-dir build-coverage -j8 # fixture order holds cmake -S . -B build-coverage -DAKSL_COVERAGE=ON -DAKSL_COVERAGE_THRESHOLD=60 ctest --test-dir build-coverage --output-on-failure # gate fails as expected ctest --test-dir build --output-on-failure # 6/6, no .gcda emitted ctest --test-dir build-asan --output-on-failure # 6/6 scripts/mutation_test.py --target src/stdlib.c --list # 173 mutants, unchanged Totals match gcov itself: 51.98% of 202 lines, 23.60% of 661 branches. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-30 01:48:05 -04:00
There are four harnesses. The first three take seconds; the fourth takes about
half an hour.
### 1. The test suite
```sh
cmake -S . -B build
cmake --build build
ctest --test-dir build --output-on-failure
```
Tests live one per file in `tests/test_<name>.c` and share the helpers in
`tests/aksl_capture.h``AKSL_CHECK()` for plain assertions (unlike `assert()`
it survives `-DNDEBUG`), `AKSL_CHECK_STATUS(call, expected)` to run a wrapper and
Test the libc wrappers: 52% -> 99% line coverage Every wrapper outside the list and tree code was untested. Six new test files close that, following the plan already written in TODO.md 1.2-1.6: test_stream.c fopen/fread/fwrite/fclose -- happy paths, the round trip, AKERR_EOF on a short read, AKERR_IO on a stream opened in the wrong mode, ENOENT, and the NULL guards test_format.c printf/fprintf/sprintf -- text *and* count asserted (stdout is pointed at a temp file to check aksl_printf), all eight NULL guards, EBADF on a read-only stream, and 512 variadic calls in a loop as sanitizer cover for the missing va_end test_convert.c ato{i,l,ll,f} happy paths, negatives, leading whitespace, NULL guards test_path.c realpath on a file and on a symlink, both compared against realpath(3) since TMPDIR may itself be a link; ENOENT, ENOTDIR, NULL path test_strhash.c djb2 known-answer vectors, len == 0, embedded NUL, stability, NULL guards test_convert_strict.c known-failing (2.1.5): the AKERR_VALUE / ERANGE contract the ato* family cannot express today test_tree.c gains the BFS AKERR_NOT_IMPLEMENTED contract, NULL arguments, and a callback error that is not AKERR_ITERATOR_BREAK propagating out. Tests deliberately say nothing about behaviour TODO.md records as defective -- unchecked ptr/mode/resolved_path, short transfers reported as success, *count left at -1, the djb2 sign extension -- so the eventual fix does not have to come with a test rewrite. Each failure case in test_path.c passes a zeroed buffer, because the wrapper's own error path formats resolved_path with %s (2.1.6). aksl_capture.h gains aksl_temp_file() with an atexit unlink backstop. Without it every test that fails before its own unlink leaves temp files behind -- which is the normal case for a known-failing test, and happens 173 times over in a mutation run. Coverage on src/stdlib.c: 52.0% -> 99.0% of lines (200/202), 23.6% -> 51.0% of branches, 8/21 -> 21/21 functions. The two uncovered lines are both `} HANDLE(e, AKERR_ITERATOR_BREAK) {`, where the macro starts with the `break;` of PROCESS's `case 0:` arm -- reachable only via a non-NULL error context whose status is zero, the pathology 2.2.1 exists to remove. Mutation score on src/stdlib.c: 46.8% -> 89.6% (155/173 killed). CI, the pre-push hook and the docs ratchet from 40 to 80 accordingly, and the 18 survivors are grouped by cause in TODO.md and README.md. A new CI coverage job gates at 90% lines / 45% branches. Verified: ctest --test-dir build # 12/12 ctest --test-dir build-asan # 12/12 under ASan + UBSan ctest --test-dir build-coverage # 14/14, report attached ctest --test-dir build -j8 --repeat until-fail:3 gcc -Wall -Wextra -c on all nine test files # no warnings python3 scripts/mutation_test.py --target src/stdlib.c # 89.6% No temp files left in /tmp after any of the above. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-30 02:07:08 -04:00
assert on the status it returns, `aksl_temp_file()` for tests that need a real
file to work on, and an `AKSL_RUN()` driver that additionally fails any test which
leaks a slot from libakerror's error pool.
One file per area of the API: `convert` (the `ato*` family), `format` (the
`printf` family), `stream` (`fopen`/`fread`/`fwrite`/`fclose`), `path`
Upgrade to libakerror 1.0.0 Bump deps/libakerror 22 commits to 5ff8790 (1.0.0), which makes the status-name table private, moves consumer status codes to a band starting at AKERR_FIRST_CONSUMER_STATUS, enforces range ownership rather than treating it as advisory, and gives the library an soname. See deps/libakerror/UPGRADING.md. src/stdlib.c needed no changes. This library defines no status codes of its own -- it raises libakerror's AKERR_* codes and propagates errno, both inside libakerror's reserved 0-255 band -- and it never referenced AKERR_MAX_ERR_VALUE, __AKERR_ERROR_NAMES, AKERR_STATUS_RANGE_OK or AKERR_STATUS_NAME_OK. What moved was everything around the code: A -DAKSL_COVERAGE=ON build stopped configuring at all. libakerror namespaces its `mutation` target when embedded but not its `coverage` target, so it collided with ours. Shadow add_custom_target for the duration of the add_subdirectory() call and rename the dependency's to akerror_coverage, alongside the existing add_test shadow. Fix upstream and delete the workaround; recorded in TODO.md. Pin the 1.0.0 floor three ways, since no single one covers every consumption path: an #error in akstdlib.h feature-testing AKERR_FIRST_CONSUMER_STATUS, because libakerror publishes no version macro; Requires: akerror >= 1.0.0 in akstdlib.pc, which also gets consumers -lakerror transitively; and find_dependency(akerror) in akstdlibConfig.cmake. The last was already broken before this bump -- the template still carried its MyLibraryConfig placeholder with the dependency commented out, so any external find_package(akstdlib) failed with a bare "akerror::akerror not found" out of the generated targets file. Branch coverage of src/stdlib.c fell from 51.0% to 44.3% with no source or test change: the 1.0.0 PREPARE_ERROR/FAIL_* macros expand to more branches at every call site, so 337/661 became 481/1087 -- 144 more branches covered, 426 more counted. Line coverage held at 99.0% (200/202) and function coverage at 100% (21/21). Re-ratchet the CI branch gate 45 -> 40 rather than chase branches that belong to libakerror's own suite. tests/test_status_registry.c pins the contract that made the status-code migration a no-op: libakstdlib reserves no consumer range, so an application may allocate from AKERR_FIRST_CONSUMER_STATUS without coordinating with it, and every status this library raises is inside the reserved band with a name actually registered -- an unnamed one degrades to "Unknown Error" in every later stack trace, which nothing else would notice. It exercises the new ownership enforcement too, so the "reserves nothing" assertion cannot pass vacuously. ctest 13/13, ASan+UBSan 13/13, coverage 15/15 at 90/40, mutation 89.6% (155/173, unchanged). Also verified out of tree: the #error fires as the first diagnostic against a stale akerror.h, pkg-config refuses akerror 0.9.0, and an external find_package(akstdlib) consumer builds and runs against a temp-prefix install. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-30 22:20:54 -04:00
(`aksl_realpath`), `strhash`, `memory`, `linkedlist`, `tree`, and
`status_registry` (this library's side of the libakerror status-registry
contract — see "The libakerror version floor" above).
To add a test, drop `tests/test_mything.c` in place and add `mything` to
`AKSL_TESTS` in `CMakeLists.txt`.
**Reading the results.** `CMakeLists.txt` splits tests into three lists, and two
of them invert the meaning of "Passed":
| List | Meaning |
|---|---|
| `AKSL_TESTS` | Ordinary tests. Must exit 0. |
| `AKSL_WILL_FAIL_TESTS` | Expected to abort by design — an unhandled error reaching `FINISH_NORETURN`, or a deliberate contract violation. Marked `WILL_FAIL`, so a non-zero exit is a pass. |
| `AKSL_KNOWN_FAILING_TESTS` | Assert the *correct* behaviour of a confirmed defect (see `TODO.md` §2.1). Also marked `WILL_FAIL`. |
So `ctest` reporting all green does **not** mean the library is defect-free — it
means the known-good tests passed and the known-bad ones are still failing in the
documented way. When a defect is fixed, its test starts passing, CTest reports it
as failed with *unexpectedly passed*, and that is the cue to move it from
`AKSL_KNOWN_FAILING_TESTS` into `AKSL_TESTS`.
Every test is capped with a 30-second CTest `TIMEOUT`. The list and tree code is
full of loops whose termination hangs on a single condition, so a bug of that
shape hangs the suite rather than failing it.
### 2. Sanitizers
```sh
cmake -S . -B build-asan -DAKSL_SANITIZE=ON
cmake --build build-asan
ctest --test-dir build-asan --output-on-failure
```
Builds the library, the tests and the vendored libakerror with ASan + UBSan and
`-fno-sanitize-recover=all`. Several of the open items in `TODO.md` §2 only
misbehave under instrumentation — the uninitialised `%s` in `aksl_realpath`, the
unbounded `vsprintf` behind `aksl_sprintf`, the missing `va_end` in the `printf`
family — so new tests for those should be run this way.
Add code coverage to the CTest suite New AKSL_COVERAGE option instruments the library and its tests with --coverage -O0 and wires the report into the suite itself, so a plain ctest --test-dir build-coverage both runs the tests and produces coverage. Two CTest entries do the work, held in order by a CTest fixture rather than by declaration order so they also hold under ctest -j: coverage_reset (FIXTURES_SETUP) clears the .gcda counters before any test, since gcov counts are cumulative and would otherwise fold in earlier runs; coverage_report (FIXTURES_CLEANUP) aggregates gcov output afterwards. AKSL_COVERAGE_THRESHOLD / AKSL_COVERAGE_BRANCH_THRESHOLD gate the report, the same regression-ratchet idea as the mutation score. The `coverage` target builds, runs and prints in one step. scripts/coverage.py parses gcov's JSON output, aggregates line, branch and function counts across translation units, and lists every uncovered line and never-called function -- the actionable half, as with surviving mutants. Python stdlib plus gcc's own gcov only: no lcov, gcovr or genhtml. It also writes coverage-summary.txt (CTest hides the output of a passing test) and a Cobertura coverage.xml for CI publishers. Instrumentation is per target, so deps/libakerror stays out of the report. The mutation harness now ignores build*/ and gcov artifacts when copying the tree, so a coverage build does not slow it down. Baseline on src/stdlib.c: 52.0% of lines, 23.6% of branches, 8 of 21 functions. The uncovered functions are the untested wrappers the mutation survivors already point at (printf, ato*, stream, realpath, strhash). Verified: cmake -S . -B build-coverage -DAKSL_COVERAGE=ON cmake --build build-coverage --target coverage # 8/8, report printed ctest --test-dir build-coverage -j8 # fixture order holds cmake -S . -B build-coverage -DAKSL_COVERAGE=ON -DAKSL_COVERAGE_THRESHOLD=60 ctest --test-dir build-coverage --output-on-failure # gate fails as expected ctest --test-dir build --output-on-failure # 6/6, no .gcda emitted ctest --test-dir build-asan --output-on-failure # 6/6 scripts/mutation_test.py --target src/stdlib.c --list # 173 mutants, unchanged Totals match gcov itself: 51.98% of 202 lines, 23.60% of 661 branches. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-30 01:48:05 -04:00
### 3. Code coverage
```sh
cmake -S . -B build-coverage -DAKSL_COVERAGE=ON
cmake --build build-coverage --target coverage
```
`-DAKSL_COVERAGE=ON` compiles the library and the tests with `--coverage -O0`,
and wires the report into the suite itself, so a plain
`ctest --test-dir build-coverage` also produces it. Two extra CTest entries
appear, held in place by a CTest fixture rather than by declaration order, so
they work under `ctest -j` too:
| Test | When | Does |
|---|---|---|
| `coverage_reset` | before every other test | deletes the accumulated `.gcda` counters |
| `coverage_report` | after every other test | aggregates `gcov` output, prints the summary, applies the threshold gate |
The reset matters: gcov counters are cumulative, so without it each report would
fold in every earlier run and overstate coverage.
Upgrade to libakerror 1.0.0 Bump deps/libakerror 22 commits to 5ff8790 (1.0.0), which makes the status-name table private, moves consumer status codes to a band starting at AKERR_FIRST_CONSUMER_STATUS, enforces range ownership rather than treating it as advisory, and gives the library an soname. See deps/libakerror/UPGRADING.md. src/stdlib.c needed no changes. This library defines no status codes of its own -- it raises libakerror's AKERR_* codes and propagates errno, both inside libakerror's reserved 0-255 band -- and it never referenced AKERR_MAX_ERR_VALUE, __AKERR_ERROR_NAMES, AKERR_STATUS_RANGE_OK or AKERR_STATUS_NAME_OK. What moved was everything around the code: A -DAKSL_COVERAGE=ON build stopped configuring at all. libakerror namespaces its `mutation` target when embedded but not its `coverage` target, so it collided with ours. Shadow add_custom_target for the duration of the add_subdirectory() call and rename the dependency's to akerror_coverage, alongside the existing add_test shadow. Fix upstream and delete the workaround; recorded in TODO.md. Pin the 1.0.0 floor three ways, since no single one covers every consumption path: an #error in akstdlib.h feature-testing AKERR_FIRST_CONSUMER_STATUS, because libakerror publishes no version macro; Requires: akerror >= 1.0.0 in akstdlib.pc, which also gets consumers -lakerror transitively; and find_dependency(akerror) in akstdlibConfig.cmake. The last was already broken before this bump -- the template still carried its MyLibraryConfig placeholder with the dependency commented out, so any external find_package(akstdlib) failed with a bare "akerror::akerror not found" out of the generated targets file. Branch coverage of src/stdlib.c fell from 51.0% to 44.3% with no source or test change: the 1.0.0 PREPARE_ERROR/FAIL_* macros expand to more branches at every call site, so 337/661 became 481/1087 -- 144 more branches covered, 426 more counted. Line coverage held at 99.0% (200/202) and function coverage at 100% (21/21). Re-ratchet the CI branch gate 45 -> 40 rather than chase branches that belong to libakerror's own suite. tests/test_status_registry.c pins the contract that made the status-code migration a no-op: libakstdlib reserves no consumer range, so an application may allocate from AKERR_FIRST_CONSUMER_STATUS without coordinating with it, and every status this library raises is inside the reserved band with a name actually registered -- an unnamed one degrades to "Unknown Error" in every later stack trace, which nothing else would notice. It exercises the new ownership enforcement too, so the "reserves nothing" assertion cannot pass vacuously. ctest 13/13, ASan+UBSan 13/13, coverage 15/15 at 90/40, mutation 89.6% (155/173, unchanged). Also verified out of tree: the #error fires as the first diagnostic against a stale akerror.h, pkg-config refuses akerror 0.9.0, and an external find_package(akstdlib) consumer builds and runs against a temp-prefix install. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-30 22:20:54 -04:00
`coverage` here is this project's target. libakerror ships a `coverage` target of
its own and, unlike its `mutation` target, does not namespace it when embedded,
so a top-level `-DAKSL_COVERAGE=ON` build would collide on the name and fail to
configure at all. `CMakeLists.txt` renames the dependency's to `akerror_coverage`
on the way past — it drives its own instrumented build tree, so
`cmake --build build-coverage --target akerror_coverage` still works. The
workaround goes away when libakerror namespaces it upstream; see `TODO.md` §2.3.
Add code coverage to the CTest suite New AKSL_COVERAGE option instruments the library and its tests with --coverage -O0 and wires the report into the suite itself, so a plain ctest --test-dir build-coverage both runs the tests and produces coverage. Two CTest entries do the work, held in order by a CTest fixture rather than by declaration order so they also hold under ctest -j: coverage_reset (FIXTURES_SETUP) clears the .gcda counters before any test, since gcov counts are cumulative and would otherwise fold in earlier runs; coverage_report (FIXTURES_CLEANUP) aggregates gcov output afterwards. AKSL_COVERAGE_THRESHOLD / AKSL_COVERAGE_BRANCH_THRESHOLD gate the report, the same regression-ratchet idea as the mutation score. The `coverage` target builds, runs and prints in one step. scripts/coverage.py parses gcov's JSON output, aggregates line, branch and function counts across translation units, and lists every uncovered line and never-called function -- the actionable half, as with surviving mutants. Python stdlib plus gcc's own gcov only: no lcov, gcovr or genhtml. It also writes coverage-summary.txt (CTest hides the output of a passing test) and a Cobertura coverage.xml for CI publishers. Instrumentation is per target, so deps/libakerror stays out of the report. The mutation harness now ignores build*/ and gcov artifacts when copying the tree, so a coverage build does not slow it down. Baseline on src/stdlib.c: 52.0% of lines, 23.6% of branches, 8 of 21 functions. The uncovered functions are the untested wrappers the mutation survivors already point at (printf, ato*, stream, realpath, strhash). Verified: cmake -S . -B build-coverage -DAKSL_COVERAGE=ON cmake --build build-coverage --target coverage # 8/8, report printed ctest --test-dir build-coverage -j8 # fixture order holds cmake -S . -B build-coverage -DAKSL_COVERAGE=ON -DAKSL_COVERAGE_THRESHOLD=60 ctest --test-dir build-coverage --output-on-failure # gate fails as expected ctest --test-dir build --output-on-failure # 6/6, no .gcda emitted ctest --test-dir build-asan --output-on-failure # 6/6 scripts/mutation_test.py --target src/stdlib.c --list # 173 mutants, unchanged Totals match gcov itself: 51.98% of 202 lines, 23.60% of 661 branches. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-30 01:48:05 -04:00
CTest hides the output of a passing test, so `coverage_report` also writes
`build-coverage/coverage-summary.txt` (the same text report) and
`build-coverage/coverage.xml` (Cobertura, for CI publishers). The `coverage`
target above prints the report to the terminal for you; otherwise read the file
or use `ctest --test-dir build-coverage -V -R coverage_report`.
The report lists per-file line, branch and function coverage, then every
uncovered line and every function the suite never called — that listing is the
actionable part, the same way surviving mutants are for the harness below.
Drive the script directly for anything narrower:
```sh
scripts/coverage.py --build build-coverage # report on disk counters
scripts/coverage.py --build build-coverage --summary-only # totals only
scripts/coverage.py --build build-coverage --include tests # coverage of the tests themselves
scripts/coverage.py --build build-coverage --run-tests # reset, run ctest, report
Upgrade to libakerror 1.0.0 Bump deps/libakerror 22 commits to 5ff8790 (1.0.0), which makes the status-name table private, moves consumer status codes to a band starting at AKERR_FIRST_CONSUMER_STATUS, enforces range ownership rather than treating it as advisory, and gives the library an soname. See deps/libakerror/UPGRADING.md. src/stdlib.c needed no changes. This library defines no status codes of its own -- it raises libakerror's AKERR_* codes and propagates errno, both inside libakerror's reserved 0-255 band -- and it never referenced AKERR_MAX_ERR_VALUE, __AKERR_ERROR_NAMES, AKERR_STATUS_RANGE_OK or AKERR_STATUS_NAME_OK. What moved was everything around the code: A -DAKSL_COVERAGE=ON build stopped configuring at all. libakerror namespaces its `mutation` target when embedded but not its `coverage` target, so it collided with ours. Shadow add_custom_target for the duration of the add_subdirectory() call and rename the dependency's to akerror_coverage, alongside the existing add_test shadow. Fix upstream and delete the workaround; recorded in TODO.md. Pin the 1.0.0 floor three ways, since no single one covers every consumption path: an #error in akstdlib.h feature-testing AKERR_FIRST_CONSUMER_STATUS, because libakerror publishes no version macro; Requires: akerror >= 1.0.0 in akstdlib.pc, which also gets consumers -lakerror transitively; and find_dependency(akerror) in akstdlibConfig.cmake. The last was already broken before this bump -- the template still carried its MyLibraryConfig placeholder with the dependency commented out, so any external find_package(akstdlib) failed with a bare "akerror::akerror not found" out of the generated targets file. Branch coverage of src/stdlib.c fell from 51.0% to 44.3% with no source or test change: the 1.0.0 PREPARE_ERROR/FAIL_* macros expand to more branches at every call site, so 337/661 became 481/1087 -- 144 more branches covered, 426 more counted. Line coverage held at 99.0% (200/202) and function coverage at 100% (21/21). Re-ratchet the CI branch gate 45 -> 40 rather than chase branches that belong to libakerror's own suite. tests/test_status_registry.c pins the contract that made the status-code migration a no-op: libakstdlib reserves no consumer range, so an application may allocate from AKERR_FIRST_CONSUMER_STATUS without coordinating with it, and every status this library raises is inside the reserved band with a name actually registered -- an unnamed one degrades to "Unknown Error" in every later stack trace, which nothing else would notice. It exercises the new ownership enforcement too, so the "reserves nothing" assertion cannot pass vacuously. ctest 13/13, ASan+UBSan 13/13, coverage 15/15 at 90/40, mutation 89.6% (155/173, unchanged). Also verified out of tree: the #error fires as the first diagnostic against a stale akerror.h, pkg-config refuses akerror 0.9.0, and an external find_package(akstdlib) consumer builds and runs against a temp-prefix install. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-30 22:20:54 -04:00
scripts/coverage.py --build build-coverage --threshold 90 --branch-threshold 40
Add code coverage to the CTest suite New AKSL_COVERAGE option instruments the library and its tests with --coverage -O0 and wires the report into the suite itself, so a plain ctest --test-dir build-coverage both runs the tests and produces coverage. Two CTest entries do the work, held in order by a CTest fixture rather than by declaration order so they also hold under ctest -j: coverage_reset (FIXTURES_SETUP) clears the .gcda counters before any test, since gcov counts are cumulative and would otherwise fold in earlier runs; coverage_report (FIXTURES_CLEANUP) aggregates gcov output afterwards. AKSL_COVERAGE_THRESHOLD / AKSL_COVERAGE_BRANCH_THRESHOLD gate the report, the same regression-ratchet idea as the mutation score. The `coverage` target builds, runs and prints in one step. scripts/coverage.py parses gcov's JSON output, aggregates line, branch and function counts across translation units, and lists every uncovered line and never-called function -- the actionable half, as with surviving mutants. Python stdlib plus gcc's own gcov only: no lcov, gcovr or genhtml. It also writes coverage-summary.txt (CTest hides the output of a passing test) and a Cobertura coverage.xml for CI publishers. Instrumentation is per target, so deps/libakerror stays out of the report. The mutation harness now ignores build*/ and gcov artifacts when copying the tree, so a coverage build does not slow it down. Baseline on src/stdlib.c: 52.0% of lines, 23.6% of branches, 8 of 21 functions. The uncovered functions are the untested wrappers the mutation survivors already point at (printf, ato*, stream, realpath, strhash). Verified: cmake -S . -B build-coverage -DAKSL_COVERAGE=ON cmake --build build-coverage --target coverage # 8/8, report printed ctest --test-dir build-coverage -j8 # fixture order holds cmake -S . -B build-coverage -DAKSL_COVERAGE=ON -DAKSL_COVERAGE_THRESHOLD=60 ctest --test-dir build-coverage --output-on-failure # gate fails as expected ctest --test-dir build --output-on-failure # 6/6, no .gcda emitted ctest --test-dir build-asan --output-on-failure # 6/6 scripts/mutation_test.py --target src/stdlib.c --list # 173 mutants, unchanged Totals match gcov itself: 51.98% of 202 lines, 23.60% of 661 branches. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-30 01:48:05 -04:00
```
It needs nothing but Python 3 and gcc's own `gcov` — no lcov, gcovr or genhtml.
To gate on coverage, set the threshold at configure time; `coverage_report` then
fails below it, and the same regression-ratchet logic applies as for the mutation
score:
```sh
cmake -S . -B build-coverage -DAKSL_COVERAGE=ON \
Upgrade to libakerror 1.0.0 Bump deps/libakerror 22 commits to 5ff8790 (1.0.0), which makes the status-name table private, moves consumer status codes to a band starting at AKERR_FIRST_CONSUMER_STATUS, enforces range ownership rather than treating it as advisory, and gives the library an soname. See deps/libakerror/UPGRADING.md. src/stdlib.c needed no changes. This library defines no status codes of its own -- it raises libakerror's AKERR_* codes and propagates errno, both inside libakerror's reserved 0-255 band -- and it never referenced AKERR_MAX_ERR_VALUE, __AKERR_ERROR_NAMES, AKERR_STATUS_RANGE_OK or AKERR_STATUS_NAME_OK. What moved was everything around the code: A -DAKSL_COVERAGE=ON build stopped configuring at all. libakerror namespaces its `mutation` target when embedded but not its `coverage` target, so it collided with ours. Shadow add_custom_target for the duration of the add_subdirectory() call and rename the dependency's to akerror_coverage, alongside the existing add_test shadow. Fix upstream and delete the workaround; recorded in TODO.md. Pin the 1.0.0 floor three ways, since no single one covers every consumption path: an #error in akstdlib.h feature-testing AKERR_FIRST_CONSUMER_STATUS, because libakerror publishes no version macro; Requires: akerror >= 1.0.0 in akstdlib.pc, which also gets consumers -lakerror transitively; and find_dependency(akerror) in akstdlibConfig.cmake. The last was already broken before this bump -- the template still carried its MyLibraryConfig placeholder with the dependency commented out, so any external find_package(akstdlib) failed with a bare "akerror::akerror not found" out of the generated targets file. Branch coverage of src/stdlib.c fell from 51.0% to 44.3% with no source or test change: the 1.0.0 PREPARE_ERROR/FAIL_* macros expand to more branches at every call site, so 337/661 became 481/1087 -- 144 more branches covered, 426 more counted. Line coverage held at 99.0% (200/202) and function coverage at 100% (21/21). Re-ratchet the CI branch gate 45 -> 40 rather than chase branches that belong to libakerror's own suite. tests/test_status_registry.c pins the contract that made the status-code migration a no-op: libakstdlib reserves no consumer range, so an application may allocate from AKERR_FIRST_CONSUMER_STATUS without coordinating with it, and every status this library raises is inside the reserved band with a name actually registered -- an unnamed one degrades to "Unknown Error" in every later stack trace, which nothing else would notice. It exercises the new ownership enforcement too, so the "reserves nothing" assertion cannot pass vacuously. ctest 13/13, ASan+UBSan 13/13, coverage 15/15 at 90/40, mutation 89.6% (155/173, unchanged). Also verified out of tree: the #error fires as the first diagnostic against a stale akerror.h, pkg-config refuses akerror 0.9.0, and an external find_package(akstdlib) consumer builds and runs against a temp-prefix install. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-30 22:20:54 -04:00
-DAKSL_COVERAGE_THRESHOLD=90 -DAKSL_COVERAGE_BRANCH_THRESHOLD=40
Add code coverage to the CTest suite New AKSL_COVERAGE option instruments the library and its tests with --coverage -O0 and wires the report into the suite itself, so a plain ctest --test-dir build-coverage both runs the tests and produces coverage. Two CTest entries do the work, held in order by a CTest fixture rather than by declaration order so they also hold under ctest -j: coverage_reset (FIXTURES_SETUP) clears the .gcda counters before any test, since gcov counts are cumulative and would otherwise fold in earlier runs; coverage_report (FIXTURES_CLEANUP) aggregates gcov output afterwards. AKSL_COVERAGE_THRESHOLD / AKSL_COVERAGE_BRANCH_THRESHOLD gate the report, the same regression-ratchet idea as the mutation score. The `coverage` target builds, runs and prints in one step. scripts/coverage.py parses gcov's JSON output, aggregates line, branch and function counts across translation units, and lists every uncovered line and never-called function -- the actionable half, as with surviving mutants. Python stdlib plus gcc's own gcov only: no lcov, gcovr or genhtml. It also writes coverage-summary.txt (CTest hides the output of a passing test) and a Cobertura coverage.xml for CI publishers. Instrumentation is per target, so deps/libakerror stays out of the report. The mutation harness now ignores build*/ and gcov artifacts when copying the tree, so a coverage build does not slow it down. Baseline on src/stdlib.c: 52.0% of lines, 23.6% of branches, 8 of 21 functions. The uncovered functions are the untested wrappers the mutation survivors already point at (printf, ato*, stream, realpath, strhash). Verified: cmake -S . -B build-coverage -DAKSL_COVERAGE=ON cmake --build build-coverage --target coverage # 8/8, report printed ctest --test-dir build-coverage -j8 # fixture order holds cmake -S . -B build-coverage -DAKSL_COVERAGE=ON -DAKSL_COVERAGE_THRESHOLD=60 ctest --test-dir build-coverage --output-on-failure # gate fails as expected ctest --test-dir build --output-on-failure # 6/6, no .gcda emitted ctest --test-dir build-asan --output-on-failure # 6/6 scripts/mutation_test.py --target src/stdlib.c --list # 173 mutants, unchanged Totals match gcov itself: 51.98% of 202 lines, 23.60% of 661 branches. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-30 01:48:05 -04:00
```
Upgrade to libakerror 1.0.0 Bump deps/libakerror 22 commits to 5ff8790 (1.0.0), which makes the status-name table private, moves consumer status codes to a band starting at AKERR_FIRST_CONSUMER_STATUS, enforces range ownership rather than treating it as advisory, and gives the library an soname. See deps/libakerror/UPGRADING.md. src/stdlib.c needed no changes. This library defines no status codes of its own -- it raises libakerror's AKERR_* codes and propagates errno, both inside libakerror's reserved 0-255 band -- and it never referenced AKERR_MAX_ERR_VALUE, __AKERR_ERROR_NAMES, AKERR_STATUS_RANGE_OK or AKERR_STATUS_NAME_OK. What moved was everything around the code: A -DAKSL_COVERAGE=ON build stopped configuring at all. libakerror namespaces its `mutation` target when embedded but not its `coverage` target, so it collided with ours. Shadow add_custom_target for the duration of the add_subdirectory() call and rename the dependency's to akerror_coverage, alongside the existing add_test shadow. Fix upstream and delete the workaround; recorded in TODO.md. Pin the 1.0.0 floor three ways, since no single one covers every consumption path: an #error in akstdlib.h feature-testing AKERR_FIRST_CONSUMER_STATUS, because libakerror publishes no version macro; Requires: akerror >= 1.0.0 in akstdlib.pc, which also gets consumers -lakerror transitively; and find_dependency(akerror) in akstdlibConfig.cmake. The last was already broken before this bump -- the template still carried its MyLibraryConfig placeholder with the dependency commented out, so any external find_package(akstdlib) failed with a bare "akerror::akerror not found" out of the generated targets file. Branch coverage of src/stdlib.c fell from 51.0% to 44.3% with no source or test change: the 1.0.0 PREPARE_ERROR/FAIL_* macros expand to more branches at every call site, so 337/661 became 481/1087 -- 144 more branches covered, 426 more counted. Line coverage held at 99.0% (200/202) and function coverage at 100% (21/21). Re-ratchet the CI branch gate 45 -> 40 rather than chase branches that belong to libakerror's own suite. tests/test_status_registry.c pins the contract that made the status-code migration a no-op: libakstdlib reserves no consumer range, so an application may allocate from AKERR_FIRST_CONSUMER_STATUS without coordinating with it, and every status this library raises is inside the reserved band with a name actually registered -- an unnamed one degrades to "Unknown Error" in every later stack trace, which nothing else would notice. It exercises the new ownership enforcement too, so the "reserves nothing" assertion cannot pass vacuously. ctest 13/13, ASan+UBSan 13/13, coverage 15/15 at 90/40, mutation 89.6% (155/173, unchanged). Also verified out of tree: the #error fires as the first diagnostic against a stale akerror.h, pkg-config refuses akerror 0.9.0, and an external find_package(akstdlib) consumer builds and runs against a temp-prefix install. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-30 22:20:54 -04:00
**Where it stands.** `src/stdlib.c` is at **99.0% of lines (200/202)**, **44.3% of
branches (481/1087)** and **21/21 functions**, so 90/40 above is a ratchet with
headroom rather than a target. The two uncovered lines are both
Test the libc wrappers: 52% -> 99% line coverage Every wrapper outside the list and tree code was untested. Six new test files close that, following the plan already written in TODO.md 1.2-1.6: test_stream.c fopen/fread/fwrite/fclose -- happy paths, the round trip, AKERR_EOF on a short read, AKERR_IO on a stream opened in the wrong mode, ENOENT, and the NULL guards test_format.c printf/fprintf/sprintf -- text *and* count asserted (stdout is pointed at a temp file to check aksl_printf), all eight NULL guards, EBADF on a read-only stream, and 512 variadic calls in a loop as sanitizer cover for the missing va_end test_convert.c ato{i,l,ll,f} happy paths, negatives, leading whitespace, NULL guards test_path.c realpath on a file and on a symlink, both compared against realpath(3) since TMPDIR may itself be a link; ENOENT, ENOTDIR, NULL path test_strhash.c djb2 known-answer vectors, len == 0, embedded NUL, stability, NULL guards test_convert_strict.c known-failing (2.1.5): the AKERR_VALUE / ERANGE contract the ato* family cannot express today test_tree.c gains the BFS AKERR_NOT_IMPLEMENTED contract, NULL arguments, and a callback error that is not AKERR_ITERATOR_BREAK propagating out. Tests deliberately say nothing about behaviour TODO.md records as defective -- unchecked ptr/mode/resolved_path, short transfers reported as success, *count left at -1, the djb2 sign extension -- so the eventual fix does not have to come with a test rewrite. Each failure case in test_path.c passes a zeroed buffer, because the wrapper's own error path formats resolved_path with %s (2.1.6). aksl_capture.h gains aksl_temp_file() with an atexit unlink backstop. Without it every test that fails before its own unlink leaves temp files behind -- which is the normal case for a known-failing test, and happens 173 times over in a mutation run. Coverage on src/stdlib.c: 52.0% -> 99.0% of lines (200/202), 23.6% -> 51.0% of branches, 8/21 -> 21/21 functions. The two uncovered lines are both `} HANDLE(e, AKERR_ITERATOR_BREAK) {`, where the macro starts with the `break;` of PROCESS's `case 0:` arm -- reachable only via a non-NULL error context whose status is zero, the pathology 2.2.1 exists to remove. Mutation score on src/stdlib.c: 46.8% -> 89.6% (155/173 killed). CI, the pre-push hook and the docs ratchet from 40 to 80 accordingly, and the 18 survivors are grouped by cause in TODO.md and README.md. A new CI coverage job gates at 90% lines / 45% branches. Verified: ctest --test-dir build # 12/12 ctest --test-dir build-asan # 12/12 under ASan + UBSan ctest --test-dir build-coverage # 14/14, report attached ctest --test-dir build -j8 --repeat until-fail:3 gcc -Wall -Wextra -c on all nine test files # no warnings python3 scripts/mutation_test.py --target src/stdlib.c # 89.6% No temp files left in /tmp after any of the above. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-30 02:07:08 -04:00
`} HANDLE(e, AKERR_ITERATOR_BREAK) {` — in libakerror that macro begins with the
`break;` belonging to `PROCESS`'s `case 0:` arm, which is only reachable when a
callback returns a non-NULL error context whose status is *zero*. That is the
pathological case §2.2.1 of `TODO.md` exists to remove, so it is left uncovered
deliberately rather than pinned by a test.
Branch coverage sits far below line coverage because most branches in this file
are inside the `FAIL_*`/`ATTEMPT`/`FINISH` macro expansions — pool exhaustion,
stack-trace buffer limits, `akerr_valid_error_address` failures — and belong to
Upgrade to libakerror 1.0.0 Bump deps/libakerror 22 commits to 5ff8790 (1.0.0), which makes the status-name table private, moves consumer status codes to a band starting at AKERR_FIRST_CONSUMER_STATUS, enforces range ownership rather than treating it as advisory, and gives the library an soname. See deps/libakerror/UPGRADING.md. src/stdlib.c needed no changes. This library defines no status codes of its own -- it raises libakerror's AKERR_* codes and propagates errno, both inside libakerror's reserved 0-255 band -- and it never referenced AKERR_MAX_ERR_VALUE, __AKERR_ERROR_NAMES, AKERR_STATUS_RANGE_OK or AKERR_STATUS_NAME_OK. What moved was everything around the code: A -DAKSL_COVERAGE=ON build stopped configuring at all. libakerror namespaces its `mutation` target when embedded but not its `coverage` target, so it collided with ours. Shadow add_custom_target for the duration of the add_subdirectory() call and rename the dependency's to akerror_coverage, alongside the existing add_test shadow. Fix upstream and delete the workaround; recorded in TODO.md. Pin the 1.0.0 floor three ways, since no single one covers every consumption path: an #error in akstdlib.h feature-testing AKERR_FIRST_CONSUMER_STATUS, because libakerror publishes no version macro; Requires: akerror >= 1.0.0 in akstdlib.pc, which also gets consumers -lakerror transitively; and find_dependency(akerror) in akstdlibConfig.cmake. The last was already broken before this bump -- the template still carried its MyLibraryConfig placeholder with the dependency commented out, so any external find_package(akstdlib) failed with a bare "akerror::akerror not found" out of the generated targets file. Branch coverage of src/stdlib.c fell from 51.0% to 44.3% with no source or test change: the 1.0.0 PREPARE_ERROR/FAIL_* macros expand to more branches at every call site, so 337/661 became 481/1087 -- 144 more branches covered, 426 more counted. Line coverage held at 99.0% (200/202) and function coverage at 100% (21/21). Re-ratchet the CI branch gate 45 -> 40 rather than chase branches that belong to libakerror's own suite. tests/test_status_registry.c pins the contract that made the status-code migration a no-op: libakstdlib reserves no consumer range, so an application may allocate from AKERR_FIRST_CONSUMER_STATUS without coordinating with it, and every status this library raises is inside the reserved band with a name actually registered -- an unnamed one degrades to "Unknown Error" in every later stack trace, which nothing else would notice. It exercises the new ownership enforcement too, so the "reserves nothing" assertion cannot pass vacuously. ctest 13/13, ASan+UBSan 13/13, coverage 15/15 at 90/40, mutation 89.6% (155/173, unchanged). Also verified out of tree: the #error fires as the first diagnostic against a stale akerror.h, pkg-config refuses akerror 0.9.0, and an external find_package(akstdlib) consumer builds and runs against a temp-prefix install. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-30 22:20:54 -04:00
libakerror's own suite rather than to this one. The libakerror 1.0.0 bump made
that gap wider without changing a line of this library: the branch denominator
went from 661 to 1087 as those macros grew, so the same tests that scored 51.0%
(337/661) now score 44.3% (481/1087). The gate moved 45 → 40 to match; line and
function coverage did not move at all.
Test the libc wrappers: 52% -> 99% line coverage Every wrapper outside the list and tree code was untested. Six new test files close that, following the plan already written in TODO.md 1.2-1.6: test_stream.c fopen/fread/fwrite/fclose -- happy paths, the round trip, AKERR_EOF on a short read, AKERR_IO on a stream opened in the wrong mode, ENOENT, and the NULL guards test_format.c printf/fprintf/sprintf -- text *and* count asserted (stdout is pointed at a temp file to check aksl_printf), all eight NULL guards, EBADF on a read-only stream, and 512 variadic calls in a loop as sanitizer cover for the missing va_end test_convert.c ato{i,l,ll,f} happy paths, negatives, leading whitespace, NULL guards test_path.c realpath on a file and on a symlink, both compared against realpath(3) since TMPDIR may itself be a link; ENOENT, ENOTDIR, NULL path test_strhash.c djb2 known-answer vectors, len == 0, embedded NUL, stability, NULL guards test_convert_strict.c known-failing (2.1.5): the AKERR_VALUE / ERANGE contract the ato* family cannot express today test_tree.c gains the BFS AKERR_NOT_IMPLEMENTED contract, NULL arguments, and a callback error that is not AKERR_ITERATOR_BREAK propagating out. Tests deliberately say nothing about behaviour TODO.md records as defective -- unchecked ptr/mode/resolved_path, short transfers reported as success, *count left at -1, the djb2 sign extension -- so the eventual fix does not have to come with a test rewrite. Each failure case in test_path.c passes a zeroed buffer, because the wrapper's own error path formats resolved_path with %s (2.1.6). aksl_capture.h gains aksl_temp_file() with an atexit unlink backstop. Without it every test that fails before its own unlink leaves temp files behind -- which is the normal case for a known-failing test, and happens 173 times over in a mutation run. Coverage on src/stdlib.c: 52.0% -> 99.0% of lines (200/202), 23.6% -> 51.0% of branches, 8/21 -> 21/21 functions. The two uncovered lines are both `} HANDLE(e, AKERR_ITERATOR_BREAK) {`, where the macro starts with the `break;` of PROCESS's `case 0:` arm -- reachable only via a non-NULL error context whose status is zero, the pathology 2.2.1 exists to remove. Mutation score on src/stdlib.c: 46.8% -> 89.6% (155/173 killed). CI, the pre-push hook and the docs ratchet from 40 to 80 accordingly, and the 18 survivors are grouped by cause in TODO.md and README.md. A new CI coverage job gates at 90% lines / 45% branches. Verified: ctest --test-dir build # 12/12 ctest --test-dir build-asan # 12/12 under ASan + UBSan ctest --test-dir build-coverage # 14/14, report attached ctest --test-dir build -j8 --repeat until-fail:3 gcc -Wall -Wextra -c on all nine test files # no warnings python3 scripts/mutation_test.py --target src/stdlib.c # 89.6% No temp files left in /tmp after any of the above. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-30 02:07:08 -04:00
Add code coverage to the CTest suite New AKSL_COVERAGE option instruments the library and its tests with --coverage -O0 and wires the report into the suite itself, so a plain ctest --test-dir build-coverage both runs the tests and produces coverage. Two CTest entries do the work, held in order by a CTest fixture rather than by declaration order so they also hold under ctest -j: coverage_reset (FIXTURES_SETUP) clears the .gcda counters before any test, since gcov counts are cumulative and would otherwise fold in earlier runs; coverage_report (FIXTURES_CLEANUP) aggregates gcov output afterwards. AKSL_COVERAGE_THRESHOLD / AKSL_COVERAGE_BRANCH_THRESHOLD gate the report, the same regression-ratchet idea as the mutation score. The `coverage` target builds, runs and prints in one step. scripts/coverage.py parses gcov's JSON output, aggregates line, branch and function counts across translation units, and lists every uncovered line and never-called function -- the actionable half, as with surviving mutants. Python stdlib plus gcc's own gcov only: no lcov, gcovr or genhtml. It also writes coverage-summary.txt (CTest hides the output of a passing test) and a Cobertura coverage.xml for CI publishers. Instrumentation is per target, so deps/libakerror stays out of the report. The mutation harness now ignores build*/ and gcov artifacts when copying the tree, so a coverage build does not slow it down. Baseline on src/stdlib.c: 52.0% of lines, 23.6% of branches, 8 of 21 functions. The uncovered functions are the untested wrappers the mutation survivors already point at (printf, ato*, stream, realpath, strhash). Verified: cmake -S . -B build-coverage -DAKSL_COVERAGE=ON cmake --build build-coverage --target coverage # 8/8, report printed ctest --test-dir build-coverage -j8 # fixture order holds cmake -S . -B build-coverage -DAKSL_COVERAGE=ON -DAKSL_COVERAGE_THRESHOLD=60 ctest --test-dir build-coverage --output-on-failure # gate fails as expected ctest --test-dir build --output-on-failure # 6/6, no .gcda emitted ctest --test-dir build-asan --output-on-failure # 6/6 scripts/mutation_test.py --target src/stdlib.c --list # 173 mutants, unchanged Totals match gcov itself: 51.98% of 202 lines, 23.60% of 661 branches. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-30 01:48:05 -04:00
Two caveats. Coverage is measured at `-O0`, because the optimizer reorders lines
until per-line counts stop matching the source — so a coverage build is not the
build to profile. And gcov flushes its counters at normal process exit, which an
`AKSL_WILL_FAIL_TESTS` entry that aborts by design never reaches: such a test
contributes no coverage data at all, so lines only it reaches are reported as
uncovered.
### 4. Mutation testing
The suite tells you the library works. Mutation testing tells you the *suite*
works: it breaks the library in small ways, one at a time, and checks that the
tests notice.
```sh
cmake --build build --target mutation # src/stdlib.c + include/akstdlib.h
```
or drive the script directly for a faster or narrower run:
```sh
scripts/mutation_test.py --target src/stdlib.c # C source only
scripts/mutation_test.py --target src/stdlib.c --list # enumerate, build nothing
scripts/mutation_test.py --target src/stdlib.c --max-mutants 20
Test the libc wrappers: 52% -> 99% line coverage Every wrapper outside the list and tree code was untested. Six new test files close that, following the plan already written in TODO.md 1.2-1.6: test_stream.c fopen/fread/fwrite/fclose -- happy paths, the round trip, AKERR_EOF on a short read, AKERR_IO on a stream opened in the wrong mode, ENOENT, and the NULL guards test_format.c printf/fprintf/sprintf -- text *and* count asserted (stdout is pointed at a temp file to check aksl_printf), all eight NULL guards, EBADF on a read-only stream, and 512 variadic calls in a loop as sanitizer cover for the missing va_end test_convert.c ato{i,l,ll,f} happy paths, negatives, leading whitespace, NULL guards test_path.c realpath on a file and on a symlink, both compared against realpath(3) since TMPDIR may itself be a link; ENOENT, ENOTDIR, NULL path test_strhash.c djb2 known-answer vectors, len == 0, embedded NUL, stability, NULL guards test_convert_strict.c known-failing (2.1.5): the AKERR_VALUE / ERANGE contract the ato* family cannot express today test_tree.c gains the BFS AKERR_NOT_IMPLEMENTED contract, NULL arguments, and a callback error that is not AKERR_ITERATOR_BREAK propagating out. Tests deliberately say nothing about behaviour TODO.md records as defective -- unchecked ptr/mode/resolved_path, short transfers reported as success, *count left at -1, the djb2 sign extension -- so the eventual fix does not have to come with a test rewrite. Each failure case in test_path.c passes a zeroed buffer, because the wrapper's own error path formats resolved_path with %s (2.1.6). aksl_capture.h gains aksl_temp_file() with an atexit unlink backstop. Without it every test that fails before its own unlink leaves temp files behind -- which is the normal case for a known-failing test, and happens 173 times over in a mutation run. Coverage on src/stdlib.c: 52.0% -> 99.0% of lines (200/202), 23.6% -> 51.0% of branches, 8/21 -> 21/21 functions. The two uncovered lines are both `} HANDLE(e, AKERR_ITERATOR_BREAK) {`, where the macro starts with the `break;` of PROCESS's `case 0:` arm -- reachable only via a non-NULL error context whose status is zero, the pathology 2.2.1 exists to remove. Mutation score on src/stdlib.c: 46.8% -> 89.6% (155/173 killed). CI, the pre-push hook and the docs ratchet from 40 to 80 accordingly, and the 18 survivors are grouped by cause in TODO.md and README.md. A new CI coverage job gates at 90% lines / 45% branches. Verified: ctest --test-dir build # 12/12 ctest --test-dir build-asan # 12/12 under ASan + UBSan ctest --test-dir build-coverage # 14/14, report attached ctest --test-dir build -j8 --repeat until-fail:3 gcc -Wall -Wextra -c on all nine test files # no warnings python3 scripts/mutation_test.py --target src/stdlib.c # 89.6% No temp files left in /tmp after any of the above. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-30 02:07:08 -04:00
scripts/mutation_test.py --target src/stdlib.c --threshold 80
```
A mutant that makes the tests fail is *killed* (good); one the tests still pass
is a *survivor*, and names a missing test. The score is `killed / total`, and the
run prints every survivor with `file:line` and the exact edit. The harness never
touches your working tree — it copies the repo to a scratch directory and mutates
the copy.
Test the libc wrappers: 52% -> 99% line coverage Every wrapper outside the list and tree code was untested. Six new test files close that, following the plan already written in TODO.md 1.2-1.6: test_stream.c fopen/fread/fwrite/fclose -- happy paths, the round trip, AKERR_EOF on a short read, AKERR_IO on a stream opened in the wrong mode, ENOENT, and the NULL guards test_format.c printf/fprintf/sprintf -- text *and* count asserted (stdout is pointed at a temp file to check aksl_printf), all eight NULL guards, EBADF on a read-only stream, and 512 variadic calls in a loop as sanitizer cover for the missing va_end test_convert.c ato{i,l,ll,f} happy paths, negatives, leading whitespace, NULL guards test_path.c realpath on a file and on a symlink, both compared against realpath(3) since TMPDIR may itself be a link; ENOENT, ENOTDIR, NULL path test_strhash.c djb2 known-answer vectors, len == 0, embedded NUL, stability, NULL guards test_convert_strict.c known-failing (2.1.5): the AKERR_VALUE / ERANGE contract the ato* family cannot express today test_tree.c gains the BFS AKERR_NOT_IMPLEMENTED contract, NULL arguments, and a callback error that is not AKERR_ITERATOR_BREAK propagating out. Tests deliberately say nothing about behaviour TODO.md records as defective -- unchecked ptr/mode/resolved_path, short transfers reported as success, *count left at -1, the djb2 sign extension -- so the eventual fix does not have to come with a test rewrite. Each failure case in test_path.c passes a zeroed buffer, because the wrapper's own error path formats resolved_path with %s (2.1.6). aksl_capture.h gains aksl_temp_file() with an atexit unlink backstop. Without it every test that fails before its own unlink leaves temp files behind -- which is the normal case for a known-failing test, and happens 173 times over in a mutation run. Coverage on src/stdlib.c: 52.0% -> 99.0% of lines (200/202), 23.6% -> 51.0% of branches, 8/21 -> 21/21 functions. The two uncovered lines are both `} HANDLE(e, AKERR_ITERATOR_BREAK) {`, where the macro starts with the `break;` of PROCESS's `case 0:` arm -- reachable only via a non-NULL error context whose status is zero, the pathology 2.2.1 exists to remove. Mutation score on src/stdlib.c: 46.8% -> 89.6% (155/173 killed). CI, the pre-push hook and the docs ratchet from 40 to 80 accordingly, and the 18 survivors are grouped by cause in TODO.md and README.md. A new CI coverage job gates at 90% lines / 45% branches. Verified: ctest --test-dir build # 12/12 ctest --test-dir build-asan # 12/12 under ASan + UBSan ctest --test-dir build-coverage # 14/14, report attached ctest --test-dir build -j8 --repeat until-fail:3 gcc -Wall -Wextra -c on all nine test files # no warnings python3 scripts/mutation_test.py --target src/stdlib.c # 89.6% No temp files left in /tmp after any of the above. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-30 02:07:08 -04:00
CI runs the `src/stdlib.c` set with `--threshold 80`. That is a regression ratchet
rather than a quality bar: the current score is **89.6% (155/173 killed)**, up
from 46.8% before the wrapper tests landed. Raise the threshold as the remaining
survivors are turned into assertions.
The 18 survivors cluster in three places, and each names a real gap rather than a
test-harness artifact:
- **Statements whose absence nothing observes** — deleting `free(ptr)`,
`obj->next = NULL`, or a `SUCCEED_RETURN` leaves behaviour the suite does not
look at (a leak, a stale pointer, a success that was already NULL).
- **The `aksl_list_append` cycle/tail walk** (`tail = slow`, `slow = slow->next`,
`tail = fast`) — the function is broken in exactly this area (`TODO.md` §2.1.1),
so its known-failing test cannot pin the internals yet.
- **`lalloc`/`lfree` defaulting in `aksl_tree_iterate`** — dead parameters
(§2.2.8): they are defaulted and then never called, so inverting the guard
changes nothing observable.
## The pre-push hook
`.githooks/pre-push` runs the fast harnesses — the default build and the
sanitizer build, each followed by `ctest` — before letting a push out. Enable it
once per clone:
```sh
git config core.hooksPath .githooks
```
It only builds when there are commits to push (a branch deletion is a no-op), and
it builds under `.git/aksl-prepush` so it never disturbs your own `build/`.
```sh
AKSL_HOOK_MUTATION=1 git push # also run the mutation gate (slow)
git push --no-verify # skip the hook entirely
```
Test the libc wrappers: 52% -> 99% line coverage Every wrapper outside the list and tree code was untested. Six new test files close that, following the plan already written in TODO.md 1.2-1.6: test_stream.c fopen/fread/fwrite/fclose -- happy paths, the round trip, AKERR_EOF on a short read, AKERR_IO on a stream opened in the wrong mode, ENOENT, and the NULL guards test_format.c printf/fprintf/sprintf -- text *and* count asserted (stdout is pointed at a temp file to check aksl_printf), all eight NULL guards, EBADF on a read-only stream, and 512 variadic calls in a loop as sanitizer cover for the missing va_end test_convert.c ato{i,l,ll,f} happy paths, negatives, leading whitespace, NULL guards test_path.c realpath on a file and on a symlink, both compared against realpath(3) since TMPDIR may itself be a link; ENOENT, ENOTDIR, NULL path test_strhash.c djb2 known-answer vectors, len == 0, embedded NUL, stability, NULL guards test_convert_strict.c known-failing (2.1.5): the AKERR_VALUE / ERANGE contract the ato* family cannot express today test_tree.c gains the BFS AKERR_NOT_IMPLEMENTED contract, NULL arguments, and a callback error that is not AKERR_ITERATOR_BREAK propagating out. Tests deliberately say nothing about behaviour TODO.md records as defective -- unchecked ptr/mode/resolved_path, short transfers reported as success, *count left at -1, the djb2 sign extension -- so the eventual fix does not have to come with a test rewrite. Each failure case in test_path.c passes a zeroed buffer, because the wrapper's own error path formats resolved_path with %s (2.1.6). aksl_capture.h gains aksl_temp_file() with an atexit unlink backstop. Without it every test that fails before its own unlink leaves temp files behind -- which is the normal case for a known-failing test, and happens 173 times over in a mutation run. Coverage on src/stdlib.c: 52.0% -> 99.0% of lines (200/202), 23.6% -> 51.0% of branches, 8/21 -> 21/21 functions. The two uncovered lines are both `} HANDLE(e, AKERR_ITERATOR_BREAK) {`, where the macro starts with the `break;` of PROCESS's `case 0:` arm -- reachable only via a non-NULL error context whose status is zero, the pathology 2.2.1 exists to remove. Mutation score on src/stdlib.c: 46.8% -> 89.6% (155/173 killed). CI, the pre-push hook and the docs ratchet from 40 to 80 accordingly, and the 18 survivors are grouped by cause in TODO.md and README.md. A new CI coverage job gates at 90% lines / 45% branches. Verified: ctest --test-dir build # 12/12 ctest --test-dir build-asan # 12/12 under ASan + UBSan ctest --test-dir build-coverage # 14/14, report attached ctest --test-dir build -j8 --repeat until-fail:3 gcc -Wall -Wextra -c on all nine test files # no warnings python3 scripts/mutation_test.py --target src/stdlib.c # 89.6% No temp files left in /tmp after any of the above. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-30 02:07:08 -04:00
Other knobs: `AKSL_MUTATION_THRESHOLD` (default 80, keep it in step with
`.gitea/workflows/ci.yaml`) and `AKSL_HOOK_BUILD_DIR`.