Test the libc wrappers: 52% -> 99% line coverage

Every wrapper outside the list and tree code was untested. Six new test
files close that, following the plan already written in TODO.md 1.2-1.6:

  test_stream.c   fopen/fread/fwrite/fclose -- happy paths, the round
                  trip, AKERR_EOF on a short read, AKERR_IO on a stream
                  opened in the wrong mode, ENOENT, and the NULL guards
  test_format.c   printf/fprintf/sprintf -- text *and* count asserted
                  (stdout is pointed at a temp file to check aksl_printf),
                  all eight NULL guards, EBADF on a read-only stream, and
                  512 variadic calls in a loop as sanitizer cover for the
                  missing va_end
  test_convert.c  ato{i,l,ll,f} happy paths, negatives, leading
                  whitespace, NULL guards
  test_path.c     realpath on a file and on a symlink, both compared
                  against realpath(3) since TMPDIR may itself be a link;
                  ENOENT, ENOTDIR, NULL path
  test_strhash.c  djb2 known-answer vectors, len == 0, embedded NUL,
                  stability, NULL guards
  test_convert_strict.c
                  known-failing (2.1.5): the AKERR_VALUE / ERANGE
                  contract the ato* family cannot express today

test_tree.c gains the BFS AKERR_NOT_IMPLEMENTED contract, NULL arguments,
and a callback error that is not AKERR_ITERATOR_BREAK propagating out.

Tests deliberately say nothing about behaviour TODO.md records as
defective -- unchecked ptr/mode/resolved_path, short transfers reported as
success, *count left at -1, the djb2 sign extension -- so the eventual fix
does not have to come with a test rewrite. Each failure case in
test_path.c passes a zeroed buffer, because the wrapper's own error path
formats resolved_path with %s (2.1.6).

aksl_capture.h gains aksl_temp_file() with an atexit unlink backstop.
Without it every test that fails before its own unlink leaves temp files
behind -- which is the normal case for a known-failing test, and happens
173 times over in a mutation run.

Coverage on src/stdlib.c: 52.0% -> 99.0% of lines (200/202), 23.6% ->
51.0% of branches, 8/21 -> 21/21 functions. The two uncovered lines are
both `} HANDLE(e, AKERR_ITERATOR_BREAK) {`, where the macro starts with
the `break;` of PROCESS's `case 0:` arm -- reachable only via a non-NULL
error context whose status is zero, the pathology 2.2.1 exists to remove.

Mutation score on src/stdlib.c: 46.8% -> 89.6% (155/173 killed). CI, the
pre-push hook and the docs ratchet from 40 to 80 accordingly, and the 18
survivors are grouped by cause in TODO.md and README.md. A new CI
coverage job gates at 90% lines / 45% branches.

Verified:
  ctest --test-dir build            # 12/12
  ctest --test-dir build-asan       # 12/12 under ASan + UBSan
  ctest --test-dir build-coverage   # 14/14, report attached
  ctest --test-dir build -j8 --repeat until-fail:3
  gcc -Wall -Wextra -c on all nine test files  # no warnings
  python3 scripts/mutation_test.py --target src/stdlib.c  # 89.6%
No temp files left in /tmp after any of the above.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-07-30 02:07:08 -04:00
parent 82c47ed773
commit 437da2960b
14 changed files with 1138 additions and 73 deletions

View File

@@ -41,8 +41,13 @@ ctest --test-dir build --output-on-failure
Tests live one per file in `tests/test_<name>.c` and share the helpers in
`tests/aksl_capture.h``AKSL_CHECK()` for plain assertions (unlike `assert()`
it survives `-DNDEBUG`), `AKSL_CHECK_STATUS(call, expected)` to run a wrapper and
assert on the status it returns, and an `AKSL_RUN()` driver that additionally
fails any test which leaks a slot from libakerror's error pool.
assert on the status it returns, `aksl_temp_file()` for tests that need a real
file to work on, and an `AKSL_RUN()` driver that additionally fails any test which
leaks a slot from libakerror's error pool.
One file per area of the API: `convert` (the `ato*` family), `format` (the
`printf` family), `stream` (`fopen`/`fread`/`fwrite`/`fclose`), `path`
(`aksl_realpath`), `strhash`, `memory`, `linkedlist` and `tree`.
To add a test, drop `tests/test_mything.c` in place and add `mything` to
`AKSL_TESTS` in `CMakeLists.txt`.
@@ -118,7 +123,7 @@ scripts/coverage.py --build build-coverage # report on disk
scripts/coverage.py --build build-coverage --summary-only # totals only
scripts/coverage.py --build build-coverage --include tests # coverage of the tests themselves
scripts/coverage.py --build build-coverage --run-tests # reset, run ctest, report
scripts/coverage.py --build build-coverage --threshold 50 --branch-threshold 25
scripts/coverage.py --build build-coverage --threshold 90 --branch-threshold 45
```
It needs nothing but Python 3 and gcc's own `gcov` — no lcov, gcovr or genhtml.
@@ -129,9 +134,23 @@ score:
```sh
cmake -S . -B build-coverage -DAKSL_COVERAGE=ON \
-DAKSL_COVERAGE_THRESHOLD=50 -DAKSL_COVERAGE_BRANCH_THRESHOLD=20
-DAKSL_COVERAGE_THRESHOLD=90 -DAKSL_COVERAGE_BRANCH_THRESHOLD=45
```
**Where it stands.** `src/stdlib.c` is at **99.0% of lines (200/202)**, **51.0% of
branches** and **21/21 functions**, so 90/45 above is a ratchet with headroom
rather than a target. The two uncovered lines are both
`} HANDLE(e, AKERR_ITERATOR_BREAK) {` — in libakerror that macro begins with the
`break;` belonging to `PROCESS`'s `case 0:` arm, which is only reachable when a
callback returns a non-NULL error context whose status is *zero*. That is the
pathological case §2.2.1 of `TODO.md` exists to remove, so it is left uncovered
deliberately rather than pinned by a test.
Branch coverage sits far below line coverage because most branches in this file
are inside the `FAIL_*`/`ATTEMPT`/`FINISH` macro expansions — pool exhaustion,
stack-trace buffer limits, `akerr_valid_error_address` failures — and belong to
libakerror's own suite rather than to this one.
Two caveats. Coverage is measured at `-O0`, because the optimizer reorders lines
until per-line counts stop matching the source — so a coverage build is not the
build to profile. And gcov flushes its counters at normal process exit, which an
@@ -155,7 +174,7 @@ or drive the script directly for a faster or narrower run:
scripts/mutation_test.py --target src/stdlib.c # C source only
scripts/mutation_test.py --target src/stdlib.c --list # enumerate, build nothing
scripts/mutation_test.py --target src/stdlib.c --max-mutants 20
scripts/mutation_test.py --target src/stdlib.c --threshold 40
scripts/mutation_test.py --target src/stdlib.c --threshold 80
```
A mutant that makes the tests fail is *killed* (good); one the tests still pass
@@ -164,10 +183,23 @@ run prints every survivor with `file:line` and the exact edit. The harness never
touches your working tree — it copies the repo to a scratch directory and mutates
the copy.
CI runs the `src/stdlib.c` set with `--threshold 40`. That is a regression
ratchet rather than a quality bar: the current score is 46.8%, and the survivors
are concentrated in the wrappers that have no tests yet. Raise the threshold as
coverage lands.
CI runs the `src/stdlib.c` set with `--threshold 80`. That is a regression ratchet
rather than a quality bar: the current score is **89.6% (155/173 killed)**, up
from 46.8% before the wrapper tests landed. Raise the threshold as the remaining
survivors are turned into assertions.
The 18 survivors cluster in three places, and each names a real gap rather than a
test-harness artifact:
- **Statements whose absence nothing observes** — deleting `free(ptr)`,
`obj->next = NULL`, or a `SUCCEED_RETURN` leaves behaviour the suite does not
look at (a leak, a stale pointer, a success that was already NULL).
- **The `aksl_list_append` cycle/tail walk** (`tail = slow`, `slow = slow->next`,
`tail = fast`) — the function is broken in exactly this area (`TODO.md` §2.1.1),
so its known-failing test cannot pin the internals yet.
- **`lalloc`/`lfree` defaulting in `aksl_tree_iterate`** — dead parameters
(§2.2.8): they are defaulted and then never called, so inverting the guard
changes nothing observable.
## The pre-push hook
@@ -187,5 +219,5 @@ AKSL_HOOK_MUTATION=1 git push # also run the mutation gate (slow)
git push --no-verify # skip the hook entirely
```
Other knobs: `AKSL_MUTATION_THRESHOLD` (default 40, keep it in step with
Other knobs: `AKSL_MUTATION_THRESHOLD` (default 80, keep it in step with
`.gitea/workflows/ci.yaml`) and `AKSL_HOOK_BUILD_DIR`.