Test the libc wrappers: 52% -> 99% line coverage
Every wrapper outside the list and tree code was untested. Six new test
files close that, following the plan already written in TODO.md 1.2-1.6:
test_stream.c fopen/fread/fwrite/fclose -- happy paths, the round
trip, AKERR_EOF on a short read, AKERR_IO on a stream
opened in the wrong mode, ENOENT, and the NULL guards
test_format.c printf/fprintf/sprintf -- text *and* count asserted
(stdout is pointed at a temp file to check aksl_printf),
all eight NULL guards, EBADF on a read-only stream, and
512 variadic calls in a loop as sanitizer cover for the
missing va_end
test_convert.c ato{i,l,ll,f} happy paths, negatives, leading
whitespace, NULL guards
test_path.c realpath on a file and on a symlink, both compared
against realpath(3) since TMPDIR may itself be a link;
ENOENT, ENOTDIR, NULL path
test_strhash.c djb2 known-answer vectors, len == 0, embedded NUL,
stability, NULL guards
test_convert_strict.c
known-failing (2.1.5): the AKERR_VALUE / ERANGE
contract the ato* family cannot express today
test_tree.c gains the BFS AKERR_NOT_IMPLEMENTED contract, NULL arguments,
and a callback error that is not AKERR_ITERATOR_BREAK propagating out.
Tests deliberately say nothing about behaviour TODO.md records as
defective -- unchecked ptr/mode/resolved_path, short transfers reported as
success, *count left at -1, the djb2 sign extension -- so the eventual fix
does not have to come with a test rewrite. Each failure case in
test_path.c passes a zeroed buffer, because the wrapper's own error path
formats resolved_path with %s (2.1.6).
aksl_capture.h gains aksl_temp_file() with an atexit unlink backstop.
Without it every test that fails before its own unlink leaves temp files
behind -- which is the normal case for a known-failing test, and happens
173 times over in a mutation run.
Coverage on src/stdlib.c: 52.0% -> 99.0% of lines (200/202), 23.6% ->
51.0% of branches, 8/21 -> 21/21 functions. The two uncovered lines are
both `} HANDLE(e, AKERR_ITERATOR_BREAK) {`, where the macro starts with
the `break;` of PROCESS's `case 0:` arm -- reachable only via a non-NULL
error context whose status is zero, the pathology 2.2.1 exists to remove.
Mutation score on src/stdlib.c: 46.8% -> 89.6% (155/173 killed). CI, the
pre-push hook and the docs ratchet from 40 to 80 accordingly, and the 18
survivors are grouped by cause in TODO.md and README.md. A new CI
coverage job gates at 90% lines / 45% branches.
Verified:
ctest --test-dir build # 12/12
ctest --test-dir build-asan # 12/12 under ASan + UBSan
ctest --test-dir build-coverage # 14/14, report attached
ctest --test-dir build -j8 --repeat until-fail:3
gcc -Wall -Wextra -c on all nine test files # no warnings
python3 scripts/mutation_test.py --target src/stdlib.c # 89.6%
No temp files left in /tmp after any of the above.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
52
README.md
52
README.md
@@ -41,8 +41,13 @@ ctest --test-dir build --output-on-failure
|
||||
Tests live one per file in `tests/test_<name>.c` and share the helpers in
|
||||
`tests/aksl_capture.h` — `AKSL_CHECK()` for plain assertions (unlike `assert()`
|
||||
it survives `-DNDEBUG`), `AKSL_CHECK_STATUS(call, expected)` to run a wrapper and
|
||||
assert on the status it returns, and an `AKSL_RUN()` driver that additionally
|
||||
fails any test which leaks a slot from libakerror's error pool.
|
||||
assert on the status it returns, `aksl_temp_file()` for tests that need a real
|
||||
file to work on, and an `AKSL_RUN()` driver that additionally fails any test which
|
||||
leaks a slot from libakerror's error pool.
|
||||
|
||||
One file per area of the API: `convert` (the `ato*` family), `format` (the
|
||||
`printf` family), `stream` (`fopen`/`fread`/`fwrite`/`fclose`), `path`
|
||||
(`aksl_realpath`), `strhash`, `memory`, `linkedlist` and `tree`.
|
||||
|
||||
To add a test, drop `tests/test_mything.c` in place and add `mything` to
|
||||
`AKSL_TESTS` in `CMakeLists.txt`.
|
||||
@@ -118,7 +123,7 @@ scripts/coverage.py --build build-coverage # report on disk
|
||||
scripts/coverage.py --build build-coverage --summary-only # totals only
|
||||
scripts/coverage.py --build build-coverage --include tests # coverage of the tests themselves
|
||||
scripts/coverage.py --build build-coverage --run-tests # reset, run ctest, report
|
||||
scripts/coverage.py --build build-coverage --threshold 50 --branch-threshold 25
|
||||
scripts/coverage.py --build build-coverage --threshold 90 --branch-threshold 45
|
||||
```
|
||||
|
||||
It needs nothing but Python 3 and gcc's own `gcov` — no lcov, gcovr or genhtml.
|
||||
@@ -129,9 +134,23 @@ score:
|
||||
|
||||
```sh
|
||||
cmake -S . -B build-coverage -DAKSL_COVERAGE=ON \
|
||||
-DAKSL_COVERAGE_THRESHOLD=50 -DAKSL_COVERAGE_BRANCH_THRESHOLD=20
|
||||
-DAKSL_COVERAGE_THRESHOLD=90 -DAKSL_COVERAGE_BRANCH_THRESHOLD=45
|
||||
```
|
||||
|
||||
**Where it stands.** `src/stdlib.c` is at **99.0% of lines (200/202)**, **51.0% of
|
||||
branches** and **21/21 functions**, so 90/45 above is a ratchet with headroom
|
||||
rather than a target. The two uncovered lines are both
|
||||
`} HANDLE(e, AKERR_ITERATOR_BREAK) {` — in libakerror that macro begins with the
|
||||
`break;` belonging to `PROCESS`'s `case 0:` arm, which is only reachable when a
|
||||
callback returns a non-NULL error context whose status is *zero*. That is the
|
||||
pathological case §2.2.1 of `TODO.md` exists to remove, so it is left uncovered
|
||||
deliberately rather than pinned by a test.
|
||||
|
||||
Branch coverage sits far below line coverage because most branches in this file
|
||||
are inside the `FAIL_*`/`ATTEMPT`/`FINISH` macro expansions — pool exhaustion,
|
||||
stack-trace buffer limits, `akerr_valid_error_address` failures — and belong to
|
||||
libakerror's own suite rather than to this one.
|
||||
|
||||
Two caveats. Coverage is measured at `-O0`, because the optimizer reorders lines
|
||||
until per-line counts stop matching the source — so a coverage build is not the
|
||||
build to profile. And gcov flushes its counters at normal process exit, which an
|
||||
@@ -155,7 +174,7 @@ or drive the script directly for a faster or narrower run:
|
||||
scripts/mutation_test.py --target src/stdlib.c # C source only
|
||||
scripts/mutation_test.py --target src/stdlib.c --list # enumerate, build nothing
|
||||
scripts/mutation_test.py --target src/stdlib.c --max-mutants 20
|
||||
scripts/mutation_test.py --target src/stdlib.c --threshold 40
|
||||
scripts/mutation_test.py --target src/stdlib.c --threshold 80
|
||||
```
|
||||
|
||||
A mutant that makes the tests fail is *killed* (good); one the tests still pass
|
||||
@@ -164,10 +183,23 @@ run prints every survivor with `file:line` and the exact edit. The harness never
|
||||
touches your working tree — it copies the repo to a scratch directory and mutates
|
||||
the copy.
|
||||
|
||||
CI runs the `src/stdlib.c` set with `--threshold 40`. That is a regression
|
||||
ratchet rather than a quality bar: the current score is 46.8%, and the survivors
|
||||
are concentrated in the wrappers that have no tests yet. Raise the threshold as
|
||||
coverage lands.
|
||||
CI runs the `src/stdlib.c` set with `--threshold 80`. That is a regression ratchet
|
||||
rather than a quality bar: the current score is **89.6% (155/173 killed)**, up
|
||||
from 46.8% before the wrapper tests landed. Raise the threshold as the remaining
|
||||
survivors are turned into assertions.
|
||||
|
||||
The 18 survivors cluster in three places, and each names a real gap rather than a
|
||||
test-harness artifact:
|
||||
|
||||
- **Statements whose absence nothing observes** — deleting `free(ptr)`,
|
||||
`obj->next = NULL`, or a `SUCCEED_RETURN` leaves behaviour the suite does not
|
||||
look at (a leak, a stale pointer, a success that was already NULL).
|
||||
- **The `aksl_list_append` cycle/tail walk** (`tail = slow`, `slow = slow->next`,
|
||||
`tail = fast`) — the function is broken in exactly this area (`TODO.md` §2.1.1),
|
||||
so its known-failing test cannot pin the internals yet.
|
||||
- **`lalloc`/`lfree` defaulting in `aksl_tree_iterate`** — dead parameters
|
||||
(§2.2.8): they are defaulted and then never called, so inverting the guard
|
||||
changes nothing observable.
|
||||
|
||||
## The pre-push hook
|
||||
|
||||
@@ -187,5 +219,5 @@ AKSL_HOOK_MUTATION=1 git push # also run the mutation gate (slow)
|
||||
git push --no-verify # skip the hook entirely
|
||||
```
|
||||
|
||||
Other knobs: `AKSL_MUTATION_THRESHOLD` (default 40, keep it in step with
|
||||
Other knobs: `AKSL_MUTATION_THRESHOLD` (default 80, keep it in step with
|
||||
`.gitea/workflows/ci.yaml`) and `AKSL_HOOK_BUILD_DIR`.
|
||||
|
||||
Reference in New Issue
Block a user