Andrew Kesterson
437da2960b
Every wrapper outside the list and tree code was untested. Six new test
files close that, following the plan already written in TODO.md 1.2-1.6:
test_stream.c fopen/fread/fwrite/fclose -- happy paths, the round
trip, AKERR_EOF on a short read, AKERR_IO on a stream
opened in the wrong mode, ENOENT, and the NULL guards
test_format.c printf/fprintf/sprintf -- text *and* count asserted
(stdout is pointed at a temp file to check aksl_printf),
all eight NULL guards, EBADF on a read-only stream, and
512 variadic calls in a loop as sanitizer cover for the
missing va_end
test_convert.c ato{i,l,ll,f} happy paths, negatives, leading
whitespace, NULL guards
test_path.c realpath on a file and on a symlink, both compared
against realpath(3) since TMPDIR may itself be a link;
ENOENT, ENOTDIR, NULL path
test_strhash.c djb2 known-answer vectors, len == 0, embedded NUL,
stability, NULL guards
test_convert_strict.c
known-failing (2.1.5): the AKERR_VALUE / ERANGE
contract the ato* family cannot express today
test_tree.c gains the BFS AKERR_NOT_IMPLEMENTED contract, NULL arguments,
and a callback error that is not AKERR_ITERATOR_BREAK propagating out.
Tests deliberately say nothing about behaviour TODO.md records as
defective -- unchecked ptr/mode/resolved_path, short transfers reported as
success, *count left at -1, the djb2 sign extension -- so the eventual fix
does not have to come with a test rewrite. Each failure case in
test_path.c passes a zeroed buffer, because the wrapper's own error path
formats resolved_path with %s (2.1.6).
aksl_capture.h gains aksl_temp_file() with an atexit unlink backstop.
Without it every test that fails before its own unlink leaves temp files
behind -- which is the normal case for a known-failing test, and happens
173 times over in a mutation run.
Coverage on src/stdlib.c: 52.0% -> 99.0% of lines (200/202), 23.6% ->
51.0% of branches, 8/21 -> 21/21 functions. The two uncovered lines are
both `} HANDLE(e, AKERR_ITERATOR_BREAK) {`, where the macro starts with
the `break;` of PROCESS's `case 0:` arm -- reachable only via a non-NULL
error context whose status is zero, the pathology 2.2.1 exists to remove.
Mutation score on src/stdlib.c: 46.8% -> 89.6% (155/173 killed). CI, the
pre-push hook and the docs ratchet from 40 to 80 accordingly, and the 18
survivors are grouped by cause in TODO.md and README.md. A new CI
coverage job gates at 90% lines / 45% branches.
Verified:
ctest --test-dir build # 12/12
ctest --test-dir build-asan # 12/12 under ASan + UBSan
ctest --test-dir build-coverage # 14/14, report attached
ctest --test-dir build -j8 --repeat until-fail:3
gcc -Wall -Wextra -c on all nine test files # no warnings
python3 scripts/mutation_test.py --target src/stdlib.c # 89.6%
No temp files left in /tmp after any of the above.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
224 lines
9.7 KiB
Markdown
224 lines
9.7 KiB
Markdown
# README
|
|
|
|
|
|
|
|
`libakstdlib` wraps C standard library functions so that they report failures
|
|
through [libakerror](https://source.starfort.tech/andrew/libakerror)'s
|
|
`ATTEMPT { ... } HANDLE { ... }` error contexts instead of through return codes
|
|
and `errno`. It also provides a few data structures built on the same
|
|
convention (a doubly-linked list and a binary tree).
|
|
|
|
Every entry point returns `akerr_ErrorContext *` and is marked `AKERR_NOIGNORE`.
|
|
See `TODO.md` for the current state of the library: what is covered by tests,
|
|
which corner cases are still open, and which libc functions are not yet wrapped.
|
|
|
|
## Building
|
|
|
|
```sh
|
|
git submodule update --init --recursive # deps/libakerror
|
|
cmake -S . -B build
|
|
cmake --build build
|
|
cmake --install build
|
|
```
|
|
|
|
A top-level build compiles the vendored `deps/libakerror`. When `libakstdlib` is
|
|
consumed as a subproject, it uses whatever `akerror::akerror` target or installed
|
|
package the parent provides instead.
|
|
|
|
## Testing
|
|
|
|
There are four harnesses. The first three take seconds; the fourth takes about
|
|
half an hour.
|
|
|
|
### 1. The test suite
|
|
|
|
```sh
|
|
cmake -S . -B build
|
|
cmake --build build
|
|
ctest --test-dir build --output-on-failure
|
|
```
|
|
|
|
Tests live one per file in `tests/test_<name>.c` and share the helpers in
|
|
`tests/aksl_capture.h` — `AKSL_CHECK()` for plain assertions (unlike `assert()`
|
|
it survives `-DNDEBUG`), `AKSL_CHECK_STATUS(call, expected)` to run a wrapper and
|
|
assert on the status it returns, `aksl_temp_file()` for tests that need a real
|
|
file to work on, and an `AKSL_RUN()` driver that additionally fails any test which
|
|
leaks a slot from libakerror's error pool.
|
|
|
|
One file per area of the API: `convert` (the `ato*` family), `format` (the
|
|
`printf` family), `stream` (`fopen`/`fread`/`fwrite`/`fclose`), `path`
|
|
(`aksl_realpath`), `strhash`, `memory`, `linkedlist` and `tree`.
|
|
|
|
To add a test, drop `tests/test_mything.c` in place and add `mything` to
|
|
`AKSL_TESTS` in `CMakeLists.txt`.
|
|
|
|
**Reading the results.** `CMakeLists.txt` splits tests into three lists, and two
|
|
of them invert the meaning of "Passed":
|
|
|
|
| List | Meaning |
|
|
|---|---|
|
|
| `AKSL_TESTS` | Ordinary tests. Must exit 0. |
|
|
| `AKSL_WILL_FAIL_TESTS` | Expected to abort by design — an unhandled error reaching `FINISH_NORETURN`, or a deliberate contract violation. Marked `WILL_FAIL`, so a non-zero exit is a pass. |
|
|
| `AKSL_KNOWN_FAILING_TESTS` | Assert the *correct* behaviour of a confirmed defect (see `TODO.md` §2.1). Also marked `WILL_FAIL`. |
|
|
|
|
So `ctest` reporting all green does **not** mean the library is defect-free — it
|
|
means the known-good tests passed and the known-bad ones are still failing in the
|
|
documented way. When a defect is fixed, its test starts passing, CTest reports it
|
|
as failed with *unexpectedly passed*, and that is the cue to move it from
|
|
`AKSL_KNOWN_FAILING_TESTS` into `AKSL_TESTS`.
|
|
|
|
Every test is capped with a 30-second CTest `TIMEOUT`. The list and tree code is
|
|
full of loops whose termination hangs on a single condition, so a bug of that
|
|
shape hangs the suite rather than failing it.
|
|
|
|
### 2. Sanitizers
|
|
|
|
```sh
|
|
cmake -S . -B build-asan -DAKSL_SANITIZE=ON
|
|
cmake --build build-asan
|
|
ctest --test-dir build-asan --output-on-failure
|
|
```
|
|
|
|
Builds the library, the tests and the vendored libakerror with ASan + UBSan and
|
|
`-fno-sanitize-recover=all`. Several of the open items in `TODO.md` §2 only
|
|
misbehave under instrumentation — the uninitialised `%s` in `aksl_realpath`, the
|
|
unbounded `vsprintf` behind `aksl_sprintf`, the missing `va_end` in the `printf`
|
|
family — so new tests for those should be run this way.
|
|
|
|
### 3. Code coverage
|
|
|
|
```sh
|
|
cmake -S . -B build-coverage -DAKSL_COVERAGE=ON
|
|
cmake --build build-coverage --target coverage
|
|
```
|
|
|
|
`-DAKSL_COVERAGE=ON` compiles the library and the tests with `--coverage -O0`,
|
|
and wires the report into the suite itself, so a plain
|
|
`ctest --test-dir build-coverage` also produces it. Two extra CTest entries
|
|
appear, held in place by a CTest fixture rather than by declaration order, so
|
|
they work under `ctest -j` too:
|
|
|
|
| Test | When | Does |
|
|
|---|---|---|
|
|
| `coverage_reset` | before every other test | deletes the accumulated `.gcda` counters |
|
|
| `coverage_report` | after every other test | aggregates `gcov` output, prints the summary, applies the threshold gate |
|
|
|
|
The reset matters: gcov counters are cumulative, so without it each report would
|
|
fold in every earlier run and overstate coverage.
|
|
|
|
CTest hides the output of a passing test, so `coverage_report` also writes
|
|
`build-coverage/coverage-summary.txt` (the same text report) and
|
|
`build-coverage/coverage.xml` (Cobertura, for CI publishers). The `coverage`
|
|
target above prints the report to the terminal for you; otherwise read the file
|
|
or use `ctest --test-dir build-coverage -V -R coverage_report`.
|
|
|
|
The report lists per-file line, branch and function coverage, then every
|
|
uncovered line and every function the suite never called — that listing is the
|
|
actionable part, the same way surviving mutants are for the harness below.
|
|
|
|
Drive the script directly for anything narrower:
|
|
|
|
```sh
|
|
scripts/coverage.py --build build-coverage # report on disk counters
|
|
scripts/coverage.py --build build-coverage --summary-only # totals only
|
|
scripts/coverage.py --build build-coverage --include tests # coverage of the tests themselves
|
|
scripts/coverage.py --build build-coverage --run-tests # reset, run ctest, report
|
|
scripts/coverage.py --build build-coverage --threshold 90 --branch-threshold 45
|
|
```
|
|
|
|
It needs nothing but Python 3 and gcc's own `gcov` — no lcov, gcovr or genhtml.
|
|
|
|
To gate on coverage, set the threshold at configure time; `coverage_report` then
|
|
fails below it, and the same regression-ratchet logic applies as for the mutation
|
|
score:
|
|
|
|
```sh
|
|
cmake -S . -B build-coverage -DAKSL_COVERAGE=ON \
|
|
-DAKSL_COVERAGE_THRESHOLD=90 -DAKSL_COVERAGE_BRANCH_THRESHOLD=45
|
|
```
|
|
|
|
**Where it stands.** `src/stdlib.c` is at **99.0% of lines (200/202)**, **51.0% of
|
|
branches** and **21/21 functions**, so 90/45 above is a ratchet with headroom
|
|
rather than a target. The two uncovered lines are both
|
|
`} HANDLE(e, AKERR_ITERATOR_BREAK) {` — in libakerror that macro begins with the
|
|
`break;` belonging to `PROCESS`'s `case 0:` arm, which is only reachable when a
|
|
callback returns a non-NULL error context whose status is *zero*. That is the
|
|
pathological case §2.2.1 of `TODO.md` exists to remove, so it is left uncovered
|
|
deliberately rather than pinned by a test.
|
|
|
|
Branch coverage sits far below line coverage because most branches in this file
|
|
are inside the `FAIL_*`/`ATTEMPT`/`FINISH` macro expansions — pool exhaustion,
|
|
stack-trace buffer limits, `akerr_valid_error_address` failures — and belong to
|
|
libakerror's own suite rather than to this one.
|
|
|
|
Two caveats. Coverage is measured at `-O0`, because the optimizer reorders lines
|
|
until per-line counts stop matching the source — so a coverage build is not the
|
|
build to profile. And gcov flushes its counters at normal process exit, which an
|
|
`AKSL_WILL_FAIL_TESTS` entry that aborts by design never reaches: such a test
|
|
contributes no coverage data at all, so lines only it reaches are reported as
|
|
uncovered.
|
|
|
|
### 4. Mutation testing
|
|
|
|
The suite tells you the library works. Mutation testing tells you the *suite*
|
|
works: it breaks the library in small ways, one at a time, and checks that the
|
|
tests notice.
|
|
|
|
```sh
|
|
cmake --build build --target mutation # src/stdlib.c + include/akstdlib.h
|
|
```
|
|
|
|
or drive the script directly for a faster or narrower run:
|
|
|
|
```sh
|
|
scripts/mutation_test.py --target src/stdlib.c # C source only
|
|
scripts/mutation_test.py --target src/stdlib.c --list # enumerate, build nothing
|
|
scripts/mutation_test.py --target src/stdlib.c --max-mutants 20
|
|
scripts/mutation_test.py --target src/stdlib.c --threshold 80
|
|
```
|
|
|
|
A mutant that makes the tests fail is *killed* (good); one the tests still pass
|
|
is a *survivor*, and names a missing test. The score is `killed / total`, and the
|
|
run prints every survivor with `file:line` and the exact edit. The harness never
|
|
touches your working tree — it copies the repo to a scratch directory and mutates
|
|
the copy.
|
|
|
|
CI runs the `src/stdlib.c` set with `--threshold 80`. That is a regression ratchet
|
|
rather than a quality bar: the current score is **89.6% (155/173 killed)**, up
|
|
from 46.8% before the wrapper tests landed. Raise the threshold as the remaining
|
|
survivors are turned into assertions.
|
|
|
|
The 18 survivors cluster in three places, and each names a real gap rather than a
|
|
test-harness artifact:
|
|
|
|
- **Statements whose absence nothing observes** — deleting `free(ptr)`,
|
|
`obj->next = NULL`, or a `SUCCEED_RETURN` leaves behaviour the suite does not
|
|
look at (a leak, a stale pointer, a success that was already NULL).
|
|
- **The `aksl_list_append` cycle/tail walk** (`tail = slow`, `slow = slow->next`,
|
|
`tail = fast`) — the function is broken in exactly this area (`TODO.md` §2.1.1),
|
|
so its known-failing test cannot pin the internals yet.
|
|
- **`lalloc`/`lfree` defaulting in `aksl_tree_iterate`** — dead parameters
|
|
(§2.2.8): they are defaulted and then never called, so inverting the guard
|
|
changes nothing observable.
|
|
|
|
## The pre-push hook
|
|
|
|
`.githooks/pre-push` runs the fast harnesses — the default build and the
|
|
sanitizer build, each followed by `ctest` — before letting a push out. Enable it
|
|
once per clone:
|
|
|
|
```sh
|
|
git config core.hooksPath .githooks
|
|
```
|
|
|
|
It only builds when there are commits to push (a branch deletion is a no-op), and
|
|
it builds under `.git/aksl-prepush` so it never disturbs your own `build/`.
|
|
|
|
```sh
|
|
AKSL_HOOK_MUTATION=1 git push # also run the mutation gate (slow)
|
|
git push --no-verify # skip the hook entirely
|
|
```
|
|
|
|
Other knobs: `AKSL_MUTATION_THRESHOLD` (default 80, keep it in step with
|
|
`.gitea/workflows/ci.yaml`) and `AKSL_HOOK_BUILD_DIR`.
|