From 54fb44cd2bbefb393b33ebcff1e0be81270ae489 Mon Sep 17 00:00:00 2001 From: Andrew Kesterson Date: Wed, 29 Jul 2026 11:51:28 -0400 Subject: [PATCH] Document the test harnesses and add a pre-push hook README.md was a build badge and nothing else. It now covers what the library is, how to build it, and how to run each of the three harnesses locally: the ctest suite, the AKSL_SANITIZE ASan/UBSan build, and mutation testing. The section on reading ctest output matters most: two of the three test lists are marked WILL_FAIL, so an all-green run does not mean the library is defect-free -- it means the known-good tests passed and the known-bad ones are still failing in the documented way. .githooks/pre-push runs the two fast harnesses (default build + ctest, then sanitizer build + ctest) before a push leaves the machine. It skips when there is nothing to test (branch deletions, empty pushes), fails the push with the captured output on any error, and builds under .git/aksl-prepush so it never disturbs your own build/. The slow mutation gate is opt-in via AKSL_HOOK_MUTATION=1, and git push --no-verify bypasses everything. Enable with: git config core.hooksPath .githooks Co-Authored-By: Claude Opus 5 (1M context) --- .githooks/pre-push | 100 +++++++++++++++++++++++++++++++++++ README.md | 129 +++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 229 insertions(+) create mode 100755 .githooks/pre-push diff --git a/.githooks/pre-push b/.githooks/pre-push new file mode 100755 index 0000000..c47d23b --- /dev/null +++ b/.githooks/pre-push @@ -0,0 +1,100 @@ +#!/usr/bin/env bash +# +# pre-push: build libakstdlib and run the test harnesses before anything leaves +# this machine. +# +# Install (once per clone): +# +# git config core.hooksPath .githooks +# +# Runs by default, a few seconds all in: +# +# * default build + ctest +# * ASan/UBSan build + ctest +# +# Opt in to the slow harness (~25 minutes -- it rebuilds and re-runs the whole +# suite once per mutant): +# +# AKSL_HOOK_MUTATION=1 git push +# +# Bypass everything (git's own escape hatch): +# +# git push --no-verify +# +# Builds go under .git/aksl-prepush so the hook never disturbs whatever is in +# your own build/ directory. Override with AKSL_HOOK_BUILD_DIR if you want them +# somewhere else. + +set -u + +# Keep in sync with the --threshold in .gitea/workflows/ci.yaml, so a push that +# would fail CI fails here first. +MUTATION_THRESHOLD="${AKSL_MUTATION_THRESHOLD:-40}" + +ZERO_SHA=0000000000000000000000000000000000000000 + +root=$(git rev-parse --show-toplevel) || exit 1 +cd "$root" || exit 1 + +# git feeds us one line per ref being pushed. A push that only *deletes* refs +# has no commits to test, so there is nothing to do. An empty stdin (nothing to +# push) lands here too, which is equally fine to skip. +has_updates=0 +while read -r _local_ref local_sha _remote_ref _remote_sha; do + if [ "$local_sha" != "$ZERO_SHA" ]; then + has_updates=1 + fi +done +if [ "$has_updates" -eq 0 ]; then + exit 0 +fi + +if [ ! -f deps/libakerror/CMakeLists.txt ]; then + echo "pre-push: deps/libakerror is empty. Run:" >&2 + echo " git submodule update --init --recursive" >&2 + exit 1 +fi + +builddir="${AKSL_HOOK_BUILD_DIR:-$(git rev-parse --git-dir)/aksl-prepush}" +mkdir -p "$builddir" || exit 1 +logfile="$builddir/last.log" + +# Run a step quietly; on failure, dump what it said and abort the push. +run() { + if ! "$@" > "$logfile" 2>&1; then + echo >&2 + echo "pre-push: FAILED: $*" >&2 + echo "---------------------------------------------------------------" >&2 + cat "$logfile" >&2 + echo "---------------------------------------------------------------" >&2 + echo "pre-push: push aborted. Use 'git push --no-verify' to override." >&2 + exit 1 + fi +} + +echo "pre-push: default build + ctest" +run cmake -S . -B "$builddir/default" +run cmake --build "$builddir/default" +run ctest --test-dir "$builddir/default" --output-on-failure + +echo "pre-push: sanitizer build + ctest" +run cmake -S . -B "$builddir/asan" -DAKSL_SANITIZE=ON +run cmake --build "$builddir/asan" +run ctest --test-dir "$builddir/asan" --output-on-failure + +if [ "${AKSL_HOOK_MUTATION:-0}" = "1" ]; then + echo "pre-push: mutation testing, threshold ${MUTATION_THRESHOLD}% (this takes a while)" + # Deliberately not wrapped in run(): this one is slow enough that you want + # to watch it make progress. + if ! python3 scripts/mutation_test.py \ + --target src/stdlib.c \ + --threshold "$MUTATION_THRESHOLD"; then + echo >&2 + echo "pre-push: mutation score below ${MUTATION_THRESHOLD}%. Push aborted." >&2 + echo "pre-push: use 'git push --no-verify' to override." >&2 + exit 1 + fi +fi + +echo "pre-push: OK" +exit 0 diff --git a/README.md b/README.md index 24f3a4c..8b1e9b3 100644 --- a/README.md +++ b/README.md @@ -1,3 +1,132 @@ # README ![build badge](https://source.starfort.tech/andrew/libakstdlib/actions/workflows/ci.yaml/badge.svg?branch=main) + +`libakstdlib` wraps C standard library functions so that they report failures +through [libakerror](https://source.starfort.tech/andrew/libakerror)'s +`ATTEMPT { ... } HANDLE { ... }` error contexts instead of through return codes +and `errno`. It also provides a few data structures built on the same +convention (a doubly-linked list and a binary tree). + +Every entry point returns `akerr_ErrorContext *` and is marked `AKERR_NOIGNORE`. +See `TODO.md` for the current state of the library: what is covered by tests, +which corner cases are still open, and which libc functions are not yet wrapped. + +## Building + +```sh +git submodule update --init --recursive # deps/libakerror +cmake -S . -B build +cmake --build build +cmake --install build +``` + +A top-level build compiles the vendored `deps/libakerror`. When `libakstdlib` is +consumed as a subproject, it uses whatever `akerror::akerror` target or installed +package the parent provides instead. + +## Testing + +There are three harnesses. The first two take seconds; the third takes about +half an hour. + +### 1. The test suite + +```sh +cmake -S . -B build +cmake --build build +ctest --test-dir build --output-on-failure +``` + +Tests live one per file in `tests/test_.c` and share the helpers in +`tests/aksl_capture.h` — `AKSL_CHECK()` for plain assertions (unlike `assert()` +it survives `-DNDEBUG`), `AKSL_CHECK_STATUS(call, expected)` to run a wrapper and +assert on the status it returns, and an `AKSL_RUN()` driver that additionally +fails any test which leaks a slot from libakerror's error pool. + +To add a test, drop `tests/test_mything.c` in place and add `mything` to +`AKSL_TESTS` in `CMakeLists.txt`. + +**Reading the results.** `CMakeLists.txt` splits tests into three lists, and two +of them invert the meaning of "Passed": + +| List | Meaning | +|---|---| +| `AKSL_TESTS` | Ordinary tests. Must exit 0. | +| `AKSL_WILL_FAIL_TESTS` | Expected to abort by design — an unhandled error reaching `FINISH_NORETURN`, or a deliberate contract violation. Marked `WILL_FAIL`, so a non-zero exit is a pass. | +| `AKSL_KNOWN_FAILING_TESTS` | Assert the *correct* behaviour of a confirmed defect (see `TODO.md` §2.1). Also marked `WILL_FAIL`. | + +So `ctest` reporting all green does **not** mean the library is defect-free — it +means the known-good tests passed and the known-bad ones are still failing in the +documented way. When a defect is fixed, its test starts passing, CTest reports it +as failed with *unexpectedly passed*, and that is the cue to move it from +`AKSL_KNOWN_FAILING_TESTS` into `AKSL_TESTS`. + +Every test is capped with a 30-second CTest `TIMEOUT`. The list and tree code is +full of loops whose termination hangs on a single condition, so a bug of that +shape hangs the suite rather than failing it. + +### 2. Sanitizers + +```sh +cmake -S . -B build-asan -DAKSL_SANITIZE=ON +cmake --build build-asan +ctest --test-dir build-asan --output-on-failure +``` + +Builds the library, the tests and the vendored libakerror with ASan + UBSan and +`-fno-sanitize-recover=all`. Several of the open items in `TODO.md` §2 only +misbehave under instrumentation — the uninitialised `%s` in `aksl_realpath`, the +unbounded `vsprintf` behind `aksl_sprintf`, the missing `va_end` in the `printf` +family — so new tests for those should be run this way. + +### 3. Mutation testing + +The suite tells you the library works. Mutation testing tells you the *suite* +works: it breaks the library in small ways, one at a time, and checks that the +tests notice. + +```sh +cmake --build build --target mutation # src/stdlib.c + include/akstdlib.h +``` + +or drive the script directly for a faster or narrower run: + +```sh +scripts/mutation_test.py --target src/stdlib.c # C source only +scripts/mutation_test.py --target src/stdlib.c --list # enumerate, build nothing +scripts/mutation_test.py --target src/stdlib.c --max-mutants 20 +scripts/mutation_test.py --target src/stdlib.c --threshold 40 +``` + +A mutant that makes the tests fail is *killed* (good); one the tests still pass +is a *survivor*, and names a missing test. The score is `killed / total`, and the +run prints every survivor with `file:line` and the exact edit. The harness never +touches your working tree — it copies the repo to a scratch directory and mutates +the copy. + +CI runs the `src/stdlib.c` set with `--threshold 40`. That is a regression +ratchet rather than a quality bar: the current score is 46.8%, and the survivors +are concentrated in the wrappers that have no tests yet. Raise the threshold as +coverage lands. + +## The pre-push hook + +`.githooks/pre-push` runs the fast harnesses — the default build and the +sanitizer build, each followed by `ctest` — before letting a push out. Enable it +once per clone: + +```sh +git config core.hooksPath .githooks +``` + +It only builds when there are commits to push (a branch deletion is a no-op), and +it builds under `.git/aksl-prepush` so it never disturbs your own `build/`. + +```sh +AKSL_HOOK_MUTATION=1 git push # also run the mutation gate (slow) +git push --no-verify # skip the hook entirely +``` + +Other knobs: `AKSL_MUTATION_THRESHOLD` (default 40, keep it in step with +`.gitea/workflows/ci.yaml`) and `AKSL_HOOK_BUILD_DIR`.