Document the test harnesses and add a pre-push hook
README.md was a build badge and nothing else. It now covers what the library is, how to build it, and how to run each of the three harnesses locally: the ctest suite, the AKSL_SANITIZE ASan/UBSan build, and mutation testing. The section on reading ctest output matters most: two of the three test lists are marked WILL_FAIL, so an all-green run does not mean the library is defect-free -- it means the known-good tests passed and the known-bad ones are still failing in the documented way. .githooks/pre-push runs the two fast harnesses (default build + ctest, then sanitizer build + ctest) before a push leaves the machine. It skips when there is nothing to test (branch deletions, empty pushes), fails the push with the captured output on any error, and builds under .git/aksl-prepush so it never disturbs your own build/. The slow mutation gate is opt-in via AKSL_HOOK_MUTATION=1, and git push --no-verify bypasses everything. Enable with: git config core.hooksPath .githooks Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
100
.githooks/pre-push
Executable file
100
.githooks/pre-push
Executable file
@@ -0,0 +1,100 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
#
|
||||||
|
# pre-push: build libakstdlib and run the test harnesses before anything leaves
|
||||||
|
# this machine.
|
||||||
|
#
|
||||||
|
# Install (once per clone):
|
||||||
|
#
|
||||||
|
# git config core.hooksPath .githooks
|
||||||
|
#
|
||||||
|
# Runs by default, a few seconds all in:
|
||||||
|
#
|
||||||
|
# * default build + ctest
|
||||||
|
# * ASan/UBSan build + ctest
|
||||||
|
#
|
||||||
|
# Opt in to the slow harness (~25 minutes -- it rebuilds and re-runs the whole
|
||||||
|
# suite once per mutant):
|
||||||
|
#
|
||||||
|
# AKSL_HOOK_MUTATION=1 git push
|
||||||
|
#
|
||||||
|
# Bypass everything (git's own escape hatch):
|
||||||
|
#
|
||||||
|
# git push --no-verify
|
||||||
|
#
|
||||||
|
# Builds go under .git/aksl-prepush so the hook never disturbs whatever is in
|
||||||
|
# your own build/ directory. Override with AKSL_HOOK_BUILD_DIR if you want them
|
||||||
|
# somewhere else.
|
||||||
|
|
||||||
|
set -u
|
||||||
|
|
||||||
|
# Keep in sync with the --threshold in .gitea/workflows/ci.yaml, so a push that
|
||||||
|
# would fail CI fails here first.
|
||||||
|
MUTATION_THRESHOLD="${AKSL_MUTATION_THRESHOLD:-40}"
|
||||||
|
|
||||||
|
ZERO_SHA=0000000000000000000000000000000000000000
|
||||||
|
|
||||||
|
root=$(git rev-parse --show-toplevel) || exit 1
|
||||||
|
cd "$root" || exit 1
|
||||||
|
|
||||||
|
# git feeds us one line per ref being pushed. A push that only *deletes* refs
|
||||||
|
# has no commits to test, so there is nothing to do. An empty stdin (nothing to
|
||||||
|
# push) lands here too, which is equally fine to skip.
|
||||||
|
has_updates=0
|
||||||
|
while read -r _local_ref local_sha _remote_ref _remote_sha; do
|
||||||
|
if [ "$local_sha" != "$ZERO_SHA" ]; then
|
||||||
|
has_updates=1
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
if [ "$has_updates" -eq 0 ]; then
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [ ! -f deps/libakerror/CMakeLists.txt ]; then
|
||||||
|
echo "pre-push: deps/libakerror is empty. Run:" >&2
|
||||||
|
echo " git submodule update --init --recursive" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
builddir="${AKSL_HOOK_BUILD_DIR:-$(git rev-parse --git-dir)/aksl-prepush}"
|
||||||
|
mkdir -p "$builddir" || exit 1
|
||||||
|
logfile="$builddir/last.log"
|
||||||
|
|
||||||
|
# Run a step quietly; on failure, dump what it said and abort the push.
|
||||||
|
run() {
|
||||||
|
if ! "$@" > "$logfile" 2>&1; then
|
||||||
|
echo >&2
|
||||||
|
echo "pre-push: FAILED: $*" >&2
|
||||||
|
echo "---------------------------------------------------------------" >&2
|
||||||
|
cat "$logfile" >&2
|
||||||
|
echo "---------------------------------------------------------------" >&2
|
||||||
|
echo "pre-push: push aborted. Use 'git push --no-verify' to override." >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
echo "pre-push: default build + ctest"
|
||||||
|
run cmake -S . -B "$builddir/default"
|
||||||
|
run cmake --build "$builddir/default"
|
||||||
|
run ctest --test-dir "$builddir/default" --output-on-failure
|
||||||
|
|
||||||
|
echo "pre-push: sanitizer build + ctest"
|
||||||
|
run cmake -S . -B "$builddir/asan" -DAKSL_SANITIZE=ON
|
||||||
|
run cmake --build "$builddir/asan"
|
||||||
|
run ctest --test-dir "$builddir/asan" --output-on-failure
|
||||||
|
|
||||||
|
if [ "${AKSL_HOOK_MUTATION:-0}" = "1" ]; then
|
||||||
|
echo "pre-push: mutation testing, threshold ${MUTATION_THRESHOLD}% (this takes a while)"
|
||||||
|
# Deliberately not wrapped in run(): this one is slow enough that you want
|
||||||
|
# to watch it make progress.
|
||||||
|
if ! python3 scripts/mutation_test.py \
|
||||||
|
--target src/stdlib.c \
|
||||||
|
--threshold "$MUTATION_THRESHOLD"; then
|
||||||
|
echo >&2
|
||||||
|
echo "pre-push: mutation score below ${MUTATION_THRESHOLD}%. Push aborted." >&2
|
||||||
|
echo "pre-push: use 'git push --no-verify' to override." >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "pre-push: OK"
|
||||||
|
exit 0
|
||||||
129
README.md
129
README.md
@@ -1,3 +1,132 @@
|
|||||||
# README
|
# README
|
||||||
|
|
||||||

|

|
||||||
|
|
||||||
|
`libakstdlib` wraps C standard library functions so that they report failures
|
||||||
|
through [libakerror](https://source.starfort.tech/andrew/libakerror)'s
|
||||||
|
`ATTEMPT { ... } HANDLE { ... }` error contexts instead of through return codes
|
||||||
|
and `errno`. It also provides a few data structures built on the same
|
||||||
|
convention (a doubly-linked list and a binary tree).
|
||||||
|
|
||||||
|
Every entry point returns `akerr_ErrorContext *` and is marked `AKERR_NOIGNORE`.
|
||||||
|
See `TODO.md` for the current state of the library: what is covered by tests,
|
||||||
|
which corner cases are still open, and which libc functions are not yet wrapped.
|
||||||
|
|
||||||
|
## Building
|
||||||
|
|
||||||
|
```sh
|
||||||
|
git submodule update --init --recursive # deps/libakerror
|
||||||
|
cmake -S . -B build
|
||||||
|
cmake --build build
|
||||||
|
cmake --install build
|
||||||
|
```
|
||||||
|
|
||||||
|
A top-level build compiles the vendored `deps/libakerror`. When `libakstdlib` is
|
||||||
|
consumed as a subproject, it uses whatever `akerror::akerror` target or installed
|
||||||
|
package the parent provides instead.
|
||||||
|
|
||||||
|
## Testing
|
||||||
|
|
||||||
|
There are three harnesses. The first two take seconds; the third takes about
|
||||||
|
half an hour.
|
||||||
|
|
||||||
|
### 1. The test suite
|
||||||
|
|
||||||
|
```sh
|
||||||
|
cmake -S . -B build
|
||||||
|
cmake --build build
|
||||||
|
ctest --test-dir build --output-on-failure
|
||||||
|
```
|
||||||
|
|
||||||
|
Tests live one per file in `tests/test_<name>.c` and share the helpers in
|
||||||
|
`tests/aksl_capture.h` — `AKSL_CHECK()` for plain assertions (unlike `assert()`
|
||||||
|
it survives `-DNDEBUG`), `AKSL_CHECK_STATUS(call, expected)` to run a wrapper and
|
||||||
|
assert on the status it returns, and an `AKSL_RUN()` driver that additionally
|
||||||
|
fails any test which leaks a slot from libakerror's error pool.
|
||||||
|
|
||||||
|
To add a test, drop `tests/test_mything.c` in place and add `mything` to
|
||||||
|
`AKSL_TESTS` in `CMakeLists.txt`.
|
||||||
|
|
||||||
|
**Reading the results.** `CMakeLists.txt` splits tests into three lists, and two
|
||||||
|
of them invert the meaning of "Passed":
|
||||||
|
|
||||||
|
| List | Meaning |
|
||||||
|
|---|---|
|
||||||
|
| `AKSL_TESTS` | Ordinary tests. Must exit 0. |
|
||||||
|
| `AKSL_WILL_FAIL_TESTS` | Expected to abort by design — an unhandled error reaching `FINISH_NORETURN`, or a deliberate contract violation. Marked `WILL_FAIL`, so a non-zero exit is a pass. |
|
||||||
|
| `AKSL_KNOWN_FAILING_TESTS` | Assert the *correct* behaviour of a confirmed defect (see `TODO.md` §2.1). Also marked `WILL_FAIL`. |
|
||||||
|
|
||||||
|
So `ctest` reporting all green does **not** mean the library is defect-free — it
|
||||||
|
means the known-good tests passed and the known-bad ones are still failing in the
|
||||||
|
documented way. When a defect is fixed, its test starts passing, CTest reports it
|
||||||
|
as failed with *unexpectedly passed*, and that is the cue to move it from
|
||||||
|
`AKSL_KNOWN_FAILING_TESTS` into `AKSL_TESTS`.
|
||||||
|
|
||||||
|
Every test is capped with a 30-second CTest `TIMEOUT`. The list and tree code is
|
||||||
|
full of loops whose termination hangs on a single condition, so a bug of that
|
||||||
|
shape hangs the suite rather than failing it.
|
||||||
|
|
||||||
|
### 2. Sanitizers
|
||||||
|
|
||||||
|
```sh
|
||||||
|
cmake -S . -B build-asan -DAKSL_SANITIZE=ON
|
||||||
|
cmake --build build-asan
|
||||||
|
ctest --test-dir build-asan --output-on-failure
|
||||||
|
```
|
||||||
|
|
||||||
|
Builds the library, the tests and the vendored libakerror with ASan + UBSan and
|
||||||
|
`-fno-sanitize-recover=all`. Several of the open items in `TODO.md` §2 only
|
||||||
|
misbehave under instrumentation — the uninitialised `%s` in `aksl_realpath`, the
|
||||||
|
unbounded `vsprintf` behind `aksl_sprintf`, the missing `va_end` in the `printf`
|
||||||
|
family — so new tests for those should be run this way.
|
||||||
|
|
||||||
|
### 3. Mutation testing
|
||||||
|
|
||||||
|
The suite tells you the library works. Mutation testing tells you the *suite*
|
||||||
|
works: it breaks the library in small ways, one at a time, and checks that the
|
||||||
|
tests notice.
|
||||||
|
|
||||||
|
```sh
|
||||||
|
cmake --build build --target mutation # src/stdlib.c + include/akstdlib.h
|
||||||
|
```
|
||||||
|
|
||||||
|
or drive the script directly for a faster or narrower run:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
scripts/mutation_test.py --target src/stdlib.c # C source only
|
||||||
|
scripts/mutation_test.py --target src/stdlib.c --list # enumerate, build nothing
|
||||||
|
scripts/mutation_test.py --target src/stdlib.c --max-mutants 20
|
||||||
|
scripts/mutation_test.py --target src/stdlib.c --threshold 40
|
||||||
|
```
|
||||||
|
|
||||||
|
A mutant that makes the tests fail is *killed* (good); one the tests still pass
|
||||||
|
is a *survivor*, and names a missing test. The score is `killed / total`, and the
|
||||||
|
run prints every survivor with `file:line` and the exact edit. The harness never
|
||||||
|
touches your working tree — it copies the repo to a scratch directory and mutates
|
||||||
|
the copy.
|
||||||
|
|
||||||
|
CI runs the `src/stdlib.c` set with `--threshold 40`. That is a regression
|
||||||
|
ratchet rather than a quality bar: the current score is 46.8%, and the survivors
|
||||||
|
are concentrated in the wrappers that have no tests yet. Raise the threshold as
|
||||||
|
coverage lands.
|
||||||
|
|
||||||
|
## The pre-push hook
|
||||||
|
|
||||||
|
`.githooks/pre-push` runs the fast harnesses — the default build and the
|
||||||
|
sanitizer build, each followed by `ctest` — before letting a push out. Enable it
|
||||||
|
once per clone:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
git config core.hooksPath .githooks
|
||||||
|
```
|
||||||
|
|
||||||
|
It only builds when there are commits to push (a branch deletion is a no-op), and
|
||||||
|
it builds under `.git/aksl-prepush` so it never disturbs your own `build/`.
|
||||||
|
|
||||||
|
```sh
|
||||||
|
AKSL_HOOK_MUTATION=1 git push # also run the mutation gate (slow)
|
||||||
|
git push --no-verify # skip the hook entirely
|
||||||
|
```
|
||||||
|
|
||||||
|
Other knobs: `AKSL_MUTATION_THRESHOLD` (default 40, keep it in step with
|
||||||
|
`.gitea/workflows/ci.yaml`) and `AKSL_HOOK_BUILD_DIR`.
|
||||||
|
|||||||
Reference in New Issue
Block a user