Add code coverage to the CTest suite

New AKSL_COVERAGE option instruments the library and its tests with
--coverage -O0 and wires the report into the suite itself, so a plain
ctest --test-dir build-coverage both runs the tests and produces coverage.

Two CTest entries do the work, held in order by a CTest fixture rather
than by declaration order so they also hold under ctest -j:
coverage_reset (FIXTURES_SETUP) clears the .gcda counters before any test,
since gcov counts are cumulative and would otherwise fold in earlier runs;
coverage_report (FIXTURES_CLEANUP) aggregates gcov output afterwards.
AKSL_COVERAGE_THRESHOLD / AKSL_COVERAGE_BRANCH_THRESHOLD gate the report,
the same regression-ratchet idea as the mutation score. The `coverage`
target builds, runs and prints in one step.

scripts/coverage.py parses gcov's JSON output, aggregates line, branch and
function counts across translation units, and lists every uncovered line
and never-called function -- the actionable half, as with surviving
mutants. Python stdlib plus gcc's own gcov only: no lcov, gcovr or
genhtml. It also writes coverage-summary.txt (CTest hides the output of a
passing test) and a Cobertura coverage.xml for CI publishers.

Instrumentation is per target, so deps/libakerror stays out of the report.
The mutation harness now ignores build*/ and gcov artifacts when copying
the tree, so a coverage build does not slow it down.

Baseline on src/stdlib.c: 52.0% of lines, 23.6% of branches, 8 of 21
functions. The uncovered functions are the untested wrappers the mutation
survivors already point at (printf, ato*, stream, realpath, strhash).

Verified:
  cmake -S . -B build-coverage -DAKSL_COVERAGE=ON
  cmake --build build-coverage --target coverage      # 8/8, report printed
  ctest --test-dir build-coverage -j8                 # fixture order holds
  cmake -S . -B build-coverage -DAKSL_COVERAGE=ON -DAKSL_COVERAGE_THRESHOLD=60
  ctest --test-dir build-coverage --output-on-failure  # gate fails as expected
  ctest --test-dir build --output-on-failure           # 6/6, no .gcda emitted
  ctest --test-dir build-asan --output-on-failure      # 6/6
  scripts/mutation_test.py --target src/stdlib.c --list # 173 mutants, unchanged
Totals match gcov itself: 51.98% of 202 lines, 23.60% of 661 branches.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-07-30 01:48:05 -04:00
parent a87cbfb26d
commit 82c47ed773
5 changed files with 667 additions and 6 deletions

View File

@@ -27,7 +27,7 @@ package the parent provides instead.
## Testing
There are three harnesses. The first two take seconds; the third takes about
There are four harnesses. The first three take seconds; the fourth takes about
half an hour.
### 1. The test suite
@@ -80,7 +80,66 @@ misbehave under instrumentation — the uninitialised `%s` in `aksl_realpath`, t
unbounded `vsprintf` behind `aksl_sprintf`, the missing `va_end` in the `printf`
family — so new tests for those should be run this way.
### 3. Mutation testing
### 3. Code coverage
```sh
cmake -S . -B build-coverage -DAKSL_COVERAGE=ON
cmake --build build-coverage --target coverage
```
`-DAKSL_COVERAGE=ON` compiles the library and the tests with `--coverage -O0`,
and wires the report into the suite itself, so a plain
`ctest --test-dir build-coverage` also produces it. Two extra CTest entries
appear, held in place by a CTest fixture rather than by declaration order, so
they work under `ctest -j` too:
| Test | When | Does |
|---|---|---|
| `coverage_reset` | before every other test | deletes the accumulated `.gcda` counters |
| `coverage_report` | after every other test | aggregates `gcov` output, prints the summary, applies the threshold gate |
The reset matters: gcov counters are cumulative, so without it each report would
fold in every earlier run and overstate coverage.
CTest hides the output of a passing test, so `coverage_report` also writes
`build-coverage/coverage-summary.txt` (the same text report) and
`build-coverage/coverage.xml` (Cobertura, for CI publishers). The `coverage`
target above prints the report to the terminal for you; otherwise read the file
or use `ctest --test-dir build-coverage -V -R coverage_report`.
The report lists per-file line, branch and function coverage, then every
uncovered line and every function the suite never called — that listing is the
actionable part, the same way surviving mutants are for the harness below.
Drive the script directly for anything narrower:
```sh
scripts/coverage.py --build build-coverage # report on disk counters
scripts/coverage.py --build build-coverage --summary-only # totals only
scripts/coverage.py --build build-coverage --include tests # coverage of the tests themselves
scripts/coverage.py --build build-coverage --run-tests # reset, run ctest, report
scripts/coverage.py --build build-coverage --threshold 50 --branch-threshold 25
```
It needs nothing but Python 3 and gcc's own `gcov` — no lcov, gcovr or genhtml.
To gate on coverage, set the threshold at configure time; `coverage_report` then
fails below it, and the same regression-ratchet logic applies as for the mutation
score:
```sh
cmake -S . -B build-coverage -DAKSL_COVERAGE=ON \
-DAKSL_COVERAGE_THRESHOLD=50 -DAKSL_COVERAGE_BRANCH_THRESHOLD=20
```
Two caveats. Coverage is measured at `-O0`, because the optimizer reorders lines
until per-line counts stop matching the source — so a coverage build is not the
build to profile. And gcov flushes its counters at normal process exit, which an
`AKSL_WILL_FAIL_TESTS` entry that aborts by design never reaches: such a test
contributes no coverage data at all, so lines only it reaches are reported as
uncovered.
### 4. Mutation testing
The suite tells you the library works. Mutation testing tells you the *suite*
works: it breaks the library in small ways, one at a time, and checks that the