Add code coverage to the CTest suite
New AKSL_COVERAGE option instruments the library and its tests with --coverage -O0 and wires the report into the suite itself, so a plain ctest --test-dir build-coverage both runs the tests and produces coverage. Two CTest entries do the work, held in order by a CTest fixture rather than by declaration order so they also hold under ctest -j: coverage_reset (FIXTURES_SETUP) clears the .gcda counters before any test, since gcov counts are cumulative and would otherwise fold in earlier runs; coverage_report (FIXTURES_CLEANUP) aggregates gcov output afterwards. AKSL_COVERAGE_THRESHOLD / AKSL_COVERAGE_BRANCH_THRESHOLD gate the report, the same regression-ratchet idea as the mutation score. The `coverage` target builds, runs and prints in one step. scripts/coverage.py parses gcov's JSON output, aggregates line, branch and function counts across translation units, and lists every uncovered line and never-called function -- the actionable half, as with surviving mutants. Python stdlib plus gcc's own gcov only: no lcov, gcovr or genhtml. It also writes coverage-summary.txt (CTest hides the output of a passing test) and a Cobertura coverage.xml for CI publishers. Instrumentation is per target, so deps/libakerror stays out of the report. The mutation harness now ignores build*/ and gcov artifacts when copying the tree, so a coverage build does not slow it down. Baseline on src/stdlib.c: 52.0% of lines, 23.6% of branches, 8 of 21 functions. The uncovered functions are the untested wrappers the mutation survivors already point at (printf, ato*, stream, realpath, strhash). Verified: cmake -S . -B build-coverage -DAKSL_COVERAGE=ON cmake --build build-coverage --target coverage # 8/8, report printed ctest --test-dir build-coverage -j8 # fixture order holds cmake -S . -B build-coverage -DAKSL_COVERAGE=ON -DAKSL_COVERAGE_THRESHOLD=60 ctest --test-dir build-coverage --output-on-failure # gate fails as expected ctest --test-dir build --output-on-failure # 6/6, no .gcda emitted ctest --test-dir build-asan --output-on-failure # 6/6 scripts/mutation_test.py --target src/stdlib.c --list # 173 mutants, unchanged Totals match gcov itself: 51.98% of 202 lines, 23.60% of 661 branches. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
63
README.md
63
README.md
@@ -27,7 +27,7 @@ package the parent provides instead.
|
||||
|
||||
## Testing
|
||||
|
||||
There are three harnesses. The first two take seconds; the third takes about
|
||||
There are four harnesses. The first three take seconds; the fourth takes about
|
||||
half an hour.
|
||||
|
||||
### 1. The test suite
|
||||
@@ -80,7 +80,66 @@ misbehave under instrumentation — the uninitialised `%s` in `aksl_realpath`, t
|
||||
unbounded `vsprintf` behind `aksl_sprintf`, the missing `va_end` in the `printf`
|
||||
family — so new tests for those should be run this way.
|
||||
|
||||
### 3. Mutation testing
|
||||
### 3. Code coverage
|
||||
|
||||
```sh
|
||||
cmake -S . -B build-coverage -DAKSL_COVERAGE=ON
|
||||
cmake --build build-coverage --target coverage
|
||||
```
|
||||
|
||||
`-DAKSL_COVERAGE=ON` compiles the library and the tests with `--coverage -O0`,
|
||||
and wires the report into the suite itself, so a plain
|
||||
`ctest --test-dir build-coverage` also produces it. Two extra CTest entries
|
||||
appear, held in place by a CTest fixture rather than by declaration order, so
|
||||
they work under `ctest -j` too:
|
||||
|
||||
| Test | When | Does |
|
||||
|---|---|---|
|
||||
| `coverage_reset` | before every other test | deletes the accumulated `.gcda` counters |
|
||||
| `coverage_report` | after every other test | aggregates `gcov` output, prints the summary, applies the threshold gate |
|
||||
|
||||
The reset matters: gcov counters are cumulative, so without it each report would
|
||||
fold in every earlier run and overstate coverage.
|
||||
|
||||
CTest hides the output of a passing test, so `coverage_report` also writes
|
||||
`build-coverage/coverage-summary.txt` (the same text report) and
|
||||
`build-coverage/coverage.xml` (Cobertura, for CI publishers). The `coverage`
|
||||
target above prints the report to the terminal for you; otherwise read the file
|
||||
or use `ctest --test-dir build-coverage -V -R coverage_report`.
|
||||
|
||||
The report lists per-file line, branch and function coverage, then every
|
||||
uncovered line and every function the suite never called — that listing is the
|
||||
actionable part, the same way surviving mutants are for the harness below.
|
||||
|
||||
Drive the script directly for anything narrower:
|
||||
|
||||
```sh
|
||||
scripts/coverage.py --build build-coverage # report on disk counters
|
||||
scripts/coverage.py --build build-coverage --summary-only # totals only
|
||||
scripts/coverage.py --build build-coverage --include tests # coverage of the tests themselves
|
||||
scripts/coverage.py --build build-coverage --run-tests # reset, run ctest, report
|
||||
scripts/coverage.py --build build-coverage --threshold 50 --branch-threshold 25
|
||||
```
|
||||
|
||||
It needs nothing but Python 3 and gcc's own `gcov` — no lcov, gcovr or genhtml.
|
||||
|
||||
To gate on coverage, set the threshold at configure time; `coverage_report` then
|
||||
fails below it, and the same regression-ratchet logic applies as for the mutation
|
||||
score:
|
||||
|
||||
```sh
|
||||
cmake -S . -B build-coverage -DAKSL_COVERAGE=ON \
|
||||
-DAKSL_COVERAGE_THRESHOLD=50 -DAKSL_COVERAGE_BRANCH_THRESHOLD=20
|
||||
```
|
||||
|
||||
Two caveats. Coverage is measured at `-O0`, because the optimizer reorders lines
|
||||
until per-line counts stop matching the source — so a coverage build is not the
|
||||
build to profile. And gcov flushes its counters at normal process exit, which an
|
||||
`AKSL_WILL_FAIL_TESTS` entry that aborts by design never reaches: such a test
|
||||
contributes no coverage data at all, so lines only it reaches are reported as
|
||||
uncovered.
|
||||
|
||||
### 4. Mutation testing
|
||||
|
||||
The suite tells you the library works. Mutation testing tells you the *suite*
|
||||
works: it breaks the library in small ways, one at a time, and checks that the
|
||||
|
||||
Reference in New Issue
Block a user