andrew/libakstdlib
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Test the libc wrappers: 52% -> 99% line coverage

Browse Source 
Every wrapper outside the list and tree code was untested. Six new test
files close that, following the plan already written in TODO.md 1.2-1.6:

  test_stream.c   fopen/fread/fwrite/fclose -- happy paths, the round
                  trip, AKERR_EOF on a short read, AKERR_IO on a stream
                  opened in the wrong mode, ENOENT, and the NULL guards
  test_format.c   printf/fprintf/sprintf -- text *and* count asserted
                  (stdout is pointed at a temp file to check aksl_printf),
                  all eight NULL guards, EBADF on a read-only stream, and
                  512 variadic calls in a loop as sanitizer cover for the
                  missing va_end
  test_convert.c  ato{i,l,ll,f} happy paths, negatives, leading
                  whitespace, NULL guards
  test_path.c     realpath on a file and on a symlink, both compared
                  against realpath(3) since TMPDIR may itself be a link;
                  ENOENT, ENOTDIR, NULL path
  test_strhash.c  djb2 known-answer vectors, len == 0, embedded NUL,
                  stability, NULL guards
  test_convert_strict.c
                  known-failing (2.1.5): the AKERR_VALUE / ERANGE
                  contract the ato* family cannot express today

test_tree.c gains the BFS AKERR_NOT_IMPLEMENTED contract, NULL arguments,
and a callback error that is not AKERR_ITERATOR_BREAK propagating out.

Tests deliberately say nothing about behaviour TODO.md records as
defective -- unchecked ptr/mode/resolved_path, short transfers reported as
success, *count left at -1, the djb2 sign extension -- so the eventual fix
does not have to come with a test rewrite. Each failure case in
test_path.c passes a zeroed buffer, because the wrapper's own error path
formats resolved_path with %s (2.1.6).

aksl_capture.h gains aksl_temp_file() with an atexit unlink backstop.
Without it every test that fails before its own unlink leaves temp files
behind -- which is the normal case for a known-failing test, and happens
173 times over in a mutation run.

Coverage on src/stdlib.c: 52.0% -> 99.0% of lines (200/202), 23.6% ->
51.0% of branches, 8/21 -> 21/21 functions. The two uncovered lines are
both `} HANDLE(e, AKERR_ITERATOR_BREAK) {`, where the macro starts with
the `break;` of PROCESS's `case 0:` arm -- reachable only via a non-NULL
error context whose status is zero, the pathology 2.2.1 exists to remove.

Mutation score on src/stdlib.c: 46.8% -> 89.6% (155/173 killed). CI, the
pre-push hook and the docs ratchet from 40 to 80 accordingly, and the 18
survivors are grouped by cause in TODO.md and README.md. A new CI
coverage job gates at 90% lines / 45% branches.

Verified:
  ctest --test-dir build            # 12/12
  ctest --test-dir build-asan       # 12/12 under ASan + UBSan
  ctest --test-dir build-coverage   # 14/14, report attached
  ctest --test-dir build -j8 --repeat until-fail:3
  gcc -Wall -Wextra -c on all nine test files  # no warnings
  python3 scripts/mutation_test.py --target src/stdlib.c  # 89.6%
No temp files left in /tmp after any of the above.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Andrew Kesterson
2026-07-30 02:07:08 -04:00
parent 82c47ed773
commit 437da2960b
14 changed files with 1138 additions and 73 deletions
Download Patch File Download Diff File Expand all files Collapse all files

130
TODO.md
View File

@@ -57,21 +57,20 @@ against `tests/aksl_capture.h` and added to the `AKSL_TESTS` list.
- [x] **Port `scripts/mutation_test.py` from libakerror.** Retargeted at `src/stdlib.c` and
`include/akstdlib.h` (178 mutants) and exposed as
`cmake --build build --target mutation`. CI runs the narrower `src/stdlib.c` set (173
mutants) as its own `mutation_test` job with `--threshold 40` and a JUnit report.
**Current score: 46.8%** — 81 killed, 92 survived. The survivors are concentrated
exactly where §1.1–1.9 have yet to be written:
mutants) as its own `mutation_test` job with `--threshold 80` and a JUnit report.
**Current score: 89.6%** — 155 killed, 18 survived. It was 46.8% (81 killed, 92
survived) before §1.2–1.6 were written; the remaining survivors are:
| surviving mutants | function |
|---|---|
| 10 | `aksl_tree_iterate` |
| 6 each | `aksl_fread`, `aksl_fwrite`, `aksl_fprintf`, `aksl_sprintf` |
| 5 | `aksl_printf` |
| 4 each | `aksl_memcpy`, `aksl_realpath`, `aksl_strhash_djb2`, `aksl_list_append` |
| 3 each | `aksl_malloc`, `aksl_free`, `aksl_memset`, `aksl_fopen`, `aksl_fclose`, `aksl_atoi`, `aksl_atol`, `aksl_atoll`, `aksl_atof` |
| 1 | `aksl_list_iterate` |
| surviving mutants | where | why |
|---|---|---|
| 5 | deleted statements whose absence nothing observes (`free(ptr)`, `obj->next = NULL`, three `SUCCEED_RETURN`s) | needs an assertion on the side effect, not on the status |
| 4 | `aksl_list_append`'s cycle/tail walk | the function is broken here (§2.1.1); its known-failing test cannot pin the internals yet |
| 4 | `lalloc`/`lfree` defaulting in `aksl_tree_iterate` | dead parameters (§2.2.8) — defaulted, then never called |
| 3 | `*count == -1` in the `printf` family | `*count` on the error path is itself a contract gap (§1.3) |
| 2 | `feof` in `aksl_fwrite`, `break` after the BFS `FAIL_RETURN` | unreachable in practice |
The threshold is a regression ratchet, not a quality bar; raise it as sections below
land. Each surviving mutant is a concrete missing test — `mutation-junit.xml` lists
The threshold is a regression ratchet, not a quality bar; raise it as those survivors
become assertions. Each one is a concrete missing test — `mutation-junit.xml` lists
them with `file:line` and the exact edit.
- [x] **Cap every test with a CTest `TIMEOUT`.** Found while measuring the above: a mutant
that deletes `fast = fast->next->next` turns `aksl_list_iterate` into an infinite
@@ -108,75 +107,93 @@ against `tests/aksl_capture.h` and added to the `AKSL_TESTS` list.
### 1.2 File I/O
- [ ] `aksl_fopen` happy path on a temp file; `*fp` is written.
- [ ] `aksl_fopen("/nonexistent/path", "r", &fp)` → `ENOENT` propagated as the status, with
Covered by `tests/test_stream.c`.
- [x] `aksl_fopen` happy path on a temp file; `*fp` is written.
- [x] `aksl_fopen("/nonexistent/path", "r", &fp)` → `ENOENT` propagated as the status, with
the pathname in the message.
- [ ] `aksl_fopen` on a mode-denied path (e.g. `/proc/1/mem`, or a `chmod 000` temp file) →
`EACCES`.
- [ ] `aksl_fopen(path, mode, NULL)` → `AKERR_NULLPOINTER`.
- [x] `aksl_fopen(path, mode, NULL)` → `AKERR_NULLPOINTER`.
- [ ] `aksl_fopen(NULL, "r", &fp)` and `aksl_fopen(path, NULL, &fp)` → currently unchecked,
see §2.2.2. Test once the guards exist.
- [ ] `aksl_fread` full read; short read at EOF → `AKERR_EOF`; read from a write-only stream
→ `AKERR_IO`; `fp == NULL` → `AKERR_NULLPOINTER`; `ptr == NULL` → should be
`AKERR_NULLPOINTER` (see §2.2.3).
- [x] `aksl_fread` full read; short read at EOF → `AKERR_EOF`; read from a write-only stream
→ `AKERR_IO`; `fp == NULL` → `AKERR_NULLPOINTER`. `ptr == NULL` is still unchecked and
untested (see §2.2.3).
- [ ] `aksl_fread` **partial read that is neither EOF nor error** — assert whatever the fixed
contract is; today this silently returns success and the caller cannot tell how many
members were read.
- [ ] `aksl_fwrite` happy path; write to a read-only stream → `AKERR_IO`; write to a full
device (`/dev/full`) → `ENOSPC`/`AKERR_IO`; `fp == NULL` → `AKERR_NULLPOINTER`.
- [ ] `aksl_fclose` happy path; `NULL` → `AKERR_NULLPOINTER`; double-close is caught or
documented as UB.
- [x] `aksl_fwrite` happy path; write to a read-only stream → `AKERR_IO`; `fp == NULL` →
`AKERR_NULLPOINTER`. The full-device (`/dev/full`) case is still open.
- [x] `aksl_fclose` happy path; `NULL` → `AKERR_NULLPOINTER`. Double-close is still
undecided, so it is neither caught nor tested.
- [ ] `aksl_fclose` on a stream whose buffered flush fails (`/dev/full`) → non-zero `fclose`
surfaces `errno`.
- [ ] Round-trip test: `fopen` → `fwrite` → `fclose` → `fopen` → `fread` → compare bytes.
- [x] Round-trip test: `fopen` → `fwrite` → `fclose` → `fopen` → `fread` → compare bytes.
### 1.3 Formatted output
- [ ] `aksl_printf` / `aksl_fprintf` / `aksl_sprintf` happy paths, asserting both the byte
count written through `count` and the produced text.
- [ ] Each of the three with every pointer argument NULL in turn → `AKERR_NULLPOINTER`.
- [ ] `aksl_fprintf` to a closed / read-only stream → error path, and confirm `*count` is not
left holding `-1` as if it were a valid length.
Covered by `tests/test_format.c`.
- [x] `aksl_printf` / `aksl_fprintf` / `aksl_sprintf` happy paths, asserting both the byte
count written through `count` and the produced text. (`aksl_printf` is checked by
pointing `stdout` at a temp file for the duration of the call.)
- [x] Each of the three with every pointer argument NULL in turn → `AKERR_NULLPOINTER`.
- [x] `aksl_fprintf` to a read-only stream → the `errno` it saw (`EBADF`) with "Short write"
in the message. `*count` is still left holding `-1`; the test asserts the status only,
so it will not have to change when that is fixed.
- [ ] `aksl_sprintf` with a format that overflows the destination — currently unbounded
(§2.2.4). Test the `aksl_snprintf` replacement once it exists.
- [ ] Regression test for the missing `va_end` (§2.1.4) — a test that calls each variadic
wrapper many times in a loop, run under valgrind/ASan.
- [x] Regression test for the missing `va_end` (§2.1.4) — 512 variadic calls in a loop, which
the sanitizer build also runs.
- [ ] Format-string/arg mismatch is caught at compile time once
`__attribute__((format(printf, ...)))` is added (§2.2.5) — a negative compile test.
### 1.4 String → number
- [ ] `aksl_atoi` / `atol` / `atoll` / `atof` happy paths, including negative values and
Happy paths and NULL guards are covered by `tests/test_convert.c`; the strict-conversion
contract is asserted by `tests/test_convert_strict.c`, which is registered as a known
failure until §2.1.5 is fixed.
- [x] `aksl_atoi` / `atol` / `atoll` / `atof` happy paths, including negative values and
leading whitespace.
- [ ] NULL `nptr` and NULL `dest` for each → `AKERR_NULLPOINTER`.
- [ ] **Non-numeric input** (`"not a number"`) — **[CONFIRMED]** currently returns *success*
with `*dest == 0`. Test the `AKERR_VALUE` behaviour once §2.1.5 is fixed.
- [ ] **Overflow** (`"99999999999999999999"`) — **[CONFIRMED]** currently returns success with
a garbage value (`-1` on this box). Test for `ERANGE`.
- [ ] Empty string, `" "`, `"12abc"` (trailing junk), `"0x10"`, `"inf"`/`"nan"` for `atof`.
- [x] NULL `nptr` and NULL `dest` for each → `AKERR_NULLPOINTER`.
- [x] **Non-numeric input** (`"not a number"`) — **[CONFIRMED]** currently returns *success*
with `*dest == 0`. `AKERR_VALUE` is asserted by the known-failing test.
- [x] **Overflow** (`"99999999999999999999"`) — **[CONFIRMED]** currently returns success with
a garbage value (`-1` on this box). `ERANGE` is asserted by the known-failing test.
- [x] Empty string, `" "` and `"12abc"` (trailing junk) → `AKERR_VALUE`, in the known-failing
test. `"0x10"` and `"inf"`/`"nan"` for `atof` are still open.
- [ ] `LONG_MIN`/`LONG_MAX`/`LLONG_MIN`/`LLONG_MAX` boundary strings round-trip exactly.
### 1.5 `aksl_realpath`
- [ ] Happy path on an existing file and on a symlink chain; result matches `realpath(3)`.
- [ ] Non-existent path → `ENOENT`.
- [ ] A path component that is not a directory → `ENOTDIR`.
Covered by `tests/test_path.c`. Every failure case there passes a zeroed `resolved_path`,
because the wrapper's own error path formats that buffer with `%s` (§2.1.6).
- [x] Happy path on an existing file and on a symlink chain; result matches `realpath(3)`.
- [x] Non-existent path → `ENOENT`.
- [x] A path component that is not a directory → `ENOTDIR`.
- [ ] Symlink loop → `ELOOP`.
- [ ] `path == NULL` → `AKERR_NULLPOINTER`.
- [x] `path == NULL` → `AKERR_NULLPOINTER`.
- [ ] `resolved_path == NULL` — currently unchecked and leaks (§2.1.6). Test once fixed.
- [ ] A failure case where `resolved_path` is an *uninitialised* buffer — this is the crash
case in §2.1.6; run it under ASan/MSan.
### 1.6 `aksl_strhash_djb2`
- [ ] Known-answer vectors: `djb2("")` == 5381; a handful of fixed strings with their
Covered by `tests/test_strhash.c`.
- [x] Known-answer vectors: `djb2("")` == 5381; a handful of fixed strings with their
pre-computed 32-bit values.
- [ ] `len == 0` returns 5381 regardless of `str` contents.
- [ ] NULL `str` and NULL `hashval` → `AKERR_NULLPOINTER`.
- [x] `len == 0` returns 5381 regardless of `str` contents.
- [x] NULL `str` and NULL `hashval` → `AKERR_NULLPOINTER`.
- [ ] **High-bit bytes** (`"\xff\xfe"`) — pins down the sign-extension bug in §2.2.6; the
expected value must be the `unsigned char` one.
- [ ] Embedded NUL bytes are hashed (the function is length-driven, not NUL-driven).
- [ ] Same input → same output across two calls (no hidden state).
expected value must be the `unsigned char` one (5868578; the wrapper returns the
sign-extended 5859874 today). Every vector in the test file is 7-bit ASCII precisely so
that it says nothing about this case either way.
- [x] Embedded NUL bytes are hashed (the function is length-driven, not NUL-driven).
- [x] Same input → same output across two calls (no hidden state).
### 1.7 Linked list
@@ -220,8 +237,11 @@ against `tests/aksl_capture.h` and added to the `AKSL_TESTS` list.
visits 0, 1, 3, so breaking at `tree[3]` must stop the walk at **3** visits, and today
it runs on to all 7. Add the equivalent for in-order (3, 1, 4, 0, 5, 2, 6 → breaking at
`tree[4]` gives 3) and post-order (3, 4, 1, 5, 6, 2, 0 → breaking at `tree[5]` gives 4).
- [ ] `AKSL_TREE_SEARCH_BFS` and `AKSL_TREE_SEARCH_BFS_RIGHT` → `AKERR_NOT_IMPLEMENTED`
today; replace with real order assertions once implemented (§3 / §2.2.9).
- [x] `AKSL_TREE_SEARCH_BFS` and `AKSL_TREE_SEARCH_BFS_RIGHT` → `AKERR_NOT_IMPLEMENTED`
today, asserted in `tests/test_tree.c` along with the callback never being called;
replace with real order assertions once implemented (§3 / §2.2.9).
- [x] `aksl_tree_iterate` NULL `root` / NULL `iter` → `AKERR_NULLPOINTER`, and a callback error
that is *not* `AKERR_ITERATOR_BREAK` propagates out to the caller.
- [ ] **Unknown `searchmode`** (e.g. `99`) — **[CONFIRMED]** currently returns *success*
having visited nothing. Should be `AKERR_VALUE`.
- [ ] **`AKSL_TREE_SEARCH_VISIT`** (defined in the header, `5`) — **[CONFIRMED]** falls into
@@ -257,11 +277,12 @@ against `tests/aksl_capture.h` and added to the `AKSL_TESTS` list.
### 2.1 Confirmed defects (reproduced against the built library)
The first three now have a failing test apiece, registered in `AKSL_KNOWN_FAILING_TESTS`
Four of these now have a failing test apiece, registered in `AKSL_KNOWN_FAILING_TESTS`
and therefore marked `WILL_FAIL` so the suite stays green while the gap stays visible:
`tests/test_list_append_chain.c`, `tests/test_list_iterate_head.c` and
`tests/test_tree_iterate_break.c`. When one of these defects is fixed, CTest reports that
test as failed with *unexpectedly passed* — that is the cue to move it into `AKSL_TESTS`.
`tests/test_list_append_chain.c`, `tests/test_list_iterate_head.c`,
`tests/test_tree_iterate_break.c` and `tests/test_convert_strict.c`. When one of these
defects is fixed, CTest reports that test as failed with *unexpectedly passed* — that is the
cue to move it into `AKSL_TESTS`.
1. **`aksl_list_append` does not find the tail — it silently truncates the list.**
`src/stdlib.c:194`. The function conflates Floyd cycle detection with tail-finding:
@@ -297,6 +318,7 @@ test as failed with *unexpectedly passed* — that is the cue to move it into `A
`endptr` check for "no digits consumed" and "trailing junk", and a range check —
raising `AKERR_VALUE` and `ERANGE` respectively. Keep the `atoi`-compatible names but
document the stricter contract, or add `aksl_strtol`-family wrappers alongside.
`tests/test_convert_strict.c` asserts that contract and is registered as a known failure.
6. **`aksl_realpath` mishandles `resolved_path`.** `src/stdlib.c:171`.
- `resolved_path` is never NULL-checked. `realpath(path, NULL)` is valid and mallocs a