From cff2a64575d6afbb6106a7257186ce10078cb1c2 Mon Sep 17 00:00:00 2001 From: tachikoma Date: Mon, 3 Aug 2026 12:53:19 -0400 Subject: [PATCH] Recount the consumer calls against this release akbasic's src/ ported onto 0.2.0 calls this library 301 times and raw libc 13 -- 4.1% bypassed, against 86.4% on the same tree before the port and 92.2% at the first count. The port builds clean at -Wall -Wextra, passes 112/112 of akbasic's ctest suite and is ASan+UBSan-clean. The method was never written down and the figure was not reproducible. scripts/consumer_calls.py is that method, and reproducing it turned up two corrections: the old 116 was 117 by its own table's arithmetic, and 119 by a complete count -- the table had no row for strncmp or memmove. Nothing was blocked by a missing wrapper. 272 of 285 sites converted; the 13 that did not are blocked by wrapper shape, and are filed as #32-#38. Say plainly what the number does not cover: akbasic makes 0 calls into list, tree, hash map and string buffer combined, so the recount is evidence about the string, memory and format surface and about nothing else. Refs #26 Co-Authored-By: Claude Opus 5 (1M context) --- README.md | 40 ++++++- TODO.md | 126 ++++++++++++++++++-- scripts/consumer_calls.py | 245 ++++++++++++++++++++++++++++++++++++++ 3 files changed, 397 insertions(+), 14 deletions(-) create mode 100755 scripts/consumer_calls.py diff --git a/README.md b/README.md index 258d77e..1a66f47 100644 --- a/README.md +++ b/README.md @@ -181,8 +181,9 @@ would notice. ## Testing -There are four harnesses. The first three take seconds; the fourth takes about -half an hour. +There are five harnesses. The first three take seconds and the fifth is instant; +the fourth takes about half an hour. The fifth is the only one that measures +something outside this repository. ### 1. The test suite @@ -469,6 +470,41 @@ right-leaning tree would have blown the stack the depth cap exists to protect), and `aksl_tree_remove` on an empty tree, which without its guard dereferences NULL. Both are in the suite now — which is what the harness is for. +### 5. Consumer adoption + +Coverage says the tests reach the code and mutation testing says they would +notice it breaking. Neither says anybody *wanted* the code. That question only has +an external answer, so there is a harness for it too: + +```sh +scripts/consumer_calls.py ../akbasic/src # the ratio +scripts/consumer_calls.py ../akbasic/src --detail --per-file # where it comes from +scripts/consumer_calls.py ../akbasic/src --baseline 301/13 # against a past count +``` + +It counts, across a consumer's source directory, how often that consumer calls +this library against how often it reaches past it to the libc function this +library wraps. Calls to libc functions **not** wrapped here — `isdigit`, `exit`, +`qsort` — score on neither side; the question is how often an *available* wrapper +gets bypassed. Comments and string literals are stripped before counting, and the +wrapped-libc set is read out of `include/akstdlib.h` rather than hardcoded, so a +recount after a release measures the surface that release actually shipped. + +A wrapper nobody calls either does not fit or is not discoverable, and both are +this library's problem rather than the consumer's. `TODO.md` carries the standing +figures and what they did and did not justify. + +Two warnings, both learned the hard way and both printed by `--baseline`: + +- **A rate is only comparable between two counts of the same tree.** If the + consumer grew between them, compare the percentage and say which commit each + number came from. Comparing the totals across a tree that tripled in size is how + a real improvement gets reported as a regression, or the reverse. +- **One consumer's ratio is evidence, not a plan.** A consumer that draws + everything from fixed pools will never call the allocator however good the + allocator is. Weight the result by what the consumer is, and get a second + consumer before treating any ranking as settled. + ## The pre-push hook `.githooks/pre-push` runs the fast harnesses — the default build and the diff --git a/TODO.md b/TODO.md index e55eb9b..871f9d9 100644 --- a/TODO.md +++ b/TODO.md @@ -22,6 +22,7 @@ it has been through grooming. | Function coverage | 100% (154/154) | | Doxygen | 100% of 154, gated — `cmake --build build --target docs` fails on an undocumented function, parameter or return | | Mutation score | 72.3% (188/260 sampled from 1701), gated at 65 | +| Consumer adoption | akbasic ported onto 0.2.0 calls this library 301 times and raw libc 13 — **4.1% bypassed**, from 92.2% at first count. Ungated, and one consumer only | The six confirmed defects that used to head this file are fixed and `AKSL_KNOWN_FAILING_TESTS` is empty. What they were, and what changed as a result, @@ -102,15 +103,16 @@ are fixed: the right child's `depth + 1` in the depth-first walk, and ## Evidence from the first full consumer -`akbasic` (`source.starfort.tech/andrew/akbasic`) is a ~6,300-line C interpreter -built on this library and `libakerror`. It was the first consumer to exercise the -whole surface rather than a corner of it, **and what it could not use is what -prioritised everything that has been built since.** +`akbasic` (`source.starfort.tech/andrew/akbasic`) is a C interpreter built on this +library and `libakerror` — ~6,300 lines of `src/` when it was first measured, +20,169 now. It was the first consumer to exercise the whole surface rather than a +corner of it, **and what it could not use is what prioritised everything that has +been built since.** **The number that started it.** Across `src/`, akbasic made **10 calls into this -library and 116 to raw libc** — a library whose value proposition is "turn silent -libc failures into error contexts", bypassed 92% of the time by the consumer most -committed to it. +library and 119 to raw libc** — a library whose value proposition is "turn silent +libc failures into error contexts", bypassed **92%** of the time by the consumer +most committed to it. | Raw libc it had to use | Count | Now available as | |---|---|---| @@ -121,8 +123,17 @@ committed to it. | `strncpy` | 15 | `aksl_strncpy` | | `strtoll` / `strtod` | 2 | `aksl_strtoll` / `aksl_strtod` | | `fgets` | 2 | `aksl_fgets` | +| `strncmp` | 1 | `aksl_strncmp` | +| `memmove` | 1 | `aksl_memmove` | | `strstr` | 1 | `aksl_strstr` | +Two corrections to that figure, both found by rebuilding it. It used to read 116; +the table it sat above summed to 117 and had no row for `strncmp` or `memmove`. +119 is what `scripts/consumer_calls.py` returns against akbasic `4e188b2`, and it +is the number everything below compares to. **The method is now a script rather +than a paragraph, because recovering it afterwards cost more than writing it down +would have.** + **All four things the port had to write for itself now exist here.** 1. **A strict `strtoll`/`strtod` wrapper** (`akbasic/src/convert.c`, ~60 lines). The @@ -144,8 +155,99 @@ committed to it. the sign-extended djb2 reads bytes unsigned; and the missing `va_end` — which akbasic's stdio text sink ran on every line of program output — is fixed. -**Still true, and still shaping the wishlist.** akbasic uses no allocator, no lists -and no trees, drawing everything from fixed pools by design. **A consumer that does -allocate would weight the `open`/`read`/`write` work far higher than this one -does**, so one consumer's count is evidence, not a plan. Re-counting against this -release is #26. +### The recount, against this release + +akbasic's `src/` was ported onto 0.2.0 and counted again (#26). The port builds +clean at `-Wall -Wextra`, passes **112/112** of akbasic's ctest suite, and is +ASan+UBSan-clean. + +| | libakstdlib | raw libc | bypassed | +|---|---|---|---| +| Baseline — akbasic `4e188b2`, 5,679 lines of `src/` | 10 | 119 | **92.2%** | +| Before the port — akbasic `330d731`, 20,169 lines | 45 | 285 | **86.4%** | +| **After the port — same tree** | **301** | **13** | **4.1%** | + +**Read the third row against the second, not the first.** The tree tripled between +the baseline and the port, so 10/119 and 45/285 are counts of two different +programs; only 86.4% → 4.1% is a like-for-like measurement. The 45 in the middle +row is worth its own note — akbasic had already adopted `aksl_f*` across +`runtime_disk.c` on its own, without anybody counting. + +**Nothing was blocked by a missing wrapper.** Every libc call akbasic makes had an +`aksl_*` counterpart. 272 of the 285 sites converted; the 13 that did not are +blocked by wrapper *shape*, and they are the useful output: + +| Why it could not be used | Sites | Where | +|---|---|---| +| No error channel to route into — the enclosing function returns `bool` or `void`, or is a `bsearch` comparator whose signature libc fixes | 8 | `structtype.c` `word_is`, `environment.c` `akbasic_environment_is_waiting_for` (public API), `scanner.c` `is_at_end` and `peek_next`, `verbs.c` `verb_compare`, `format.c` `overflow`, `sink_akgl.c` `scroll` (×2) | +| Truncation is the answer, not the error | 4 | `format.c`, `structtype.c`, `runtime_struct.c`, `renumber.c` | +| Short-circuit is memory-safety-load-bearing and the compare cannot be hoisted past the NULL arm guarding it | 1 | `runtime_trap.c` | + +The truncation four are worth spelling out, because they are a contract decision +rather than an accident. `PRINT USING "###"; 1E300` prints `***` today: the render +truncates, the truncated text has no `.`, and the formatter takes its overflow +path on exactly that. Through `aksl_snprintf` it raises `AKERR_OUTOFBOUNDS` out of +the interpreter instead. Two more are truncation-tolerant renderers that print +what fits and stop, and the fourth uses `snprintf`'s return to raise akbasic's +own `AKBASIC_ERR_BOUNDS` with its own message. + +### What the recount found, and where it went + +Every one of the 13 blocked sites came back to wrapper *shape* rather than a +missing wrapper, and the same seven shapes recurred across ten independent +conversion passes. They are filed, not listed here: + +| Finding | Filed as | +|---|---| +| `aksl_snprintf`'s `count` out-param is required, so ~20 sites carry an `int written` that is written and never read. Raised by all ten passes. `-Wall -Wextra` cannot see it — `&written` is a use | #32 | +| No equality comparison. All 43 comparison sites flatten the three-way `int` to `== 0`; not one wants an ordering, and five now need a sentinel whose *initial value is load-bearing* | #33 | +| No truncating format and no length query, which is the whole of the truncation-four above | #34 | +| `aksl_hashmap_*` carries one payload, which is the only reason `akbasic/src/symtab.c` still exists | #35 | +| `aksl_fgets` signals end of input by raising, so a read loop cannot be a condition | #36 | +| A caller cannot add its own context to a wrapper's error, so it raises and discards instead — eight lines where there were two | #37 | +| No form a `bool` predicate or a `void` function can call, which is 8 of the 13 blocked sites. Carries the `ctype.h` question and the infallible-`memset` question with it | #38 | + +`#14` already covered the `bsearch` comparator, and the port confirmed it from the +consumer side. + +**The one thing the wrappers did better than the libc they replaced** is worth +recording next to the complaints: `aksl_fgets`'s `len_out` **deleted** two `strlen` +calls rather than converting them, and is more correct than what it replaced for a +line containing an embedded NUL. It is the only one of 272 conversions that +produced less code than it started with. + +### Still true, and still the reason one count is not a plan + +akbasic uses no allocator, no lists and no trees, drawing everything from fixed +pools by design, and porting it did not change that. Of the 301 calls it now +makes: + +| Area | Calls | | +|---|---|---| +| Strings | 122 | 40.5% | +| Memory | 69 | 22.9% | +| Formatted output | 59 | 19.6% | +| Streams | 38 | 12.6% | +| String → number | 12 | 4.0% | +| Hashing | 1 | 0.3% | +| **Collections** | **0** | **0%** | + +**Four fifths of the evidence is strings, memory and formatting.** The collections +work — list, tree, hash map, string buffer, `src/collections.c` and the largest +single body of code in this library — has **not one consumer call site**, and the +single hashing call next to it is `aksl_strhash_djb2` feeding a hash table akbasic +wrote for itself. **A consumer that does allocate would weight the +`open`/`read`/`write` work far higher than this one does**, so this remains +evidence and not a plan. + +**The number to distrust is not the 4.1%; it is the 0%.** A recount that moves +92% to 4% on one consumer says the string, memory and format wrappers fit the +consumer that asked for them. It says nothing at all about the half of the library +that consumer never calls, and it cannot, however many times it is run. What would +say something is a second consumer with different shape — one that allocates. + +`akbasic/src/symtab.c` is the sharpest instance. It is the hand-rolled fixed-capacity +string-keyed hash table `aksl_hashmap_*` was generalised from, it survived the port +untouched, and the reason turned out to be one field rather than a design +disagreement — everything else about the two already lines up. #35 has it, and it +is the first collections work with a consumer actually waiting for it. diff --git a/scripts/consumer_calls.py b/scripts/consumer_calls.py new file mode 100755 index 0000000..b8f01f1 --- /dev/null +++ b/scripts/consumer_calls.py @@ -0,0 +1,245 @@ +#!/usr/bin/env python3 +""" +Consumer adoption harness for libakstdlib. + +Answers one question about a consumer's source tree: how often does it call +this library, and how often does it reach past this library to the libc +function this library wraps? + +The ratio is the only external evidence there is about whether the surface that +got built is the surface anyone wanted. A wrapper nobody calls is a wrapper that +either does not fit or is not discoverable, and both are the library's problem. + +The number this reports is only worth something if it is reproducible, which is +why this is a script and not a paragraph. TODO.md's first consumer figure was +recorded without one, and recovering the method afterwards cost more than +writing it down would have. + +WHAT COUNTS + + * The corpus is every *.c and *.h directly under the given directory. It does + not recurse: a consumer's src/ is the thing being measured, not its vendored + dependencies, and those are usually a subdirectory. + + * Comments and string/character literals are stripped before anything is + counted, so a function named in prose or inside a format string does not + score. This matters more than it sounds -- "strlen" appears in doc comments + throughout a codebase that has been thinking about strlen. + + * A call site is IDENT immediately followed by '(', where IDENT is not + preceded by an identifier character. Declarations are not distinguished from + calls; a consumer that declares a function named for a libc entry point will + over-count by one per declaration, which is visible in --detail. + + * A library call is any IDENT matching ^aksl_. + + * A bypass is any IDENT naming a libc function this library wraps. That set is + read out of include/akstdlib.h rather than hardcoded, so it grows when the + library grows and a recount after a release measures the surface that + release actually shipped. + + * libc functions this library does NOT wrap -- isdigit, exit, qsort -- score + on neither side. The question is how often a consumer bypasses an available + wrapper, not how much libc it uses. Adding a wrapper for something and + having it ignored is a finding; a consumer calling exit() is not. + +WHAT IT CANNOT TELL YOU + + One consumer's ratio is evidence, not a plan. A consumer that draws + everything from fixed pools will never call the allocator no matter how good + the allocator is, and will weight the string wrappers accordingly. Weight the + result by what the consumer is, and get a second consumer before treating any + ranking as settled. + +Usage: + scripts/consumer_calls.py DIR [options] + + DIR consumer source directory to measure, e.g. + ../akbasic/src + --header PATH akstdlib.h to read the wrapped-libc set from + (default: include/akstdlib.h beside this script's repo) + --detail list the per-function breakdown on both sides + --per-file list per-file counts, worst bypass ratio first + --baseline A/B compare against a previous count, e.g. --baseline 10/119 + --json emit the whole result as JSON instead of text +""" + +import argparse +import json +import os +import re +import sys +from collections import Counter + +# Wrapper families that are this library's own constructs rather than a libc +# function under a new name. aksl_list_append has no libc counterpart, so +# "append" must not become a name a consumer can be scored for bypassing. +LIBRARY_ONLY_PREFIXES = ("hashmap", "list", "tree", "strbuf", "version", + "strhash") + +# Library-only names whose first underscore-separated word is shared with a real +# libc entry point, so a prefix rule cannot separate them. aksl_realpath wraps +# realpath(3) and must score; aksl_realpath_alloc is this library's own. +LIBRARY_ONLY_NAMES = frozenset(("freep", "realpath_alloc")) + +CALL = re.compile(r"(?5}{raw:>6}") + + return 0 + + +if __name__ == "__main__": + sys.exit(main())