Execute every documented example as a test
Some checks failed
akbasic CI Build / cmake_build (push) Successful in 3m2s
akbasic CI Build / sanitizers (push) Successful in 3m52s
akbasic CI Build / coverage (push) Failing after 3m24s
akbasic CI Build / akgl_build (push) Failing after 20s
akbasic CI Build / mutation_test (push) Has been cancelled

docs/ and README.md carry 85 fenced blocks. Every one was checked by hand
exactly once, when it was written, which is not a standard that survives a
changing interpreter -- and four were already wrong: two transcripts showing a
leading space PRINT does not emit, akbasic_TextSink in README.md missing the
two members it had grown hours earlier, and FILTER's refusal quoted with
wording the code does not use.

tests/docs_examples.sh reads a fence-tag vocabulary and runs what it finds.
BASIC programs and transcripts run and are byte-compared against an `output`
block; C snippets compile with -fsyntax-only against the real include path,
which CMake writes out because it is transitive through akerror, akstdlib and
akgl; shell blocks run in a sandbox. Anything that would reconfigure the build
tree, hit the network or re-enter the suite is tagged norun with the reason in
MAINTENANCE.md, and the two cmake blocks stay hand-maintained by decision.

An untagged block is a failure rather than a default, and the pass line
reports what it executed by kind. Both exist because the way a harness like
this dies is by quietly matching nothing and passing -- which it duly did on
the first CTest run, where a generator expression evaluating to nothing still
contributed an empty argument that the script read as a filename. The count is
what caught it.

The excerpt check earns its own mention: a block tagged
`c excerpt=include/akbasic/sink.h` must still appear in that header, comments
and whitespace ignored. Compiling it would only redefine the type, so a
compile check could not have found the stale struct, and did not.

Registered as the CTest case docs_examples in both configurations. Fixing the
four wrong examples turned up two interpreter defects, fixed in the previous
commit and recorded in TODO.md section 8.

MAINTENANCE.md is new: the fence-tag reference, what to do when the case
fails, and the conventions that until now only existed inside source comments
-- the three test lists and how two of them invert "passed", the sorted verb
table, that a golden file is never edited to suit this interpreter, and that a
fix gets mutation-checked with a file copy rather than git checkout.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-07-31 22:41:36 -04:00
parent 6f49f6a7f2
commit 342e4c07da
29 changed files with 1303 additions and 103 deletions

36
TODO.md
View File

@@ -1527,8 +1527,9 @@ requirement; exactly one case has diverged on purpose since, and
| Gate | Result |
|---|---|
| `ctest` | 77/77 — 41 reference golden cases, 9 local ones, 25 unit tests, 2 embedding examples. No known-failing tests: the list is empty |
| `ctest` with `-DAKBASIC_WITH_AKGL=ON` | 77/77 on a machine with a display; 76 passed + 1 skipped headless. The same set minus the three `no_device` cases the SDL driver contradicts, plus `akgl_backends`, `akgl_frontend` and `akgl_typing` — the last of which is the skip, and the `akgl_build` CI job is where it skips |
| `ctest` | 95/95 — 41 reference golden cases, 15 local ones, 36 unit tests, 2 embedding examples, and `docs_examples` |
| `ctest` with `-DAKBASIC_WITH_AKGL=ON` | 94/94 on a machine with a display; 93 passed + 1 skipped headless. The same set minus the three `no_device` cases the SDL driver contradicts, plus `akgl_backends`, `akgl_frontend` and `akgl_typing` — the last of which is the skip, and the `akgl_build` CI job is where it skips |
| `docs_examples` | Every fenced block in `README.md`, `MAINTENANCE.md` and `docs/` executed and byte-compared: 37 programs, 9 transcripts, 45 output comparisons, 6 C snippets, 1 excerpt and 2 shell blocks in the default build; 50 programs and 7 C snippets in the AKGL one. `MAINTENANCE.md` documents the fence-tag convention |
| Golden corpus | 41/41 byte-exact from `tests/reference/`**and 41/41 again through the SDL binary**, which is most of what proves the frontend changes no output |
| ASan + UBSan | 77/77 |
| Line coverage | 94.6% (3857/4076) — above the 90% gate |
@@ -1537,6 +1538,25 @@ requirement; exactly one case has diverged on purpose since, and
| `doxygen Doxyfile` | clean |
| Mutation (`src/symtab.c`) | 74.1%, against a gate of 65 — see `.gitea/workflows/ci.yaml` |
**Two defects surfaced from writing that harness**, both fixed with a test that fails when the
fix is reverted:
1. **`DLOAD` overwrote the last line of every program it loaded from a prompt.** It scans the
file through the runtime's own scanner, so on return the tokens of the calling line are gone
and `hadlinenumber` and `environment->lineno` describe the last line of the *file*. The REPL
carried on parsing what it believed was still its own buffer, decided the leftovers were
program text, and filed the `DLOAD` command under that line number.
`src/runtime_commands.c` now sets `skiprestofline`. `tests/disk_verbs.c`.
2. **A BASIC error typed at the prompt terminated the driver.** `process_line_run()` had always
swallowed the context after `interpret()` put the error line on the sink — goal 3, in the
comment, in so many words — but the direct-mode branch of `process_line_repl()` used a bare
`PASS`. A `VERIFY` against a file that did not match, which is an ordinary user answer rather
than a fault, came out as a stack trace and would have taken an embedding host with it.
`tests/housekeeping_verbs.c`.
Neither was on any list. Both were found by writing down what a chapter claimed and then
running it — which is the argument for the harness rather than a footnote to it.
Coverage of the verb groups added for goal 2, since they are the new surface: `src/audio_tables.c`
and `src/graphics_tables.c` 100%, `src/runtime_input.c` 100%, `src/runtime_graphics.c` and
`src/runtime_audio.c` 98%, `src/play.c` 87%. The `akbasic_akgl` target is not in the coverage
@@ -1707,3 +1727,15 @@ What remains, in priority order:
`src/value.c` takes about 70 minutes to mutate in full (368 mutants, ~11s each, because
almost everything links against it), which is why CI runs `src/symtab.c` instead and the
whole-tree run is a local `cmake --build build --target mutation`.
9. Use the actual full SDL window for graphics operation width
The current documentation states that graphics operations always use a 320x200 space regardless of the window's size. These verbs should be able to access the entirety of the SDL window to which they are bound.
11. Error codes are undefined for the user
In the documentation, include a complete table of all possible values of ER# and EL# in the documentation in an appendix.
12. Screenshots for graphics operations
Chapter 8 of the documentation should include screenshots of various graphics operations being performed.