Execute every documented example as a test
Some checks failed
akbasic CI Build / cmake_build (push) Successful in 3m2s
akbasic CI Build / sanitizers (push) Successful in 3m52s
akbasic CI Build / coverage (push) Failing after 3m24s
akbasic CI Build / akgl_build (push) Failing after 20s
akbasic CI Build / mutation_test (push) Has been cancelled
Some checks failed
akbasic CI Build / cmake_build (push) Successful in 3m2s
akbasic CI Build / sanitizers (push) Successful in 3m52s
akbasic CI Build / coverage (push) Failing after 3m24s
akbasic CI Build / akgl_build (push) Failing after 20s
akbasic CI Build / mutation_test (push) Has been cancelled
docs/ and README.md carry 85 fenced blocks. Every one was checked by hand exactly once, when it was written, which is not a standard that survives a changing interpreter -- and four were already wrong: two transcripts showing a leading space PRINT does not emit, akbasic_TextSink in README.md missing the two members it had grown hours earlier, and FILTER's refusal quoted with wording the code does not use. tests/docs_examples.sh reads a fence-tag vocabulary and runs what it finds. BASIC programs and transcripts run and are byte-compared against an `output` block; C snippets compile with -fsyntax-only against the real include path, which CMake writes out because it is transitive through akerror, akstdlib and akgl; shell blocks run in a sandbox. Anything that would reconfigure the build tree, hit the network or re-enter the suite is tagged norun with the reason in MAINTENANCE.md, and the two cmake blocks stay hand-maintained by decision. An untagged block is a failure rather than a default, and the pass line reports what it executed by kind. Both exist because the way a harness like this dies is by quietly matching nothing and passing -- which it duly did on the first CTest run, where a generator expression evaluating to nothing still contributed an empty argument that the script read as a filename. The count is what caught it. The excerpt check earns its own mention: a block tagged `c excerpt=include/akbasic/sink.h` must still appear in that header, comments and whitespace ignored. Compiling it would only redefine the type, so a compile check could not have found the stale struct, and did not. Registered as the CTest case docs_examples in both configurations. Fixing the four wrong examples turned up two interpreter defects, fixed in the previous commit and recorded in TODO.md section 8. MAINTENANCE.md is new: the fence-tag reference, what to do when the case fails, and the conventions that until now only existed inside source comments -- the three test lists and how two of them invert "passed", the sorted verb table, that a golden file is never edited to suit this interpreter, and that a fix gets mutation-checked with a file copy rather than git checkout. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
36
TODO.md
36
TODO.md
@@ -1527,8 +1527,9 @@ requirement; exactly one case has diverged on purpose since, and
|
||||
|
||||
| Gate | Result |
|
||||
|---|---|
|
||||
| `ctest` | 77/77 — 41 reference golden cases, 9 local ones, 25 unit tests, 2 embedding examples. No known-failing tests: the list is empty |
|
||||
| `ctest` with `-DAKBASIC_WITH_AKGL=ON` | 77/77 on a machine with a display; 76 passed + 1 skipped headless. The same set minus the three `no_device` cases the SDL driver contradicts, plus `akgl_backends`, `akgl_frontend` and `akgl_typing` — the last of which is the skip, and the `akgl_build` CI job is where it skips |
|
||||
| `ctest` | 95/95 — 41 reference golden cases, 15 local ones, 36 unit tests, 2 embedding examples, and `docs_examples` |
|
||||
| `ctest` with `-DAKBASIC_WITH_AKGL=ON` | 94/94 on a machine with a display; 93 passed + 1 skipped headless. The same set minus the three `no_device` cases the SDL driver contradicts, plus `akgl_backends`, `akgl_frontend` and `akgl_typing` — the last of which is the skip, and the `akgl_build` CI job is where it skips |
|
||||
| `docs_examples` | Every fenced block in `README.md`, `MAINTENANCE.md` and `docs/` executed and byte-compared: 37 programs, 9 transcripts, 45 output comparisons, 6 C snippets, 1 excerpt and 2 shell blocks in the default build; 50 programs and 7 C snippets in the AKGL one. `MAINTENANCE.md` documents the fence-tag convention |
|
||||
| Golden corpus | 41/41 byte-exact from `tests/reference/` — **and 41/41 again through the SDL binary**, which is most of what proves the frontend changes no output |
|
||||
| ASan + UBSan | 77/77 |
|
||||
| Line coverage | 94.6% (3857/4076) — above the 90% gate |
|
||||
@@ -1537,6 +1538,25 @@ requirement; exactly one case has diverged on purpose since, and
|
||||
| `doxygen Doxyfile` | clean |
|
||||
| Mutation (`src/symtab.c`) | 74.1%, against a gate of 65 — see `.gitea/workflows/ci.yaml` |
|
||||
|
||||
**Two defects surfaced from writing that harness**, both fixed with a test that fails when the
|
||||
fix is reverted:
|
||||
|
||||
1. **`DLOAD` overwrote the last line of every program it loaded from a prompt.** It scans the
|
||||
file through the runtime's own scanner, so on return the tokens of the calling line are gone
|
||||
and `hadlinenumber` and `environment->lineno` describe the last line of the *file*. The REPL
|
||||
carried on parsing what it believed was still its own buffer, decided the leftovers were
|
||||
program text, and filed the `DLOAD` command under that line number.
|
||||
`src/runtime_commands.c` now sets `skiprestofline`. `tests/disk_verbs.c`.
|
||||
2. **A BASIC error typed at the prompt terminated the driver.** `process_line_run()` had always
|
||||
swallowed the context after `interpret()` put the error line on the sink — goal 3, in the
|
||||
comment, in so many words — but the direct-mode branch of `process_line_repl()` used a bare
|
||||
`PASS`. A `VERIFY` against a file that did not match, which is an ordinary user answer rather
|
||||
than a fault, came out as a stack trace and would have taken an embedding host with it.
|
||||
`tests/housekeeping_verbs.c`.
|
||||
|
||||
Neither was on any list. Both were found by writing down what a chapter claimed and then
|
||||
running it — which is the argument for the harness rather than a footnote to it.
|
||||
|
||||
Coverage of the verb groups added for goal 2, since they are the new surface: `src/audio_tables.c`
|
||||
and `src/graphics_tables.c` 100%, `src/runtime_input.c` 100%, `src/runtime_graphics.c` and
|
||||
`src/runtime_audio.c` 98%, `src/play.c` 87%. The `akbasic_akgl` target is not in the coverage
|
||||
@@ -1707,3 +1727,15 @@ What remains, in priority order:
|
||||
`src/value.c` takes about 70 minutes to mutate in full (368 mutants, ~11s each, because
|
||||
almost everything links against it), which is why CI runs `src/symtab.c` instead and the
|
||||
whole-tree run is a local `cmake --build build --target mutation`.
|
||||
|
||||
9. Use the actual full SDL window for graphics operation width
|
||||
|
||||
The current documentation states that graphics operations always use a 320x200 space regardless of the window's size. These verbs should be able to access the entirety of the SDL window to which they are bound.
|
||||
|
||||
11. Error codes are undefined for the user
|
||||
|
||||
In the documentation, include a complete table of all possible values of ER# and EL# in the documentation in an appendix.
|
||||
|
||||
12. Screenshots for graphics operations
|
||||
|
||||
Chapter 8 of the documentation should include screenshots of various graphics operations being performed.
|
||||
|
||||
Reference in New Issue
Block a user