Move BASIC fixtures into the editable language corpus
All checks were successful
akbasic CI Build / cmake_build (push) Successful in 3m34s
akbasic CI Build / coverage (push) Successful in 4m4s
akbasic CI Build / sanitizers (push) Successful in 6m59s
akbasic CI Build / akgl_build (push) Successful in 7m57s
akbasic CI Build / mutation_test (push) Successful in 23m28s

Move every program and expectation out of tests/reference and register the unified tests/language corpus as local cases. Remove the old immutable-corpus protections from build, maintenance, and documentation paths.

Co-authored-by: andrew <andrew@aklabs.net>
This commit is contained in:
2026-08-04 16:21:59 -04:00
parent fac84acdaa
commit 8a02674af5
96 changed files with 69 additions and 168 deletions

37
TODO.md
View File

@@ -38,7 +38,7 @@ What it changes:
- **§1.8's message-text contract is now a convention.** Improving a message is allowed; it costs
a golden file, which is a cost rather than a veto.
- **§5's bar drops** from "defensible against the golden suite" to defensible on its own merits.
- **`tests/reference/` becomes a regression suite rather than a specification.** Diverging from
- **`tests/language/` is the editable language corpus rather than a protected specification.** Diverging from
it is allowed and must be deliberate and recorded — see its README.
What it does **not** change:
@@ -306,13 +306,13 @@ to `Println`, which adds another.
**This used to be a hard contract and is now a default.** The Go implementation is deprecated
and will not be updated, so the two projects are no longer required to match — see §0.1. What
survives is the practical half: these strings and this newline behaviour are what every
expectation in `tests/reference/` was written against, so changing one means changing golden
expectation in `tests/language/` was written against, so changing one means changing the paired
files, and that is worth doing on purpose rather than by accident. A message that reads
awkwardly *may* now be improved; do it deliberately, move the expectations in the same commit,
and add a line to §5.
Numeric formatting still matches the reference: integers via `%" PRId64 "`, floats via `%f`
(Go's `%f` and C's `%f` both give six decimals — `tests/reference/language/arithmetic/float.txt`
(Go's `%f` and C's `%f` both give six decimals — `tests/language/arithmetic/float.txt`
confirms). No reason to change it, which is different from not being allowed to.
### 1.9 Which `libakstdlib` calls are cleared for use — **the bans are lifted**
@@ -375,7 +375,7 @@ Phases 0 through 6 of the original plan are done. The interpreter builds clean u
It *did* reproduce the reference byte for byte, and that claim is retired rather than broken:
§0.1 released it, and one case has since diverged deliberately (§6 item 16, listed in
`tests/reference/README.md`). Everything else still matches, which is worth knowing but is no
`tests/language/README.md`). Everything else still matches, which is worth knowing but is no
longer a gate.
```sh
@@ -408,17 +408,16 @@ and the only path that existed — `AKBASIC_MODE_RUNSTREAM` reading through the
is the code the README quotes, built by every build and registered as a CTest case so a
signature change breaks the build rather than rotting the document.
**The acceptance suite is the reference's own corpus, checked in at `tests/reference/`.** All
41 `.bas` files are registered as individual CTest cases and byte-compared against their `.txt`
**The acceptance suite is the editable language corpus, checked in at `tests/language/`.** All
65 `.bas` files are registered as individual CTest cases and compared against their `.txt`
— including the trailing double newline on an error line (§1.8).
It was driven *in place* out of `deps/basicinterpret` until 2026-07-31, on the reasoning that
copying a submodule's corpus guarantees drift. That reasoning was sound and was overruled
deliberately: the Go dependency is being deprecated, and a build that cannot run its own
acceptance suite without cloning the implementation it replaced is not finished. The copy is
byte-identical to `basicinterpreter@d76162c`, and `tests/reference/README.md` records the
provenance, the cost of the drift nobody is watching for now, and the rule that those
expectations are never edited to suit this interpreter.
originally byte-identical to `basicinterpreter@d76162c`, and `tests/language/README.md` records the
provenance and the rule that programs and expectations are edited together deliberately.
**Nothing in the build or the suite needs `deps/basicinterpret` any more**, and that is checked
rather than assumed — both configurations were configured, built and run from scratch with the
@@ -1026,7 +1025,7 @@ deviations from the reference's *program*: `main.go` and the SDL half of
**What it costs a program:** a listing that used `INPUT$`, `LEN#` or `GOTO%` as a variable
stops parsing, and the fix is to rename the variable. One case in the reference's own corpus
did exactly that; see `tests/reference/README.md`.
did exactly that; see `tests/language/examples/strreverse.bas`.
### Deviations in statement separation
@@ -1391,10 +1390,10 @@ deviations from the reference's *program*: `main.go` and the SDL half of
**This one moved a golden file.** A line with no number used to be filed under the loader's
cursor unchanged — that is, on top of the line before it — so two unnumbered lines in a row
silently lost the first, and a *blank* line erased whatever preceded it. The reference does
the same, and `tests/reference/language/arithmetic/integer.bas` is the proof: four `PRINT`
the same, and `tests/language/arithmetic/integer.bas` is the proof: four `PRINT`
statements, an expectation with three values, and a trailing blank line that erased
`40 PRINT 4 - 2` before the program ran. The expectation is now `4 4 2 2` and
`tests/reference/README.md` records it.
`tests/language/README.md` records it.
In its place: `akbasic_runtime_file_line()` (`src/runtime.c`) is the one implementation of
the rule, shared by `akbasic_runtime_load()`, RUNSTREAM and `DLOAD`. A numbered line is
@@ -1586,9 +1585,9 @@ be reproduced before it can be fixed.
**It cost one golden case, exactly as predicted, and the cost turned out to be nothing.**
The reference's `examples/strreverse.bas` names a variable `INPUT$`; the variable is renamed
to `SOURCE$` in `tests/reference/`, the expectation is byte-for-byte unchanged — the program
to `SOURCE$` in `tests/language/examples/strreverse.bas`, the expectation is byte-for-byte unchanged — the program
still prints `REVERSED: OLLEH` — and the case keeps every bit of its coverage. Recorded in
`tests/reference/README.md`'s divergence table, which this is the first entry in.
`tests/language/README.md` records the corpus editing rule.
This item sat parked because the corpus was the acceptance contract and lived in a submodule
this repository could not edit. Both premises are gone: the corpus is checked in, and
@@ -2232,10 +2231,10 @@ update --init --recursive` gets them.
## 8. Status
**The port is done.** The C interpreter passes the Go reference's entire corpus and passes clean
**The port is done.** The C interpreter passes the language corpus and passes clean
under ASan and UBSan. It reproduced that corpus byte for byte until §0.1 retired the
requirement; two cases have diverged on purpose since, and
`tests/reference/README.md` lists them.
`tests/language/README.md` lists them.
| Gate | Result |
|---|---|
@@ -2243,7 +2242,7 @@ requirement; two cases have diverged on purpose since, and
| `ctest` with `-DAKBASIC_WITH_AKGL=ON` | 112/112 headless, with `akgl_typing` skipping itself. The same set minus the four `no_device` cases the SDL driver contradicts, plus `akgl_backends`, `akgl_frontend`, `docs_screenshots` and `akgl_typing` — the last of which is the skip, and the `akgl_build` CI job is where it skips |
| `docs_examples` | Every fenced block in `README.md`, `MAINTENANCE.md` and `docs/` executed and byte-compared: 71 programs, 9 transcripts, 79 output comparisons, 4 C snippets, 2 excerpts, 2 shell blocks and 19 figures in the default build. The C-snippet count reads 0 when the harness is run by hand without `--cflags-file`; CTest passes it. `MAINTENANCE.md` documents the fence-tag convention |
| `docs_screenshots` | 19/19 figures re-rendered and byte-identical to the checked-in PNGs. AKGL build only — rendering a picture needs the SDL half |
| Golden corpus | 41/41 byte-exact from `tests/reference/` — **and 41/41 again through the SDL binary**, which is most of what proves the frontend changes no output |
| Language corpus | 65/65 paired expectations — **and 65/65 again through the SDL binary**, which is most of what proves the frontend changes no output |
| ASan + UBSan | 112/112 |
| Line coverage | 94.1% (7227/7681) — above the 90% gate |
| Function coverage | 97.9% (474/484) |
@@ -2553,7 +2552,7 @@ What remains, in priority order:
this whole arrangement exists against.
The byte comparison is a deliberate bet that the dummy driver and the software renderer
are reproducible, which is the same bet `tests/reference/` already makes about golden
are reproducible, which is the same bet `tests/language/` makes about golden
output. Verified run-to-run and build-to-build here; what is untested is an SDL upgrade
that moves one pixel of a diagonal. **If that happens, regenerate the figures in the same
commit as the bump** — do not weaken the test to a size check, which would pass for every
@@ -2636,7 +2635,7 @@ reduced against `build/basic`, the stdio build, unless it says otherwise.
**It is narrower than it looks, and the golden corpus is why.** The first attempt
released the scope on *any* skip, which broke
`tests/reference/language/flowcontrol/nestedforloopwaitingforcommand.bas`: a zero-
`tests/language/flowcontrol/nestedforloopwaitingforcommand.bas`: a zero-
iteration `FOR` skips its body by the same mechanism, and there the orphan is
load-bearing -- it is what absorbs the inner `NEXT` so the outer one still finds its
`FOR`. Releasing it turns that case into "NEXT outside the context of FOR". So the