1 Commits
34 ... 40

Author SHA1 Message Date
8a02674af5 Move BASIC fixtures into the editable language corpus
All checks were successful
akbasic CI Build / cmake_build (push) Successful in 3m34s
akbasic CI Build / coverage (push) Successful in 4m4s
akbasic CI Build / sanitizers (push) Successful in 6m59s
akbasic CI Build / akgl_build (push) Successful in 7m57s
akbasic CI Build / mutation_test (push) Successful in 23m28s
Move every program and expectation out of tests/reference and register the unified tests/language corpus as local cases. Remove the old immutable-corpus protections from build, maintenance, and documentation paths.

Co-authored-by: andrew <andrew@aklabs.net>
2026-08-04 16:22:16 -04:00
96 changed files with 69 additions and 168 deletions

View File

@@ -17,7 +17,7 @@ jobs:
# Not recursive, deliberately. The top-level build needs # Not recursive, deliberately. The top-level build needs
# deps/libakerror and deps/libakstdlib, via add_subdirectory, and # deps/libakerror and deps/libakstdlib, via add_subdirectory, and
# nothing else: the golden corpus and the Commodore font now live in # nothing else: the golden corpus and the Commodore font now live in
# this repository (tests/reference/ and assets/fonts/), so # this repository (tests/language/ and assets/fonts/), so
# deps/basicinterpret is no longer a build dependency at all. # deps/basicinterpret is no longer a build dependency at all.
# It does *not* need deps/libakgl, which is guarded behind # It does *not* need deps/libakgl, which is guarded behind
# AKBASIC_WITH_AKGL and defaults OFF -- and recursing into it would # AKBASIC_WITH_AKGL and defaults OFF -- and recursing into it would
@@ -62,11 +62,9 @@ jobs:
run: | run: |
cmake -S . -B build cmake -S . -B build
cmake --build build --parallel 2 cmake --build build --parallel 2
# The suite is 78 cases: 41 golden files byte-compared against the Go # The suite is 112 cases: 65 language files with sibling expectations,
# reference's own corpus (checked in at tests/reference/, see its README), # 43 unit tests, 3 embedding examples, and docs_examples.
# 9 local golden cases for verbs the reference never implemented, 25 unit # Some unit tests assert the *correct* contract for known defects (TODO.md
# tests, 2 embedding examples, and 1 known-failing test that asserts the
# *correct* contract for defects carried over from the reference (TODO.md
# section 6). A green run therefore does not mean defect-free -- see # section 6). A green run therefore does not mean defect-free -- see
# AKBASIC_KNOWN_FAILING_TESTS. # AKBASIC_KNOWN_FAILING_TESTS.
# #

View File

@@ -57,9 +57,9 @@ repeating where you will see them:
`include/akgl/SDL_GameControllerDB.h`. Change the template or the generator script. `include/akgl/SDL_GameControllerDB.h`. Change the template or the generator script.
- **Do not reformat code you are not otherwise changing.** Several files mix tabs and spaces - **Do not reformat code you are not otherwise changing.** Several files mix tabs and spaces
and there is no repo-wide formatter; style conversions get their own commit. and there is no repo-wide formatter; style conversions get their own commit.
- **Do not edit `tests/reference/`.** Those expectations came from the Go implementation and - **Keep `tests/language/` editable.** Its `.bas` programs and sibling `.txt` expectations
are never edited to suit this interpreter. A deliberate divergence goes in are changed together when behavior changes. Record deliberate language decisions in
`tests/reference/README.md`'s divergence table and `docs/13-differences.md`. `TODO.md` or `docs/13-differences.md`.
- **Open an issue for outstanding work; do not add it to `TODO.md`.** - **Open an issue for outstanding work; do not add it to `TODO.md`.**
<https://source.starfort.tech/andrew/akbasic/issues>, or `tea issues create --repo <https://source.starfort.tech/andrew/akbasic/issues>, or `tea issues create --repo
andrew/akbasic`. Name the file and line, the functional consequence, and what closing it would andrew/akbasic`. Name the file and line, the functional consequence, and what closing it would

View File

@@ -448,7 +448,7 @@ if(AKBASIC_WITH_AKGL)
# against. # against.
# #
# **A byte comparison of a rendered PNG is a deliberate bet**, the same bet # **A byte comparison of a rendered PNG is a deliberate bet**, the same bet
# tests/reference/ already makes about golden output: that the dummy video # the language corpus makes about golden output: that the dummy video
# driver and the software renderer are reproducible. They are, run to run and # driver and the software renderer are reproducible. They are, run to run and
# build to build. What is untested is an SDL upgrade that shifts one pixel of # build to build. What is untested is an SDL upgrade that shifts one pixel of
# a diagonal, and the answer to that is to regenerate the figures in the same # a diagonal, and the answer to that is to regenerate the figures in the same
@@ -531,63 +531,20 @@ if(AKBASIC_WILL_FAIL_TESTS OR AKBASIC_KNOWN_FAILING_TESTS)
) )
endif() endif()
# The reference's own corpus, byte-compared against the sibling .txt. One CTest # The editable language corpus. One CTest case per .bas so a failure names the
# case per .bas so a failure names the file. # file. Each program is paired with a sibling .txt expectation; see
# tests/language/README.md for the editing rule.
# #
# It used to be driven in place out of deps/basicinterpret, on the reasoning that # The corpus includes programs carried over from the deprecated Go implementation
# copying a submodule's corpus guarantees drift. That reasoning was sound and it # and cases written for this interpreter. Their provenance is useful when
# has been overruled deliberately: the Go dependency is being deprecated, and a # investigating a regression, but it does not make any case immutable.
# build that cannot run its acceptance suite without cloning the implementation
# it replaced is not finished. The copy is byte-identical to
# basicinterpreter@d76162c and tests/reference/README.md records
# where it came from and what the drift now costs.
file(GLOB_RECURSE AKBASIC_GOLDEN_CASES
RELATIVE "${CMAKE_CURRENT_SOURCE_DIR}/tests/reference"
"${CMAKE_CURRENT_SOURCE_DIR}/tests/reference/*.bas"
)
foreach(_case IN LISTS AKBASIC_GOLDEN_CASES)
string(REGEX REPLACE "^tests/" "" _name "${_case}")
string(REGEX REPLACE "\\.bas$" "" _name "${_name}")
string(REPLACE "/" "_" _name "${_name}")
_add_test(
NAME golden_${_name}
COMMAND ${CMAKE_COMMAND}
-DBASIC=$<TARGET_FILE:basic>
-DCASE=${CMAKE_CURRENT_SOURCE_DIR}/tests/reference/${_case}
-P ${CMAKE_CURRENT_SOURCE_DIR}/tests/golden.cmake
)
endforeach()
if(AKBASIC_GOLDEN_CASES)
set(AKBASIC_GOLDEN_NAMES)
foreach(_case IN LISTS AKBASIC_GOLDEN_CASES)
string(REGEX REPLACE "^tests/" "" _name "${_case}")
string(REGEX REPLACE "\\.bas$" "" _name "${_name}")
string(REPLACE "/" "_" _name "${_name}")
list(APPEND AKBASIC_GOLDEN_NAMES golden_${_name})
endforeach()
_set_tests_properties(${AKBASIC_GOLDEN_NAMES} PROPERTIES TIMEOUT 30)
# An AKGL build of `basic` opens a window, and forty-one of them is not what
# anybody running the suite wanted. The dummy driver produces the same stdout,
# which is the only thing a golden case compares.
if(AKBASIC_WITH_AKGL)
_set_tests_properties(${AKBASIC_GOLDEN_NAMES} PROPERTIES
ENVIRONMENT "SDL_VIDEODRIVER=dummy;SDL_AUDIODRIVER=dummy;SDL_RENDER_DRIVER=software")
endif()
endif()
# The local golden corpus, for verbs the reference never implemented.
# #
# Still separate now that the reference's corpus lives in this repository too, # Programs carried over from the deprecated Go implementation and cases written
# and the reason changed rather than went away: tests/reference/ is a *record* of # for this interpreter use the same editable `.bas`/`.txt` contract. Registered
# what the Go implementation did and nothing in it should ever be edited to suit # under local_ so every failure identifies the program that produced it.
# this one, while tests/language/ is ours to change. Registered under local_ so a
# failure says at a glance which of the two it came from -- and so a diff that
# touches tests/reference/ stands out as the thing it is.
# #
# Note what this can and cannot cover. The graphics and sound verbs draw and play # Note what this can and cannot cover. The graphics and sound verbs draw and play
# rather than print, so what a golden file sees of them is their *refusals* and # rather than print, so what a golden file sees of them is their refusals and
# whatever a program can PRINT about the state they changed. The behaviour that # whatever a program can PRINT about the state they changed. The behaviour that
# reaches a device is asserted against tests/mockdevice.h instead. # reaches a device is asserted against tests/mockdevice.h instead.
file(GLOB_RECURSE AKBASIC_LOCAL_CASES file(GLOB_RECURSE AKBASIC_LOCAL_CASES

View File

@@ -88,7 +88,7 @@ source stays readable as documentation of what the original did. Neither is bind
**It is not a build or test dependency.** Both configurations have been configured, built and **It is not a build or test dependency.** Both configurations have been configured, built and
run from scratch with it moved out of the tree. Its acceptance corpus is checked in at run from scratch with it moved out of the tree. Its acceptance corpus is checked in at
`tests/reference/` and its Commodore font at `assets/fonts/`. `tests/language/` and its Commodore font at `assets/fonts/`.
```sh norun ```sh norun
cd deps/basicinterpret cd deps/basicinterpret
@@ -351,15 +351,10 @@ name. That is not cosmetic: `add_executable` creates a dependency's targets even
### The golden corpora ### The golden corpora
`tests/reference/` is the Go implementation's own acceptance suite, byte-compared. `tests/language/` is the editable language corpus. It includes cases carried over from the
**Nothing in it is ever edited to suit this interpreter.** If a case fails, either this deprecated Go implementation as well as cases written for this interpreter. Every `.bas` file
interpreter is wrong or the divergence is deliberate — and a deliberate one goes in has a sibling `.txt` expectation, and a new language feature needs that pair as well as unit
`tests/reference/README.md`'s divergence table and `docs/13-differences.md`, not into the tests. Change both deliberately in the same commit; provenance does not make a case immutable.
expectation file. `tests/reference/README.md`
says the same thing at more length.
`tests/language/` is ours and may be changed freely. A new language feature needs a
`.bas`/`.txt` pair there as well as unit tests.
### Mutation-check a fix before you believe it ### Mutation-check a fix before you believe it

View File

@@ -10,7 +10,7 @@ implementation that started from the Java Lox instructions in
[craftinginterpreters.com](https://craftinginterpreters.com) and then struck off on its own. That [craftinginterpreters.com](https://craftinginterpreters.com) and then struck off on its own. That
project is deprecated. It is vendored here as the behavioural spec to read when a question about project is deprecated. It is vendored here as the behavioural spec to read when a question about
semantics comes up, and its acceptance corpus is checked in at semantics comes up, and its acceptance corpus is checked in at
[`tests/reference/`](tests/reference/README.md) and runs on every build — so nothing about [`tests/language/`](tests/language/README.md) and runs on every build — so nothing about
building or testing this project needs it. building or testing this project needs it.
## Quickstart ## Quickstart
@@ -24,7 +24,7 @@ ctest --test-dir build --output-on-failure
```sh norun ```sh norun
./build/basic # the REPL ./build/basic # the REPL
./build/basic tests/reference/language/functions.bas # run a program ./build/basic tests/language/functions.bas # run a program
``` ```
```basic ```basic
@@ -129,7 +129,7 @@ version are catalogued in [`TODO.md`](TODO.md) and summarised for a BASIC progra
| [`docs/`](docs/README.md) | The guide: eighteen chapters, the language then each hardware area then a reference section for every verb and function, [Chapter 14](docs/14-architecture.md) on the interpreter's own architecture, [Chapter 15](docs/15-error-codes.md) listing every error code, and [Chapters 17](docs/17-tutorial-breakout.md) and [18](docs/18-tutorial-breakout-artwork.md) building a whole game twice | | [`docs/`](docs/README.md) | The guide: eighteen chapters, the language then each hardware area then a reference section for every verb and function, [Chapter 14](docs/14-architecture.md) on the interpreter's own architecture, [Chapter 15](docs/15-error-codes.md) listing every error code, and [Chapters 17](docs/17-tutorial-breakout.md) and [18](docs/18-tutorial-breakout-artwork.md) building a whole game twice |
| [`MAINTENANCE.md`](MAINTENANCE.md) | For contributors and maintainers: the documentation-example harness, the three test lists, mutation testing, error-code allocation, style | | [`MAINTENANCE.md`](MAINTENANCE.md) | For contributors and maintainers: the documentation-example harness, the three test lists, mutation testing, error-code allocation, style |
| [`TODO.md`](TODO.md) | Outstanding defects, with file, line and consequence | | [`TODO.md`](TODO.md) | Outstanding defects, with file, line and consequence |
| [`tests/reference/README.md`](tests/reference/README.md) | Where the golden corpus came from, and the rule for changing it | | [`tests/language/README.md`](tests/language/README.md) | The editable language corpus and the rule for changing it |
API documentation builds with `doxygen Doxyfile`, into `build/docs/html`. API documentation builds with `doxygen Doxyfile`, into `build/docs/html`.

37
TODO.md
View File

@@ -38,7 +38,7 @@ What it changes:
- **§1.8's message-text contract is now a convention.** Improving a message is allowed; it costs - **§1.8's message-text contract is now a convention.** Improving a message is allowed; it costs
a golden file, which is a cost rather than a veto. a golden file, which is a cost rather than a veto.
- **§5's bar drops** from "defensible against the golden suite" to defensible on its own merits. - **§5's bar drops** from "defensible against the golden suite" to defensible on its own merits.
- **`tests/reference/` becomes a regression suite rather than a specification.** Diverging from - **`tests/language/` is the editable language corpus rather than a protected specification.** Diverging from
it is allowed and must be deliberate and recorded — see its README. it is allowed and must be deliberate and recorded — see its README.
What it does **not** change: What it does **not** change:
@@ -306,13 +306,13 @@ to `Println`, which adds another.
**This used to be a hard contract and is now a default.** The Go implementation is deprecated **This used to be a hard contract and is now a default.** The Go implementation is deprecated
and will not be updated, so the two projects are no longer required to match — see §0.1. What and will not be updated, so the two projects are no longer required to match — see §0.1. What
survives is the practical half: these strings and this newline behaviour are what every survives is the practical half: these strings and this newline behaviour are what every
expectation in `tests/reference/` was written against, so changing one means changing golden expectation in `tests/language/` was written against, so changing one means changing the paired
files, and that is worth doing on purpose rather than by accident. A message that reads files, and that is worth doing on purpose rather than by accident. A message that reads
awkwardly *may* now be improved; do it deliberately, move the expectations in the same commit, awkwardly *may* now be improved; do it deliberately, move the expectations in the same commit,
and add a line to §5. and add a line to §5.
Numeric formatting still matches the reference: integers via `%" PRId64 "`, floats via `%f` Numeric formatting still matches the reference: integers via `%" PRId64 "`, floats via `%f`
(Go's `%f` and C's `%f` both give six decimals — `tests/reference/language/arithmetic/float.txt` (Go's `%f` and C's `%f` both give six decimals — `tests/language/arithmetic/float.txt`
confirms). No reason to change it, which is different from not being allowed to. confirms). No reason to change it, which is different from not being allowed to.
### 1.9 Which `libakstdlib` calls are cleared for use — **the bans are lifted** ### 1.9 Which `libakstdlib` calls are cleared for use — **the bans are lifted**
@@ -375,7 +375,7 @@ Phases 0 through 6 of the original plan are done. The interpreter builds clean u
It *did* reproduce the reference byte for byte, and that claim is retired rather than broken: It *did* reproduce the reference byte for byte, and that claim is retired rather than broken:
§0.1 released it, and one case has since diverged deliberately (§6 item 16, listed in §0.1 released it, and one case has since diverged deliberately (§6 item 16, listed in
`tests/reference/README.md`). Everything else still matches, which is worth knowing but is no `tests/language/README.md`). Everything else still matches, which is worth knowing but is no
longer a gate. longer a gate.
```sh ```sh
@@ -408,17 +408,16 @@ and the only path that existed — `AKBASIC_MODE_RUNSTREAM` reading through the
is the code the README quotes, built by every build and registered as a CTest case so a is the code the README quotes, built by every build and registered as a CTest case so a
signature change breaks the build rather than rotting the document. signature change breaks the build rather than rotting the document.
**The acceptance suite is the reference's own corpus, checked in at `tests/reference/`.** All **The acceptance suite is the editable language corpus, checked in at `tests/language/`.** All
41 `.bas` files are registered as individual CTest cases and byte-compared against their `.txt` 65 `.bas` files are registered as individual CTest cases and compared against their `.txt`
— including the trailing double newline on an error line (§1.8). — including the trailing double newline on an error line (§1.8).
It was driven *in place* out of `deps/basicinterpret` until 2026-07-31, on the reasoning that It was driven *in place* out of `deps/basicinterpret` until 2026-07-31, on the reasoning that
copying a submodule's corpus guarantees drift. That reasoning was sound and was overruled copying a submodule's corpus guarantees drift. That reasoning was sound and was overruled
deliberately: the Go dependency is being deprecated, and a build that cannot run its own deliberately: the Go dependency is being deprecated, and a build that cannot run its own
acceptance suite without cloning the implementation it replaced is not finished. The copy is acceptance suite without cloning the implementation it replaced is not finished. The copy is
byte-identical to `basicinterpreter@d76162c`, and `tests/reference/README.md` records the originally byte-identical to `basicinterpreter@d76162c`, and `tests/language/README.md` records the
provenance, the cost of the drift nobody is watching for now, and the rule that those provenance and the rule that programs and expectations are edited together deliberately.
expectations are never edited to suit this interpreter.
**Nothing in the build or the suite needs `deps/basicinterpret` any more**, and that is checked **Nothing in the build or the suite needs `deps/basicinterpret` any more**, and that is checked
rather than assumed — both configurations were configured, built and run from scratch with the rather than assumed — both configurations were configured, built and run from scratch with the
@@ -1026,7 +1025,7 @@ deviations from the reference's *program*: `main.go` and the SDL half of
**What it costs a program:** a listing that used `INPUT$`, `LEN#` or `GOTO%` as a variable **What it costs a program:** a listing that used `INPUT$`, `LEN#` or `GOTO%` as a variable
stops parsing, and the fix is to rename the variable. One case in the reference's own corpus stops parsing, and the fix is to rename the variable. One case in the reference's own corpus
did exactly that; see `tests/reference/README.md`. did exactly that; see `tests/language/examples/strreverse.bas`.
### Deviations in statement separation ### Deviations in statement separation
@@ -1391,10 +1390,10 @@ deviations from the reference's *program*: `main.go` and the SDL half of
**This one moved a golden file.** A line with no number used to be filed under the loader's **This one moved a golden file.** A line with no number used to be filed under the loader's
cursor unchanged — that is, on top of the line before it — so two unnumbered lines in a row cursor unchanged — that is, on top of the line before it — so two unnumbered lines in a row
silently lost the first, and a *blank* line erased whatever preceded it. The reference does silently lost the first, and a *blank* line erased whatever preceded it. The reference does
the same, and `tests/reference/language/arithmetic/integer.bas` is the proof: four `PRINT` the same, and `tests/language/arithmetic/integer.bas` is the proof: four `PRINT`
statements, an expectation with three values, and a trailing blank line that erased statements, an expectation with three values, and a trailing blank line that erased
`40 PRINT 4 - 2` before the program ran. The expectation is now `4 4 2 2` and `40 PRINT 4 - 2` before the program ran. The expectation is now `4 4 2 2` and
`tests/reference/README.md` records it. `tests/language/README.md` records it.
In its place: `akbasic_runtime_file_line()` (`src/runtime.c`) is the one implementation of In its place: `akbasic_runtime_file_line()` (`src/runtime.c`) is the one implementation of
the rule, shared by `akbasic_runtime_load()`, RUNSTREAM and `DLOAD`. A numbered line is the rule, shared by `akbasic_runtime_load()`, RUNSTREAM and `DLOAD`. A numbered line is
@@ -1586,9 +1585,9 @@ be reproduced before it can be fixed.
**It cost one golden case, exactly as predicted, and the cost turned out to be nothing.** **It cost one golden case, exactly as predicted, and the cost turned out to be nothing.**
The reference's `examples/strreverse.bas` names a variable `INPUT$`; the variable is renamed The reference's `examples/strreverse.bas` names a variable `INPUT$`; the variable is renamed
to `SOURCE$` in `tests/reference/`, the expectation is byte-for-byte unchanged — the program to `SOURCE$` in `tests/language/examples/strreverse.bas`, the expectation is byte-for-byte unchanged — the program
still prints `REVERSED: OLLEH` — and the case keeps every bit of its coverage. Recorded in still prints `REVERSED: OLLEH` — and the case keeps every bit of its coverage. Recorded in
`tests/reference/README.md`'s divergence table, which this is the first entry in. `tests/language/README.md` records the corpus editing rule.
This item sat parked because the corpus was the acceptance contract and lived in a submodule This item sat parked because the corpus was the acceptance contract and lived in a submodule
this repository could not edit. Both premises are gone: the corpus is checked in, and this repository could not edit. Both premises are gone: the corpus is checked in, and
@@ -2232,10 +2231,10 @@ update --init --recursive` gets them.
## 8. Status ## 8. Status
**The port is done.** The C interpreter passes the Go reference's entire corpus and passes clean **The port is done.** The C interpreter passes the language corpus and passes clean
under ASan and UBSan. It reproduced that corpus byte for byte until §0.1 retired the under ASan and UBSan. It reproduced that corpus byte for byte until §0.1 retired the
requirement; two cases have diverged on purpose since, and requirement; two cases have diverged on purpose since, and
`tests/reference/README.md` lists them. `tests/language/README.md` lists them.
| Gate | Result | | Gate | Result |
|---|---| |---|---|
@@ -2243,7 +2242,7 @@ requirement; two cases have diverged on purpose since, and
| `ctest` with `-DAKBASIC_WITH_AKGL=ON` | 112/112 headless, with `akgl_typing` skipping itself. The same set minus the four `no_device` cases the SDL driver contradicts, plus `akgl_backends`, `akgl_frontend`, `docs_screenshots` and `akgl_typing` — the last of which is the skip, and the `akgl_build` CI job is where it skips | | `ctest` with `-DAKBASIC_WITH_AKGL=ON` | 112/112 headless, with `akgl_typing` skipping itself. The same set minus the four `no_device` cases the SDL driver contradicts, plus `akgl_backends`, `akgl_frontend`, `docs_screenshots` and `akgl_typing` — the last of which is the skip, and the `akgl_build` CI job is where it skips |
| `docs_examples` | Every fenced block in `README.md`, `MAINTENANCE.md` and `docs/` executed and byte-compared: 71 programs, 9 transcripts, 79 output comparisons, 4 C snippets, 2 excerpts, 2 shell blocks and 19 figures in the default build. The C-snippet count reads 0 when the harness is run by hand without `--cflags-file`; CTest passes it. `MAINTENANCE.md` documents the fence-tag convention | | `docs_examples` | Every fenced block in `README.md`, `MAINTENANCE.md` and `docs/` executed and byte-compared: 71 programs, 9 transcripts, 79 output comparisons, 4 C snippets, 2 excerpts, 2 shell blocks and 19 figures in the default build. The C-snippet count reads 0 when the harness is run by hand without `--cflags-file`; CTest passes it. `MAINTENANCE.md` documents the fence-tag convention |
| `docs_screenshots` | 19/19 figures re-rendered and byte-identical to the checked-in PNGs. AKGL build only — rendering a picture needs the SDL half | | `docs_screenshots` | 19/19 figures re-rendered and byte-identical to the checked-in PNGs. AKGL build only — rendering a picture needs the SDL half |
| Golden corpus | 41/41 byte-exact from `tests/reference/` — **and 41/41 again through the SDL binary**, which is most of what proves the frontend changes no output | | Language corpus | 65/65 paired expectations — **and 65/65 again through the SDL binary**, which is most of what proves the frontend changes no output |
| ASan + UBSan | 112/112 | | ASan + UBSan | 112/112 |
| Line coverage | 94.1% (7227/7681) — above the 90% gate | | Line coverage | 94.1% (7227/7681) — above the 90% gate |
| Function coverage | 97.9% (474/484) | | Function coverage | 97.9% (474/484) |
@@ -2553,7 +2552,7 @@ What remains, in priority order:
this whole arrangement exists against. this whole arrangement exists against.
The byte comparison is a deliberate bet that the dummy driver and the software renderer The byte comparison is a deliberate bet that the dummy driver and the software renderer
are reproducible, which is the same bet `tests/reference/` already makes about golden are reproducible, which is the same bet `tests/language/` makes about golden
output. Verified run-to-run and build-to-build here; what is untested is an SDL upgrade output. Verified run-to-run and build-to-build here; what is untested is an SDL upgrade
that moves one pixel of a diagonal. **If that happens, regenerate the figures in the same that moves one pixel of a diagonal. **If that happens, regenerate the figures in the same
commit as the bump** — do not weaken the test to a size check, which would pass for every commit as the bump** — do not weaken the test to a size check, which would pass for every
@@ -2636,7 +2635,7 @@ reduced against `build/basic`, the stdio build, unless it says otherwise.
**It is narrower than it looks, and the golden corpus is why.** The first attempt **It is narrower than it looks, and the golden corpus is why.** The first attempt
released the scope on *any* skip, which broke released the scope on *any* skip, which broke
`tests/reference/language/flowcontrol/nestedforloopwaitingforcommand.bas`: a zero- `tests/language/flowcontrol/nestedforloopwaitingforcommand.bas`: a zero-
iteration `FOR` skips its body by the same mechanism, and there the orphan is iteration `FOR` skips its body by the same mechanism, and there the orphan is
load-bearing -- it is what absorbs the inner `NEXT` so the outer one still finds its load-bearing -- it is what absorbs the inner `NEXT` so the outer one still finds its
`FOR`. Releasing it turns that case into "NEXT outside the context of FOR". So the `FOR`. Releasing it turns that case into "NEXT outside the context of FOR". So the

View File

@@ -743,8 +743,8 @@ way to see that something pushed a scope and never popped it.
```sh norun ```sh norun
ctest --test-dir build --output-on-failure -R for_next # one unit test ctest --test-dir build --output-on-failure -R for_next # one unit test
ctest --test-dir build --output-on-failure -R golden_ # the reference corpus ctest --test-dir build --output-on-failure -R local_ # the language corpus
./build/basic tests/reference/language/functions.bas | diff - tests/reference/language/functions.txt ./build/basic tests/language/functions.bas | diff - tests/language/functions.txt
./tests/docs_examples.sh --root . --basic ./build/basic \ ./tests/docs_examples.sh --root . --basic ./build/basic \
--cflags-file build/docs_cflags.txt docs/14-architecture.md --cflags-file build/docs_cflags.txt docs/14-architecture.md
``` ```

View File

@@ -282,9 +282,8 @@ def copy_tree(src, dst):
# and a coverage tree drags along every .gcno/.gcda as well. # and a coverage tree drags along every .gcno/.gcda as well.
# #
# deps/ is copied wholesale and has to be: the build pulls libakerror and # deps/ is copied wholesale and has to be: the build pulls libakerror and
# libakstdlib in with add_subdirectory. The golden corpus used to be the # libakstdlib in with add_subdirectory. The language corpus is the other
# other reason -- it was driven in place out of deps/basicinterpret -- and # reason it comes along with the rest of the tree.
# now lives in tests/reference/, which comes along with the rest of the tree.
# Only vendored Windows DLLs are dead weight, hence "*.dll". # Only vendored Windows DLLs are dead weight, hence "*.dll".
# #
# **PNGs are ours or theirs, and the difference matters.** This used to drop # **PNGs are ours or theirs, and the difference matters.** This used to drop

View File

@@ -893,7 +893,7 @@ akerr_ErrorContext *akbasic_runtime_interpret(akbasic_Runtime *obj, akbasic_ASTL
* *
* **Only a block skip**, which is what the `BEND` test is for. A * **Only a block skip**, which is what the `BEND` test is for. A
* zero-iteration `FOR` skips its body the same way, and there the * zero-iteration `FOR` skips its body the same way, and there the
* orphan is load-bearing: `tests/reference/.../nestedforloopwaiting * orphan is load-bearing: `tests/language/.../nestedforloopwaiting
* forcommand.bas` nests a loop inside one that runs zero times, and * forcommand.bas` nests a loop inside one that runs zero times, and
* the inner scope is what absorbs the inner `NEXT` so the outer `NEXT` * the inner scope is what absorbs the inner `NEXT` so the outer `NEXT`
* still finds its `FOR`. Popping it there turns that case into "NEXT * still finds its `FOR`. Popping it there turns that case into "NEXT
@@ -1262,7 +1262,7 @@ akerr_ErrorContext *akbasic_runtime_process_line_runstream(akbasic_Runtime *obj)
* This mode used to file it like any other, under the cursor -- which for a * This mode used to file it like any other, under the cursor -- which for a
* blank line is the number of the line *before* it. A file ending in a blank * blank line is the number of the line *before* it. A file ending in a blank
* line therefore had its last line erased before it ever ran, silently. The * line therefore had its last line erased before it ever ran, silently. The
* reference did the same, and `tests/reference/language/arithmetic/integer.bas` * reference did the same, and `tests/language/arithmetic/integer.bas`
* has an expectation with three values for four PRINT statements to prove it. * has an expectation with three values for four PRINT statements to prove it.
*/ */
if ( buffer[0] == '\0' ) { if ( buffer[0] == '\0' ) {

View File

@@ -81,7 +81,7 @@ static void test_counter_survives_the_loop(void)
* `FOR I = 1 TO 1` executes the body one time on every BASIC there has ever * `FOR I = 1 TO 1` executes the body one time on every BASIC there has ever
* been. Here the entry test treats "the counter has reached the limit" as * been. Here the entry test treats "the counter has reached the limit" as
* "do not enter", so the body is skipped entirely -- and * "do not enter", so the body is skipped entirely -- and
* tests/reference/language/flowcontrol/forloopwaitingforcommand.bas pins that, * tests/language/flowcontrol/forloopwaitingforcommand.bas pins that,
* which is why this cannot simply be corrected. * which is why this cannot simply be corrected.
*/ */
static void test_single_iteration_loop(void) static void test_single_iteration_loop(void)

16
tests/language/README.md Normal file
View File

@@ -0,0 +1,16 @@
# The language test corpus
The `.bas` files in this directory and their sibling `.txt` files are the editable language
corpus. CMake registers every program as an individual `local_*` CTest case and compares its
output with the sibling expectation.
The corpus includes cases carried over from the deprecated Go implementation as well as cases
written for this interpreter. Their provenance is useful when investigating a regression, but it
does not make any case immutable or turn the old implementation into a specification.
**Change a program and its expectation deliberately and in the same commit.** If the changed
output represents an intentional language decision, record the reason in `TODO.md` or the
relevant documentation. If it is an accidental change, fix the interpreter instead.
The `examples/` subdirectory contains complete BASIC programs used as worked examples. It is
part of the same corpus and follows the same `.bas`/`.txt` pairing rule.

View File

@@ -1,63 +0,0 @@
# The reference interpreter's own test corpus
These 41 `.bas` files and their 41 `.txt` expectations are **not this project's tests.** They
are a byte-for-byte copy of the acceptance corpus belonging to the Go implementation this
interpreter was ported from:
| | |
|---|---|
| Source | `https://source.starfort.tech/andrew/basicinterpreter`, `tests/` |
| Copied at | commit `d76162cb37eeaccddae2cf6cab7af7bc8175d6d2` ("Update README.md") |
| Copied on | 2026-07-31 |
| Verified | every file `cmp`-identical to the submodule at the time of the copy |
They are registered as the `golden_*` CTest cases and run against `basic` on every build, in
both the default and the `-DAKBASIC_WITH_AKGL=ON` configurations.
## Why they are a copy, when they deliberately were not
Until 2026-07-31 these were driven *in place* out of `deps/basicinterpret`, and the reasoning
was written down in `CMakeLists.txt`: the corpus is a submodule, and copying a submodule's
corpus guarantees drift.
That reasoning was sound and it stopped applying. The Go project is deprecated and will not be
updated, so there is nothing left to drift *from* — and a build that cannot run its own
acceptance suite without cloning the implementation it replaced is not finished.
## What these are for now
**A regression suite, not a specification.** That distinction is the whole of how to treat
them, and it changed on 2026-07-31 (`TODO.md` §0.1): matching the Go implementation is no
longer a goal, so a difference between it and this interpreter is not automatically a bug in
this one.
What they are still good for is unchanged and is worth a lot: forty-one real BASIC programs
with known-good output, covering arithmetic, arrays, flow control, every builtin function,
`READ`/`DATA`, labels and three worked examples. That catches regressions whatever its
provenance.
## The rule for editing these
**Diverge deliberately, never by accident.**
A failure here means this interpreter's behaviour changed. Nine times in ten that is a bug you
just introduced and the answer is to fix the code. The tenth time it is an improvement on the
reference, and then the answer is:
1. Change the expectation in the same commit as the code.
2. Say why in the commit message.
3. Add a line to `TODO.md` §5, which is the list of deliberate deviations.
What is not acceptable is editing a `.txt` to make a red suite green without deciding which of
the two you are doing.
New cases for this project's own behaviour go in `tests/language/`, registered under `local_*`.
The prefix says which corpus a failure came from, and a diff touching `tests/reference/` stands
out as the thing it is.
## Divergences so far
| Case | Change | Why |
|---|---|---|
| `language/arithmetic/integer.txt` | Expectation grew from `4 4 2` to `4 4 2 2` | The program has **four** `PRINT` statements and the expectation had three values. The file ends in a blank line, and both interpreters filed a blank line under the loader's cursor — which for a line with no number of its own is the number of the line before it. So `40 PRINT 4 - 2` was erased before the program ran. The reference lost it the same way, which is why the expectation was written short. Fixed by skipping blank lines in RUNSTREAM, as `akbasic_runtime_load()` and `DLOAD` already did. `TODO.md` §5. |
| `examples/strreverse.bas` | Variable `INPUT$` renamed to `SOURCE$` | `INPUT$` is a reserved word with a type suffix, which a real C128 refuses and which this interpreter now refuses too (`TODO.md` §6 item 16). The reference accepted it only because its own reserved-word check never fired. The expectation is byte-for-byte unchanged — the program still prints `REVERSED: OLLEH` — so the case keeps all of its value. |

View File

@@ -358,7 +358,7 @@ static void test_skipped_block_releases_its_loop_scope(void)
* inner `NEXT`, so the outer `NEXT` still finds its own `FOR`. Releasing it * inner `NEXT`, so the outer `NEXT` still finds its own `FOR`. Releasing it
* turns this into "NEXT outside the context of FOR". * turns this into "NEXT outside the context of FOR".
* *
* tests/reference/language/flowcontrol/nestedforloopwaitingforcommand.bas is the * tests/language/flowcontrol/nestedforloopwaitingforcommand.bas is the
* golden case that caught exactly that, and this is the same shape asserted * golden case that caught exactly that, and this is the same shape asserted
* where a reader of this file will see it. * where a reader of this file will see it.
*/ */

View File

@@ -182,7 +182,7 @@ static void test_runstream_assigns_the_same_slots(void)
/** /**
* @brief A file ending in a blank line keeps its last line. * @brief A file ending in a blank line keeps its last line.
* *
* `tests/reference/language/arithmetic/integer.bas` is this shape, and both this * `tests/language/arithmetic/integer.bas` is this shape, and both this
* interpreter and the reference used to erase the last line before running it -- * interpreter and the reference used to erase the last line before running it --
* the blank line was filed under the cursor, which for a line with no number of * the blank line was filed under the cursor, which for a line with no number of
* its own is the number of the line before it. * its own is the number of the line before it.