3 Commits

Author SHA1 Message Date
d715bc0625 Report prescan errors on their source lines
Some checks failed
akbasic CI Build / coverage (push) Failing after 27s
akbasic CI Build / cmake_build (push) Failing after 33s
akbasic CI Build / akgl_build (push) Failing after 18s
akbasic CI Build / sanitizers (push) Failing after 1m7s
akbasic CI Build / mutation_test (push) Failing after 39s
2026-08-05 06:40:14 -04:00
350deb3a45 Merge pull request 'Move BASIC fixtures into the editable language corpus' (#42) from 40 into main
All checks were successful
akbasic CI Build / cmake_build (push) Successful in 4m2s
akbasic CI Build / sanitizers (push) Successful in 4m59s
akbasic CI Build / coverage (push) Successful in 4m35s
akbasic CI Build / akgl_build (push) Successful in 9m17s
akbasic CI Build / mutation_test (push) Successful in 29m15s
Reviewed-on: #42
2026-08-04 16:30:25 -04:00
8a02674af5 Move BASIC fixtures into the editable language corpus
All checks were successful
akbasic CI Build / cmake_build (push) Successful in 3m34s
akbasic CI Build / coverage (push) Successful in 4m4s
akbasic CI Build / sanitizers (push) Successful in 6m59s
akbasic CI Build / akgl_build (push) Successful in 7m57s
akbasic CI Build / mutation_test (push) Successful in 23m28s
Move every program and expectation out of tests/reference and register the unified tests/language corpus as local cases. Remove the old immutable-corpus protections from build, maintenance, and documentation paths.

Co-authored-by: andrew <andrew@aklabs.net>
2026-08-04 16:22:16 -04:00
128 changed files with 273 additions and 436 deletions

View File

@@ -17,7 +17,7 @@ jobs:
# Not recursive, deliberately. The top-level build needs # Not recursive, deliberately. The top-level build needs
# deps/libakerror and deps/libakstdlib, via add_subdirectory, and # deps/libakerror and deps/libakstdlib, via add_subdirectory, and
# nothing else: the golden corpus and the Commodore font now live in # nothing else: the golden corpus and the Commodore font now live in
# this repository (tests/reference/ and assets/fonts/), so # this repository (tests/language/ and assets/fonts/), so
# deps/basicinterpret is no longer a build dependency at all. # deps/basicinterpret is no longer a build dependency at all.
# It does *not* need deps/libakgl, which is guarded behind # It does *not* need deps/libakgl, which is guarded behind
# AKBASIC_WITH_AKGL and defaults OFF -- and recursing into it would # AKBASIC_WITH_AKGL and defaults OFF -- and recursing into it would
@@ -62,11 +62,9 @@ jobs:
run: | run: |
cmake -S . -B build cmake -S . -B build
cmake --build build --parallel 2 cmake --build build --parallel 2
# The suite is 78 cases: 41 golden files byte-compared against the Go # The suite is 112 cases: 65 language files with sibling expectations,
# reference's own corpus (checked in at tests/reference/, see its README), # 43 unit tests, 3 embedding examples, and docs_examples.
# 9 local golden cases for verbs the reference never implemented, 25 unit # Some unit tests assert the *correct* contract for known defects (TODO.md
# tests, 2 embedding examples, and 1 known-failing test that asserts the
# *correct* contract for defects carried over from the reference (TODO.md
# section 6). A green run therefore does not mean defect-free -- see # section 6). A green run therefore does not mean defect-free -- see
# AKBASIC_KNOWN_FAILING_TESTS. # AKBASIC_KNOWN_FAILING_TESTS.
# #

View File

@@ -57,9 +57,9 @@ repeating where you will see them:
`include/akgl/SDL_GameControllerDB.h`. Change the template or the generator script. `include/akgl/SDL_GameControllerDB.h`. Change the template or the generator script.
- **Do not reformat code you are not otherwise changing.** Several files mix tabs and spaces - **Do not reformat code you are not otherwise changing.** Several files mix tabs and spaces
and there is no repo-wide formatter; style conversions get their own commit. and there is no repo-wide formatter; style conversions get their own commit.
- **Do not edit `tests/reference/`.** Those expectations came from the Go implementation and - **Keep `tests/language/` editable.** Its `.bas` programs and sibling `.txt` expectations
are never edited to suit this interpreter. A deliberate divergence goes in are changed together when behavior changes. Record deliberate language decisions in
`tests/reference/README.md`'s divergence table and `docs/13-differences.md`. `TODO.md` or `docs/13-differences.md`.
- **Open an issue for outstanding work; do not add it to `TODO.md`.** - **Open an issue for outstanding work; do not add it to `TODO.md`.**
<https://source.starfort.tech/andrew/akbasic/issues>, or `tea issues create --repo <https://source.starfort.tech/andrew/akbasic/issues>, or `tea issues create --repo
andrew/akbasic`. Name the file and line, the functional consequence, and what closing it would andrew/akbasic`. Name the file and line, the functional consequence, and what closing it would

View File

@@ -448,7 +448,7 @@ if(AKBASIC_WITH_AKGL)
# against. # against.
# #
# **A byte comparison of a rendered PNG is a deliberate bet**, the same bet # **A byte comparison of a rendered PNG is a deliberate bet**, the same bet
# tests/reference/ already makes about golden output: that the dummy video # the language corpus makes about golden output: that the dummy video
# driver and the software renderer are reproducible. They are, run to run and # driver and the software renderer are reproducible. They are, run to run and
# build to build. What is untested is an SDL upgrade that shifts one pixel of # build to build. What is untested is an SDL upgrade that shifts one pixel of
# a diagonal, and the answer to that is to regenerate the figures in the same # a diagonal, and the answer to that is to regenerate the figures in the same
@@ -531,63 +531,20 @@ if(AKBASIC_WILL_FAIL_TESTS OR AKBASIC_KNOWN_FAILING_TESTS)
) )
endif() endif()
# The reference's own corpus, byte-compared against the sibling .txt. One CTest # The editable language corpus. One CTest case per .bas so a failure names the
# case per .bas so a failure names the file. # file. Each program is paired with a sibling .txt expectation; see
# tests/language/README.md for the editing rule.
# #
# It used to be driven in place out of deps/basicinterpret, on the reasoning that # The corpus includes programs carried over from the deprecated Go implementation
# copying a submodule's corpus guarantees drift. That reasoning was sound and it # and cases written for this interpreter. Their provenance is useful when
# has been overruled deliberately: the Go dependency is being deprecated, and a # investigating a regression, but it does not make any case immutable.
# build that cannot run its acceptance suite without cloning the implementation
# it replaced is not finished. The copy is byte-identical to
# basicinterpreter@d76162c and tests/reference/README.md records
# where it came from and what the drift now costs.
file(GLOB_RECURSE AKBASIC_GOLDEN_CASES
RELATIVE "${CMAKE_CURRENT_SOURCE_DIR}/tests/reference"
"${CMAKE_CURRENT_SOURCE_DIR}/tests/reference/*.bas"
)
foreach(_case IN LISTS AKBASIC_GOLDEN_CASES)
string(REGEX REPLACE "^tests/" "" _name "${_case}")
string(REGEX REPLACE "\\.bas$" "" _name "${_name}")
string(REPLACE "/" "_" _name "${_name}")
_add_test(
NAME golden_${_name}
COMMAND ${CMAKE_COMMAND}
-DBASIC=$<TARGET_FILE:basic>
-DCASE=${CMAKE_CURRENT_SOURCE_DIR}/tests/reference/${_case}
-P ${CMAKE_CURRENT_SOURCE_DIR}/tests/golden.cmake
)
endforeach()
if(AKBASIC_GOLDEN_CASES)
set(AKBASIC_GOLDEN_NAMES)
foreach(_case IN LISTS AKBASIC_GOLDEN_CASES)
string(REGEX REPLACE "^tests/" "" _name "${_case}")
string(REGEX REPLACE "\\.bas$" "" _name "${_name}")
string(REPLACE "/" "_" _name "${_name}")
list(APPEND AKBASIC_GOLDEN_NAMES golden_${_name})
endforeach()
_set_tests_properties(${AKBASIC_GOLDEN_NAMES} PROPERTIES TIMEOUT 30)
# An AKGL build of `basic` opens a window, and forty-one of them is not what
# anybody running the suite wanted. The dummy driver produces the same stdout,
# which is the only thing a golden case compares.
if(AKBASIC_WITH_AKGL)
_set_tests_properties(${AKBASIC_GOLDEN_NAMES} PROPERTIES
ENVIRONMENT "SDL_VIDEODRIVER=dummy;SDL_AUDIODRIVER=dummy;SDL_RENDER_DRIVER=software")
endif()
endif()
# The local golden corpus, for verbs the reference never implemented.
# #
# Still separate now that the reference's corpus lives in this repository too, # Programs carried over from the deprecated Go implementation and cases written
# and the reason changed rather than went away: tests/reference/ is a *record* of # for this interpreter use the same editable `.bas`/`.txt` contract. Registered
# what the Go implementation did and nothing in it should ever be edited to suit # under local_ so every failure identifies the program that produced it.
# this one, while tests/language/ is ours to change. Registered under local_ so a
# failure says at a glance which of the two it came from -- and so a diff that
# touches tests/reference/ stands out as the thing it is.
# #
# Note what this can and cannot cover. The graphics and sound verbs draw and play # Note what this can and cannot cover. The graphics and sound verbs draw and play
# rather than print, so what a golden file sees of them is their *refusals* and # rather than print, so what a golden file sees of them is their refusals and
# whatever a program can PRINT about the state they changed. The behaviour that # whatever a program can PRINT about the state they changed. The behaviour that
# reaches a device is asserted against tests/mockdevice.h instead. # reaches a device is asserted against tests/mockdevice.h instead.
file(GLOB_RECURSE AKBASIC_LOCAL_CASES file(GLOB_RECURSE AKBASIC_LOCAL_CASES

View File

@@ -88,7 +88,7 @@ source stays readable as documentation of what the original did. Neither is bind
**It is not a build or test dependency.** Both configurations have been configured, built and **It is not a build or test dependency.** Both configurations have been configured, built and
run from scratch with it moved out of the tree. Its acceptance corpus is checked in at run from scratch with it moved out of the tree. Its acceptance corpus is checked in at
`tests/reference/` and its Commodore font at `assets/fonts/`. `tests/language/` and its Commodore font at `assets/fonts/`.
```sh norun ```sh norun
cd deps/basicinterpret cd deps/basicinterpret
@@ -351,15 +351,10 @@ name. That is not cosmetic: `add_executable` creates a dependency's targets even
### The golden corpora ### The golden corpora
`tests/reference/` is the Go implementation's own acceptance suite, byte-compared. `tests/language/` is the editable language corpus. It includes cases carried over from the
**Nothing in it is ever edited to suit this interpreter.** If a case fails, either this deprecated Go implementation as well as cases written for this interpreter. Every `.bas` file
interpreter is wrong or the divergence is deliberate — and a deliberate one goes in has a sibling `.txt` expectation, and a new language feature needs that pair as well as unit
`tests/reference/README.md`'s divergence table and `docs/13-differences.md`, not into the tests. Change both deliberately in the same commit; provenance does not make a case immutable.
expectation file. `tests/reference/README.md`
says the same thing at more length.
`tests/language/` is ours and may be changed freely. A new language feature needs a
`.bas`/`.txt` pair there as well as unit tests.
### Mutation-check a fix before you believe it ### Mutation-check a fix before you believe it

View File

@@ -10,7 +10,7 @@ implementation that started from the Java Lox instructions in
[craftinginterpreters.com](https://craftinginterpreters.com) and then struck off on its own. That [craftinginterpreters.com](https://craftinginterpreters.com) and then struck off on its own. That
project is deprecated. It is vendored here as the behavioural spec to read when a question about project is deprecated. It is vendored here as the behavioural spec to read when a question about
semantics comes up, and its acceptance corpus is checked in at semantics comes up, and its acceptance corpus is checked in at
[`tests/reference/`](tests/reference/README.md) and runs on every build — so nothing about [`tests/language/`](tests/language/README.md) and runs on every build — so nothing about
building or testing this project needs it. building or testing this project needs it.
## Quickstart ## Quickstart
@@ -24,7 +24,7 @@ ctest --test-dir build --output-on-failure
```sh norun ```sh norun
./build/basic # the REPL ./build/basic # the REPL
./build/basic tests/reference/language/functions.bas # run a program ./build/basic tests/language/functions.bas # run a program
``` ```
```basic ```basic
@@ -129,7 +129,7 @@ version are catalogued in [`TODO.md`](TODO.md) and summarised for a BASIC progra
| [`docs/`](docs/README.md) | The guide: eighteen chapters, the language then each hardware area then a reference section for every verb and function, [Chapter 14](docs/14-architecture.md) on the interpreter's own architecture, [Chapter 15](docs/15-error-codes.md) listing every error code, and [Chapters 17](docs/17-tutorial-breakout.md) and [18](docs/18-tutorial-breakout-artwork.md) building a whole game twice | | [`docs/`](docs/README.md) | The guide: eighteen chapters, the language then each hardware area then a reference section for every verb and function, [Chapter 14](docs/14-architecture.md) on the interpreter's own architecture, [Chapter 15](docs/15-error-codes.md) listing every error code, and [Chapters 17](docs/17-tutorial-breakout.md) and [18](docs/18-tutorial-breakout-artwork.md) building a whole game twice |
| [`MAINTENANCE.md`](MAINTENANCE.md) | For contributors and maintainers: the documentation-example harness, the three test lists, mutation testing, error-code allocation, style | | [`MAINTENANCE.md`](MAINTENANCE.md) | For contributors and maintainers: the documentation-example harness, the three test lists, mutation testing, error-code allocation, style |
| [`TODO.md`](TODO.md) | Outstanding defects, with file, line and consequence | | [`TODO.md`](TODO.md) | Outstanding defects, with file, line and consequence |
| [`tests/reference/README.md`](tests/reference/README.md) | Where the golden corpus came from, and the rule for changing it | | [`tests/language/README.md`](tests/language/README.md) | The editable language corpus and the rule for changing it |
API documentation builds with `doxygen Doxyfile`, into `build/docs/html`. API documentation builds with `doxygen Doxyfile`, into `build/docs/html`.

37
TODO.md
View File

@@ -38,7 +38,7 @@ What it changes:
- **§1.8's message-text contract is now a convention.** Improving a message is allowed; it costs - **§1.8's message-text contract is now a convention.** Improving a message is allowed; it costs
a golden file, which is a cost rather than a veto. a golden file, which is a cost rather than a veto.
- **§5's bar drops** from "defensible against the golden suite" to defensible on its own merits. - **§5's bar drops** from "defensible against the golden suite" to defensible on its own merits.
- **`tests/reference/` becomes a regression suite rather than a specification.** Diverging from - **`tests/language/` is the editable language corpus rather than a protected specification.** Diverging from
it is allowed and must be deliberate and recorded — see its README. it is allowed and must be deliberate and recorded — see its README.
What it does **not** change: What it does **not** change:
@@ -306,13 +306,13 @@ to `Println`, which adds another.
**This used to be a hard contract and is now a default.** The Go implementation is deprecated **This used to be a hard contract and is now a default.** The Go implementation is deprecated
and will not be updated, so the two projects are no longer required to match — see §0.1. What and will not be updated, so the two projects are no longer required to match — see §0.1. What
survives is the practical half: these strings and this newline behaviour are what every survives is the practical half: these strings and this newline behaviour are what every
expectation in `tests/reference/` was written against, so changing one means changing golden expectation in `tests/language/` was written against, so changing one means changing the paired
files, and that is worth doing on purpose rather than by accident. A message that reads files, and that is worth doing on purpose rather than by accident. A message that reads
awkwardly *may* now be improved; do it deliberately, move the expectations in the same commit, awkwardly *may* now be improved; do it deliberately, move the expectations in the same commit,
and add a line to §5. and add a line to §5.
Numeric formatting still matches the reference: integers via `%" PRId64 "`, floats via `%f` Numeric formatting still matches the reference: integers via `%" PRId64 "`, floats via `%f`
(Go's `%f` and C's `%f` both give six decimals — `tests/reference/language/arithmetic/float.txt` (Go's `%f` and C's `%f` both give six decimals — `tests/language/arithmetic/float.txt`
confirms). No reason to change it, which is different from not being allowed to. confirms). No reason to change it, which is different from not being allowed to.
### 1.9 Which `libakstdlib` calls are cleared for use — **the bans are lifted** ### 1.9 Which `libakstdlib` calls are cleared for use — **the bans are lifted**
@@ -375,7 +375,7 @@ Phases 0 through 6 of the original plan are done. The interpreter builds clean u
It *did* reproduce the reference byte for byte, and that claim is retired rather than broken: It *did* reproduce the reference byte for byte, and that claim is retired rather than broken:
§0.1 released it, and one case has since diverged deliberately (§6 item 16, listed in §0.1 released it, and one case has since diverged deliberately (§6 item 16, listed in
`tests/reference/README.md`). Everything else still matches, which is worth knowing but is no `tests/language/README.md`). Everything else still matches, which is worth knowing but is no
longer a gate. longer a gate.
```sh ```sh
@@ -408,17 +408,16 @@ and the only path that existed — `AKBASIC_MODE_RUNSTREAM` reading through the
is the code the README quotes, built by every build and registered as a CTest case so a is the code the README quotes, built by every build and registered as a CTest case so a
signature change breaks the build rather than rotting the document. signature change breaks the build rather than rotting the document.
**The acceptance suite is the reference's own corpus, checked in at `tests/reference/`.** All **The acceptance suite is the editable language corpus, checked in at `tests/language/`.** All
41 `.bas` files are registered as individual CTest cases and byte-compared against their `.txt` 65 `.bas` files are registered as individual CTest cases and compared against their `.txt`
— including the trailing double newline on an error line (§1.8). — including the trailing double newline on an error line (§1.8).
It was driven *in place* out of `deps/basicinterpret` until 2026-07-31, on the reasoning that It was driven *in place* out of `deps/basicinterpret` until 2026-07-31, on the reasoning that
copying a submodule's corpus guarantees drift. That reasoning was sound and was overruled copying a submodule's corpus guarantees drift. That reasoning was sound and was overruled
deliberately: the Go dependency is being deprecated, and a build that cannot run its own deliberately: the Go dependency is being deprecated, and a build that cannot run its own
acceptance suite without cloning the implementation it replaced is not finished. The copy is acceptance suite without cloning the implementation it replaced is not finished. The copy is
byte-identical to `basicinterpreter@d76162c`, and `tests/reference/README.md` records the originally byte-identical to `basicinterpreter@d76162c`, and `tests/language/README.md` records the
provenance, the cost of the drift nobody is watching for now, and the rule that those provenance and the rule that programs and expectations are edited together deliberately.
expectations are never edited to suit this interpreter.
**Nothing in the build or the suite needs `deps/basicinterpret` any more**, and that is checked **Nothing in the build or the suite needs `deps/basicinterpret` any more**, and that is checked
rather than assumed — both configurations were configured, built and run from scratch with the rather than assumed — both configurations were configured, built and run from scratch with the
@@ -1026,7 +1025,7 @@ deviations from the reference's *program*: `main.go` and the SDL half of
**What it costs a program:** a listing that used `INPUT$`, `LEN#` or `GOTO%` as a variable **What it costs a program:** a listing that used `INPUT$`, `LEN#` or `GOTO%` as a variable
stops parsing, and the fix is to rename the variable. One case in the reference's own corpus stops parsing, and the fix is to rename the variable. One case in the reference's own corpus
did exactly that; see `tests/reference/README.md`. did exactly that; see `tests/language/examples/strreverse.bas`.
### Deviations in statement separation ### Deviations in statement separation