diff --git a/CLAUDE.md b/CLAUDE.md index 23c69bf..0447fb0 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -25,7 +25,7 @@ scripting engine for game authors. | [`MAINTENANCE.md`](MAINTENANCE.md) | Everything above. Start here | | [`TODO.md`](TODO.md) | Outstanding defects, with file, line and consequence. §0.1 first — it retires the byte-for-byte fidelity constraint several later sections were written on | | [`README.md`](README.md) | What the project is and why, for somebody who has not seen it | -| [`docs/`](docs/README.md) | The language itself: thirteen chapters, verb and function reference | +| [`docs/`](docs/README.md) | The language itself: fourteen chapters, verb and function reference. [Chapter 14](docs/14-architecture.md) is the interpreter's architecture — the step loop, the pools, the two kinds of error, and how to debug it | | `deps/libakerror/AGENTS.md` | The `ATTEMPT`/`CLEANUP`/`PROCESS`/`HANDLE`/`FINISH` protocol, authoritatively | | `deps/libakerror/UPGRADING.md` | 1.0.0's status registry. Required before writing an error code | | `deps//AGENTS.md` | Per-repo rules. Read the relevant one **before editing a submodule** | diff --git a/MAINTENANCE.md b/MAINTENANCE.md index c81ad10..046f03e 100644 --- a/MAINTENANCE.md +++ b/MAINTENANCE.md @@ -8,6 +8,11 @@ enforced by nothing at all and therefore written down here. `CLAUDE.md` is for agents, and it is deliberately thin: it points here. Everything an agent is told about this project is in this file, so there is one copy to keep true. +One thing is deliberately *not* here. **How the interpreter actually works — the step loop, +the four modes, the pools, the scopes that double as block state, the two kinds of error, +and how to debug all of it — is [`docs/14-architecture.md`](docs/14-architecture.md).** This +file is the conventions for changing the thing; that chapter is the thing. + --- ## What this project is @@ -570,9 +575,16 @@ a missing decision rather than a free pass. | `sh setup=NAME` | The same, after `tests/docs_setups/NAME.sh` | | `sh norun` | Shown, not run | | `cmake` | **Never executed.** Hand-maintained, by decision | +| `text` | **Never executed.** A block diagram, or a stack trace captured from a real run | Attributes combine: `basic requires=akgl setup=ship` is a real tag in `docs/08-sprites.md`. +`text` exists because `docs/14-architecture.md` needed block diagrams and an untagged block +is a hard error. It carries no executable claim, which is exactly why it has to be *said*: +the alternative was an indented code block that the extractor never sees, and a picture +nobody decided about is indistinguishable from a test nobody ran. Keep the diagrams inside +about 88 columns so they do not wrap on a forge. + ### Which one to reach for - **A program with visible output** — `basic` plus an `output` block. Preferred: it is the @@ -589,6 +601,8 @@ Attributes combine: `basic requires=akgl setup=ship` is a real tag in `docs/08-s `output` block. Refusals carry a trailing blank line; the block has to have it too. - **`norun`** — for a fragment (`60 DATA 255, 129, ...`), an infinite loop, or a command that would reconfigure the tree the suite is running inside. Say why in the prose. +- **A diagram or a captured trace** — `text`. Nothing is checked, so anything in one that + *could* be checked belongs in a block that is. ### `sh` blocks: what is not run, and why @@ -648,6 +662,11 @@ Run one document at a time while you work: --cflags-file build/docs_cflags.txt docs/04-control-flow.md ``` +`--root` and `--basic` are made absolute by the script before it starts, because both are +used from inside a sandbox directory it `cd`s into. Relative ones used to fail every +example with `exited 127` and every `setup=` with "setup failed" — invisible under CTest, +which passes absolute paths, and immediate for anyone running one document by hand. + The pass line reports **what it executed, by kind**. Read it. A harness that passes because it stopped matching anything looks exactly like a harness that passes because the documentation is correct, and the count is the only thing that tells them apart — it has diff --git a/README.md b/README.md index acd9e42..eb7238e 100644 --- a/README.md +++ b/README.md @@ -126,7 +126,7 @@ version are catalogued in [`TODO.md`](TODO.md) and summarised for a BASIC progra | | | |---|---| -| [`docs/`](docs/README.md) | The guide: thirteen chapters, the language then each hardware area then a reference section for every verb and function | +| [`docs/`](docs/README.md) | The guide: fourteen chapters, the language then each hardware area then a reference section for every verb and function, and [Chapter 14](docs/14-architecture.md) on the interpreter's own architecture | | [`MAINTENANCE.md`](MAINTENANCE.md) | For contributors and maintainers: the documentation-example harness, the three test lists, mutation testing, error-code allocation, style | | [`TODO.md`](TODO.md) | Outstanding defects, with file, line and consequence | | [`tests/reference/README.md`](tests/reference/README.md) | Where the golden corpus came from, and the rule for changing it | diff --git a/TODO.md b/TODO.md index b352ef0..68abf40 100644 --- a/TODO.md +++ b/TODO.md @@ -1558,6 +1558,34 @@ fix is reverted: Neither was on any list. Both were found by writing down what a chapter claimed and then running it — which is the argument for the harness rather than a footnote to it. +**A third is open, and it is the same defect as item 2 on a path that fix did not cover.** +Writing `docs/14-architecture.md` meant describing where the boundary between a script's +error and the host's error actually sits, and the answer is: around parsing and around +`interpret()`, but **not around scanning**. `akbasic_runtime_process_line_repl()` and +`akbasic_runtime_process_line_run()` both call `akbasic_scanner_scan()` under a bare `PASS` +(`src/runtime.c:825` and `:921`), so any error the scanner raises leaves `step()` as an +interpreter error. The reachable one is the token ceiling: + +```sh norun +$ printf 'PRINT 1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1\n' | ./build/basic +src/scanner.c:add_token:87: 515 (Out Of Bounds) : Line 0 has more than 32 tokens +... eight more frames ... +akbasic terminated on an unhandled error 515 (Out Of Bounds) +``` + +Exit status 1, a stack trace on stderr, and the prompt gone — for a line a person typed. An +embedding host is handed a context for what is plainly the script's mistake, which is what +goal 3 exists to prevent. It should print `? N : PARSE ERROR Line N has more than 32 tokens` +and stop the run, exactly as a parse error does two lines further down. + +The fix is the same `ATTEMPT`/`HANDLE_DEFAULT`/`akbasic_runtime_error()` wrapper already +sitting around `akbasic_parser_parse()` in both functions, moved up to cover the scan. +Three call sites to check, because `process_line_runstream()` has the same bare `PASS` and +its answer may want to differ: a bad line arriving from a file mid-load is not the same +situation as one typed at a prompt. Wants a test in `tests/scanner_tokens.c` asserting the +error line rather than the raised status, and one in `tests/housekeeping_verbs.c` asserting +the REPL survives it — the shape item 2's test already has. + Coverage of the verb groups added for goal 2, since they are the new surface: `src/audio_tables.c` and `src/graphics_tables.c` 100%, `src/runtime_input.c` 100%, `src/runtime_graphics.c` and `src/runtime_audio.c` 98%, `src/play.c` 87%. The `akbasic_akgl` target is not in the coverage diff --git a/docs/14-architecture.md b/docs/14-architecture.md new file mode 100644 index 0000000..99cbc10 --- /dev/null +++ b/docs/14-architecture.md @@ -0,0 +1,711 @@ +# 14. Architecture + +This chapter is for whoever has to understand the interpreter from the inside: to +embed it, to debug something it did, or to change it. Everything before this chapter +describes the language. This one describes the machine that runs it. + +It is deliberately not a repeat of the API. **The headers under `include/akbasic/` are +the authority on every function's contract** — `doxygen Doxyfile` renders them, and each +one carries its parameters, its return, and the errors it raises. Where a rule is +subtle, the header comment states it at more length than a chapter should. What is here +instead is the shape: what talks to what, in what order, and why it was built that way. + +| Question | Read | +|---|---| +| What does this function do? | the header, or the Doxygen it renders to | +| What does this verb do? | [Chapter 11](11-verb-reference.md), and `src/runtime_*.c` | +| Why is it built like this? | this chapter | +| What are the conventions for changing it? | [`MAINTENANCE.md`](../MAINTENANCE.md) | +| What is currently wrong with it? | [`TODO.md`](../TODO.md) | + +## Three libraries and a driver + +The build separates the interpreter from everything that could own a screen, and the +separation is in the build graph rather than in a comment. + +```text + +---------------------------+ +---------------------------+ + | basic (src/main.c) | | your game | + | argv, QUIT, | | window, renderer, | + | FINISH_NORETURN | | frame loop | + +-------------+-------------+ +-------------+-------------+ + | | + +-------------v-------------+ | + | akbasic_frontend | | a host that already + | SDL window, font, event | | has a renderer skips + | pump, 256 steps a frame | | this target entirely + +-------------+-------------+ | + | | + +-------------v---------------------------------v-------------+ + | akbasic_akgl sink | graphics | audio | input | sprites | + | function-pointer records that draw and play | + | through a renderer somebody else created | + +------------------------------+------------------------------+ + | + +------------------------------v------------------------------+ + | akbasic scanner -> parser -> evaluator, the pools, | + | the verb table, values, environments | + | no SDL, no malloc, no exit(), no globals | + +------------------------------+------------------------------+ + | + +------------------------------v------------------------------+ + | libakstdlib / libakerror | + +-------------------------------------------------------------+ +``` + +| Target | What it is | A game links it? | +|---|---|---| +| `akbasic` | The interpreter. No SDL, nothing that terminates the process | Always | +| `akbasic_akgl` | The sink and the four device backends, drawing through *your* renderer | If you want them | +| `akbasic_frontend` | The standalone program's host: creates the window, owns the loop | **No.** You are the host | +| `basic` | The driver: argv, `QUIT`, and the one `FINISH_NORETURN` in the tree | — | + +Where a thing lives: + +| Area | Source | +|---|---| +| Tokens, AST leaves | `src/grammar.c`, `include/akbasic/grammar.h` | +| Scanner | `src/scanner.c` | +| Parser | `src/parser.c`, and the verbs with their own syntax in `src/parser_commands.c` | +| The step loop, evaluation, pools | `src/runtime.c` | +| Verb and function implementations | `src/runtime_*.c`, one file per verb group | +| The dispatch table | `src/verbs.c` | +| Scopes and block state | `src/environment.c` | +| Values, variables, symbol table | `src/value.c`, `src/variable.c`, `src/symtab.c` | +| Text sinks | `src/sink_stdio.c`, `src/sink_tee.c`, `src/sink_akgl.c` | +| Device backends | `src/graphics_akgl.c`, `src/audio_akgl.c`, `src/input_akgl.c`, `src/sprite_akgl.c` | + +## One struct holds everything + +There is no file-scope mutable state anywhere in the library. Every pool, every cursor +and every piece of subsystem state hangs off one `akbasic_Runtime`, which the caller +owns and passes by pointer. Two interpreters in one process do not interfere, which is +the whole reason for the rule. + +It is several megabytes, because the pools are inline. Put it in static storage or in +your own state — never on a default stack. `src/main.c`, `tests/harness.h` and both +examples all do the same thing for the same reason. + +`akbasic_Runtime` carries five kinds of thing, and it is worth knowing which is which +before you go reading it: + +| Kind | Fields | +|---|---| +| The program | `source[]`, indexed **by line number**, and `sourcepath` | +| Pools | `environments[]`, `variables[]`, `functions[]`, `valuepool` | +| Execution state | `mode`, `run_finished_mode`, `environment`, `errclass`, `skiprestofline`, `trace`, `stopped` | +| Borrowed devices | `sink`, `graphics`, `audio`, `input`, `sprites` — any of the last four may be `NULL` | +| Per-subsystem BASIC state | `gfx`, `audio_state`, `input_state`, `sprite_state`, `format_state`, `console_state`, `data_state`, `disk_state` | + +That last row is the one people ask about. `COLOR`, `TEMPO`, `SCALE`, the `PUDEF` +characters and the eight sprites' positions live on the *runtime*, not on the device +that renders them, because they are the program's state and not the hardware's. A host +that swaps one renderer for another does not expect the script's colours to go with it, +and `RSPPOS` answers correctly with no sprite device attached at all. + +`source[]` being indexed by line number is worth pausing on. A line's number *is* its +array slot, so `GOTO 500` is one assignment — `environment->nextline = 500` — and no +search. Blank slots are skipped by the step loop. That is why line numbers are capped at +9998 and why a nine-line program still costs a 9999-entry array. + +## One step + +`akbasic_runtime_step()` is the whole loop, unrolled to a single iteration. The +reference's `run()` did not return until the program quit; here the caller owns the loop +and the library owns exactly one turn of it. + +```text + akbasic_runtime_step(rt) + | + +-- akbasic_play_service() release the next queued note if its time is up + +-- akbasic_sprite_service() advance continuous MOVSPR motion + +-- akbasic_collision_service() look for collisions where the sprites now are + | + +-- mode == QUIT ? ------------------------------------> return + | + +-- akbasic_input_service() GETKEY waiting? -----> return (no line runs) + +-- akbasic_console_update_clock() + +-- akbasic_console_service() SLEEP or WAIT holding? -> return + | + +-- akbasic_runtime_zero() reset the per-line value pool + +-- akbasic_scanner_zero() reset the scanner cursor + | + +-- switch ( mode ) + | RUNSTREAM -> process_line_runstream() read a line, file it + | REPL -> process_line_repl() read a line, run it or file it + | RUN -> service_interrupts() enter a handler if one is due + | process_line_run() run source[nextline] + | + +-- errclass set ? -> set_mode(run_finished_mode) +``` + +Three things in that order are deliberate. + +**The service calls run before the `QUIT` check.** A program's last notes still come out +while a host keeps calling `step()` after the script has ended. + +**They run before the blocking checks, too.** A program sitting in `GETKEY` still has its +music paced and its sprites moved. `GETKEY` holding the program means *this step +executes no source line* — it does not mean the step does not return. It always returns. + +**Interrupts are serviced between lines and nowhere else.** A handler entered mid-statement +would have to return into the middle of a line, and the parser keeps no state that could +resume there. + +`akbasic_runtime_run(rt, n)` is `step()` in a `while` with a budget: at most `n` steps, +then return regardless. That bound is the only thing standing between a script containing +`10 GOTO 10` and your frame rate. `n <= 0` means unbounded, which is right for a test and +wrong for a game. The SDL frontend uses 256; the stdio driver uses 1, because it wants to +refresh the clock between steps. + +Time comes in from outside. `akbasic_runtime_settime(rt, ms)` is how the interpreter +learns what time it is; it reads no clock, because it owns no loop and must not block. A +host that never calls it leaves time frozen at zero, and every duration then expires on +the step after it starts — audible, but never a hang. That is the intended way for it to +fail. + +## Four modes + +```text + akbasic_runtime_start(rt, mode) + | + +-------------------+-------------------+ + | | + v v + RUNSTREAM REPL <---------------+ + read a line, file it, read a line: file it if it | + execute nothing has a number, run it if not | + | | | + | end of input RUN, CONT | | + +---------------> RUN <---------------+ | + | | + | END, STOP, a BASIC error, or | + | running off the end of the source | + v | + run_finished_mode ---------- == REPL ----------+ + | + | == QUIT, or the QUIT verb, or + v end of input at the prompt + QUIT +``` + +| Mode | What one step does | How it leaves | +|---|---|---| +| `AKBASIC_MODE_REPL` | Read one line from the sink. With a line number, file it; without one, **run it now** | `QUIT` verb, or end of input | +| `AKBASIC_MODE_RUN` | Execute `source[nextline]` | Running off the end, `END`, `STOP`, or a BASIC error | +| `AKBASIC_MODE_RUNSTREAM` | Read one line from the sink and file it. Executes nothing | End of input, which switches to `RUN` | +| `AKBASIC_MODE_QUIT` | Nothing | — | + +`run_finished_mode` decides where a finished program lands. `akbasic_runtime_start(rt, +AKBASIC_MODE_REPL)` sets it to `REPL`, so a program that ends drops back to a prompt; +anything else sets it to `QUIT`, so `basic program.bas` exits. It is one field, and it is +the whole difference between an interactive session and a script runner. + +**`akbasic_runtime_set_mode()` is not just an assignment.** Entering `REPL` prints +`READY`. Entering `RUN` does two prescans of the whole program first: + +- **Labels.** Every `LABEL` in the source is filed before anything executes, textually + rather than by parsing. Without it a label would exist only from the moment its `LABEL` + statement ran, so `GOTO` could reach backwards and never forwards — and an interrupt + handler, which by definition sits where normal flow does not fall, could not be named + by label at all. +- **`DATA` items.** `READ` walks a cursor along a list built before the program runs, so a + `DATA` line above its `READ` is found. + +Every path into a run — `akbasic_runtime_start()`, the `RUN` verb, `CONT`, and the end of +a `RUNSTREAM` load — goes through that one function, which is why the prescans live there +and not in any of the four callers. + +The REPL's split between *file it* and *run it now* is one boolean: the scanner sets +`hadlinenumber` when the line it just read began with a number. A line typed with a +number is program text; a line typed without one is direct mode and runs immediately. + +## From a line of text to an effect + +Nothing is compiled and nothing is cached. Every time a line executes it is scanned and +parsed again, from the source text, into per-line pools that are reset on the way in. + +```text + source[42] "IF A# = 5 THEN PRINT \"FIVE\"" + | + | akbasic_scanner_scan() verb names via the dispatch table; + v no keyword maps of its own + environment->tokens[32] COMMAND(IF) IDENT_INT(A#) ASSIGNMENT LITERAL_INT(5) + | COMMAND(THEN) COMMAND(PRINT) LITERAL_STRING(FIVE) + | + | akbasic_parser_parse() one statement per call; the caller + v loops until the tokens are spent + environment->leaves[32] BRANCH + | / \ + | (= A# 5) COMMAND PRINT + | | + | LITERAL_STRING "FIVE" + | + | akbasic_runtime_interpret() suppressed while this scope is + v skipping forward to a verb + akbasic_runtime_evaluate() switch on leaf type; a COMMAND leaf looks its + | handler up in the table and calls it + v + verb->exec() -> the sink, a device backend, or a variable +``` + +The statement loop lives in `process_line_run()` and `process_line_repl()`, not in the +parser: a line can hold several statements separated by colons, so the caller loops on +`akbasic_parser_is_at_end()`. **`akbasic_parser_parse()` may return `NULL` on success** — +that is a line that was nothing but separators — and a caller that does not skip it will +hand `NULL` to the interpreter. + +`skiprestofline` is how a branch tells that loop to stop. BASIC 7.0 scopes everything +after `THEN` to the condition, but the parser takes only *one* statement per arm and the +rest arrive at the statement loop as ordinary top-level statements. So the branch raises a +flag and the loop obeys it. The rule is not "skip when false": the remainder belongs to +whichever arm was written last, so it is skipped exactly when *that* arm is the one not +taken. [Chapter 4](04-control-flow.md) has the truth table; the `AKBASIC_LEAF_BRANCH` case +in `akbasic_runtime_evaluate()` has the code. + +Two details in that diagram catch people out: + +**A lone `=` is scanned as `ASSIGNMENT`, not as equality.** The scanner cannot know +whether it is looking at a statement or a condition. The parser can: `akbasic_Parser` +carries a `comparing` flag, set around a condition and cleared afterwards, and while it is +set the relation rule accepts `ASSIGNMENT` as a seventh operator and rewrites it to +`EQUAL`. Outside a condition `=` has to stay an assignment or `FOR I# = 1 TO 5` never +initialises its counter. + +**The token and leaf numbering is the Go reference's, on purpose.** `grammar.h` keeps the +original values so a debugging session against either implementation reads the same. + +`akbasic_leaf_to_string()` renders a tree in prefix form — `(+ A# 42)`, `(group (+ A# 42))` +— which is the fastest way to see what the parser actually built. It is a plain function, +so you can call it from a debugger against any leaf pointer you have. + +## The dispatch table + +The Go reference resolved a verb by reflection: `MethodByName("Command" + NAME)`. C has +no reflection and none is being added. All three of its lookups — verbs, functions, and +verbs with their own parse path — collapse into one sorted table in `src/verbs.c`. + +```c excerpt=include/akbasic/verbs.h +typedef struct +{ + const char *name; + akbasic_TokenType tokentype; + int arity; + akbasic_ParseHandler parse; + akbasic_ExecHandler exec; +} akbasic_Verb; +``` + +One row per name, and both handlers are optional: + +| Field | Meaning | +|---|---| +| `tokentype` | What the **scanner** gives this name. It has no keyword maps of its own | +| `arity` | Argument count for a function; `-1` where it does not apply | +| `parse == NULL` | The verb's rval parses as a plain expression | +| `parse != NULL` | The verb has its own syntax — `src/parser_commands.c` | +| `exec == NULL` | The token is consumed by another verb's parser and never evaluated alone: `THEN`, `ELSE`, `TO`, `STEP` | + +`AKBASIC_TOK_COMMAND_IMMEDIATE` marks a verb the REPL may run against a line that *does* +carry a number — `RUN`, `LIST`, `NEW`, `DLOAD`. Everything else typed with a number is +filed as program text. + +**The table is searched with `bsearch`, so it must stay sorted.** A mis-sorted row does +not fail to compile; it silently becomes an unfindable verb, and the symptom is +`Unknown command PRINT` a long way from the cause. `tests/verbs_table.c` asserts the +ordering, which is the only reason that is a safe thing to say. + +## Nothing is allocated + +Every object comes from a fixed pool inside the runtime. Exhausting one is a diagnosable +error naming the pool, not a crash and not a slow leak. + +| Pool | Size | Lives on | Exhaustion says | +|---|---|---|---| +| `AKBASIC_MAX_SOURCE_LINES` | 9999 | runtime | `Line number N is outside 0..9998` | +| `AKBASIC_MAX_ENVIRONMENTS` | 32 | runtime | `Environment pool exhausted at line N (32 in use)` | +| `AKBASIC_MAX_VARIABLES` | 128 | runtime | `Maximum runtime variables reached` | +| `AKBASIC_MAX_FUNCTIONS` | 64 | runtime | `Maximum function definitions reached` | +| `AKBASIC_MAX_ARRAY_VALUES` | 4096 | runtime (`valuepool`) | `Array of N elements does not fit in the M remaining value slots` | +| `AKBASIC_MAX_TOKENS` | 32 | **environment** | `Line N has more than 32 tokens` | +| `AKBASIC_MAX_LEAVES` | 32 | **environment** | `No more leaves available` | +| `AKBASIC_MAX_VALUES` | 64 | **environment** | `Maximum values per line reached` | + +The numbers are in `include/akbasic/types.h`, transcribed from the reference's `main.go` +plus three the Go version did not need because it called `make()`. +[Chapter 13](13-differences.md) states the same budget from a BASIC programmer's side. + +The per-environment three are reset at the top of every line, which is what makes +"roughly 16 operations per line" a real limit and not a leak: a line that uses 30 leaves +is fine, and a line that needs 33 tokens is refused. Because they are per environment +rather than per runtime, a `FOR` body running inside a pushed scope gets its own 32. + +`valuepool` is a bump allocator for array storage and it does not release. Nothing in +BASIC destroys a variable, so there is nothing to release *to* — but re-`DIM`ming an array +larger takes fresh slots and abandons the old ones, so a program that does that in a loop +will exhaust the pool. Bounded and diagnosable, which is the point. + +## A scope is also the block state + +`akbasic_Environment` does two jobs at once, and the second one is the surprising one. It +is a variable scope, chained to its parent — and it is the in-flight state of whatever +block structure is executing: the `FOR` bounds and step, the `DO`/`LOOP` condition, the +`GOSUB` return line, the `READ` cursor, and the line counters. + +| What pushes a scope | What pops it | +|---|---| +| `FOR` — **during parsing**, not execution | the `NEXT` whose condition is met, or an `EXIT` | +| `GOSUB` | `RETURN` | +| A call to a multi-line user function | that function's `RETURN` | +| An interrupt firing | the handler's `RETURN` | + +That `FOR` entry is not a typo. `akbasic_parse_for()` pushes the new environment while +parsing the line, parks `TO` and `STEP` in it as unevaluated leaves, and makes it active +only after parsing is finished — because the loop body is scanned against the *parent's* +token stream. If you are chasing a scope that appeared earlier than you expected, that is +why. + +### `waitingForCommand`, and why loops work at all + +A loop's condition is evaluated at the *bottom* of the structure, which leaves an obvious +problem: how does a zero-iteration `FOR` avoid running its body once? The answer is a +string on the environment. + +```text + 10 FOR I# = 5 TO 1 FOR pushes a scope, evaluates the condition, + 20 PRINT "NEVER" finds it already met, and calls + 30 NEXT I# wait_for_command(env, "NEXT") + + step: line 20 interpret() sees the scope is waiting, and the leaf is not + COMMAND "NEXT" -> evaluates nothing, returns static true + step: line 30 the leaf *is* COMMAND "NEXT" -> runs. NEXT stops the wait, + pops the scope, and hands nextline back to the parent +``` + +`akbasic_runtime_interpret()` is where that is enforced, and it is the first thing it +does. Five places arm a wait, and between them they are the whole of block structure: + +| Verb | Waits for | Because | +|---|---|---| +| `FOR` | `NEXT` | the condition was already met — zero iterations | +| `DO WHILE` / `DO UNTIL` | `LOOP` | the same, at the top of a `DO` | +| `EXIT` | `NEXT` or `LOOP` | where the loop *ends* is not known until a `NEXT` has run once | +| `IF ... THEN BEGIN` | `BEND` | the arm not taken has to skip *lines*, and `skiprestofline` only reaches the end of this one | +| a multi-line `DEF` | `RETURN` | the definition must not execute its own body | + +Three consequences follow, and all three are things people report as bugs: + +- **Block skipping walks source *lines*.** A whole `FOR ... NEXT` written on one line does + not loop, because there is no next line for the wait to skip to. +- **A `FOR` counter does not survive its loop.** It lives in the loop's own scope. On a + C128 it keeps its final value. +- **A host must use `akbasic_runtime_global()`.** `akbasic_environment_get()` creates in + whatever scope is active, and a script suspended part-way through a bounded + `akbasic_runtime_run()` is usually inside a `FOR` or `GOSUB` body — so the script reads + the value correctly inside the loop and gets `0` immediately after it, with nothing + raised anywhere. + +## Values + +`akbasic_Value` carries its string **inline**, not behind a pointer, so a copy is a struct +assignment with no allocator, no refcount and no lifetime question. It costs 256 bytes per +value; that is the trade, and it is recorded in `TODO.md`. + +Type comes from the identifier's suffix and nothing else: `A#` integer, `A%` float, `A$` +string, and a bare name with no suffix is a *label*. That is `akbasic_leaf_identifier_type()`, +and it is why `ER` and `EL` are spelled `ER#` and `EL#` here. + +One field on the runtime changes how identifiers evaluate. `eval_clone_identifiers` is +normally true, and evaluating `A#` then yields a clone drawn from the per-line value pool. +`POKE` and `POINTER` need the address of the real storage, so they clear it around their +own evaluation. If you are debugging an assignment that wrote to the wrong place, that +flag is the first thing to check. + +## Errors come in two kinds, and the distinction is the point + +```text + a script's mistake the interpreter's mistake + ----------------- ------------------------- + PRINT 1/0, GOTO nowhere, pool exhausted, NULL argument, + type mismatch, a refused verb a sink that failed to write + | | + v v + report_and_reraise() propagates out through + writes "? 20 : RUNTIME ..." to the sink akbasic_runtime_step() + | | + v v + swallowed by process_line_run() handed to the host, which + or process_line_repl(); the run stops, decides what to do. The + the host is never told driver prints a stack trace + and exits 1 +``` + +A script's mistakes are the script's problem and your program keeps running. That is goal +3, and it is easier to get wrong than it looks: the direct-mode branch of +`process_line_repl()` once used a bare `PASS`, so a `VERIFY` against a file that did not +match — an ordinary user answer — tore down the driver with a stack trace. + +The BASIC-visible line is `? : `, where the class is one of `IO`, +`PARSE`, `SYNTAX` or `RUNTIME`. The message is expected to end in a newline and the sink +adds another, **so an error line is followed by a blank one**. That is the contract, not an +accident, and the golden corpus depends on it. + +`errclass` being set is what ends a run: `step()` sees it and switches to +`run_finished_mode`. Returning to a prompt clears it; quitting does not. + +**`akbasic_runtime_error()` is the single choke point** — the one place a BASIC-visible +error is reported *and* the one place a run is stopped. That is not an accident of +factoring, it is what makes `TRAP` implementable: an armed trap turns both off in one +place. Nothing is printed, `errclass` stays clear so the step loop carries on, and the +handler is entered at the next line boundary by the same machinery `COLLISION` uses. If +you ever need to intercept an error class, that function is where to stand. + +Interpreter errors carry a status from the `akbasic` band, 512–767, declared as an `enum` +in `include/akbasic/error.h` and named in `akbasic_error_register()`. `MAINTENANCE.md` +carries the coordinated range map across the whole dependency stack, and it is required +reading before adding a code. + +**One gap worth knowing before it surprises you.** The `ATTEMPT` blocks that convert a +script's mistake into an error line are wrapped around *parsing* and *interpretation*, +not around *scanning*. A line with more than 32 tokens therefore escapes as an +interpreter error — the driver prints a stack trace and exits 1, and an embedding host is +handed a context for what is really a script's mistake. It is the same shape as the +`VERIFY` defect above, on a path that fix did not cover. Filed in `TODO.md`; until it is +closed, a host that cannot tolerate that should refuse over-long lines itself. + +## Devices, and how a capability is withheld + +The sink and the four device backends are records of function pointers plus whatever +state you hang off `self`. That is the house pattern for anything that varies, and it is +what keeps SDL out of the core: the whole test suite runs on a machine with no SDL +installed at all. + +```text + akbasic_runtime_set_devices(rt, graphics, audio, input, sprites) + | | | | + any of them may be NULL, and NULL is not + "do nothing" -- it is "refuse by name": + + SOUND with no audio backend -> + "? 10 : RUNTIME ERROR SOUND needs an audio + device and this runtime has none" +``` + +A backend record with a `NULL` *entry point* behaves the same way, one verb at a time — +`CHAR` against a stdio sink that cannot move a cursor refuses rather than printing in the +wrong place. Nothing is ever silently ignored; that is the rule the whole device layer is +built to keep. + +`akbasic_sink_init_tee()` composes two sinks into one, which is how the SDL build puts +`PRINT` in a window *and* on stdout. The second write belongs out here rather than inside +the interpreter — the reference hardcoded it, and that is exactly what made its output +untestable without a display. + +[Chapter 10](10-embedding.md) is the practical side of all of this, with a complete host +loop. + +## Interrupts + +`COLLISION` and `TRAP` arm a slot; a device backend, the collision service, or +`akbasic_runtime_error()` calls `akbasic_runtime_raise_interrupt()`, which only records +that it fired. The handler is entered later, by `step()`, between two source lines. + +Raising is cheap and safe to call every frame whether or not anything is armed — an +unarmed source records nothing — so a host does not have to ask what the script has +subscribed to before reporting a collision. + +Entering one is a `GOSUB` the program did not write: a scope is pushed, its return line is +the line that was about to run, and the handler's `RETURN` pops back to it — so a handler +must end in `RETURN`, exactly as on a C128. + +`handlerenv` does two jobs. It is the "a handler is running" flag, which is what stops a +collision that persists across the handler from recursing until the environment pool is +gone. And it is the identity `RETURN` compares against, so the flag clears on the `RETURN` +that leaves *this* handler rather than on the first `RETURN` of any `GOSUB` the handler +itself makes. A depth counter would have got that wrong. + +## Debugging it + +### What the language gives you + +`TRON` prints each line's number in brackets, inline and with no newline, which is what a +C128 does: + +```basic +10 TRON +20 FOR I# = 1 TO 3 +30 PRINT I# +40 NEXT I# +50 TROFF +60 PRINT "DONE" +``` + +```output +[20][30]1 +[40][30]2 +[40][30]3 +[40][50]DONE +``` + +Read that trace as evidence about the *loop*, not just about the lines: `[40][30]` says +`NEXT` sent control back to 30, and the missing `[20]` on later passes says the `FOR` line +is not re-entered. A trace that shows a line you expected to be skipped is a +`waitingForCommand` question; a trace that shows the right lines with the wrong output is +an evaluation question. + +`HELP` re-displays the last error line, and that is the whole of what it does. + +### Reading a stack trace + +An interpreter-level error that nothing handled reaches the driver, which prints the +`akerror` trace on **stderr** and exits 1. Every frame is `file:function:line`: + +```text +src/scanner.c:add_token:87: 515 (Out Of Bounds) : Line 10 has more than 32 tokens +src/scanner.c:akbasic_scanner_scan:414 +src/runtime.c:akbasic_runtime_process_line_runstream:788 +src/runtime.c:akbasic_runtime_step:1302 +src/runtime.c:akbasic_runtime_run:1367 +src/main.c:drive:94 +src/main.c:run_stdio:130 +src/main.c:main:203 +src/main.c:main:214: akbasic terminated on an unhandled error 515 (Out Of Bounds) +``` + +The paths are `__FILE__` as your build spells it, so an out-of-tree build prints them +absolute. The top frame is where it was raised and the bottom is where it was reported. +The number is the status code — 512 and up is ours, and `include/akbasic/error.h` names +them. A code printing as `Unknown Error` means somebody added it to the `enum` and forgot +to name it in `akbasic_error_register()`. + +Note what this trace is *not*: it is not on stdout, so a golden comparison never sees it, +and `? 10 : RUNTIME ERROR ...` is a different thing entirely — that one is the script's +error, on stdout, and the process exits 0. + +### Under a debugger + +Four breakpoints answer most questions: + +| Break on | Answers | +|---|---| +| `akbasic_runtime_step` | Which mode, and what the step decided to do | +| `akbasic_parser_parse` | What the parser built — then render it | +| `akbasic_runtime_evaluate` | Every leaf, in evaluation order. Noisy; condition it on `expr->leaftype` | +| the verb's own `akbasic_cmd_*` | What arguments a verb actually received | + +Useful expressions once you are stopped. `rt` is whatever your host calls the runtime; in +the standalone driver it is the file-static `RUNTIME`, so it is `p RUNTIME.mode` there: + +```text + p rt->mode 1 REPL, 2 RUN, 3 RUNSTREAM, 4 QUIT + p rt->environment->lineno the line executing + p rt->environment->nextline the line that will execute next + p rt->environment->waitingForCommand "" or the verb being skipped forward to + p rt->environment->parent NULL means this is the root scope + p rt->source[42].code the stored text of line 42 + p rt->errclass non-zero means the run is ending + p *rt->environment->forNextVariable the FOR counter, when there is one + + call akbasic_leaf_to_string(leaf, buf, sizeof buf) render a subtree: (+ A# 42) +``` + +Walking `parent` from `rt->environment` gives you the scope stack, and it is the fastest +way to see that something pushed a scope and never popped it. + +### Narrowing it with the suite + +```sh norun +ctest --test-dir build --output-on-failure -R for_next # one unit test +ctest --test-dir build --output-on-failure -R golden_ # the reference corpus +./build/basic tests/reference/language/functions.bas | diff - tests/reference/language/functions.txt +./tests/docs_examples.sh --root . --basic ./build/basic \ + --cflags-file build/docs_cflags.txt docs/14-architecture.md +``` + +The unit tests build a runtime against memory buffers (`tests/harness.h`), so they can +assert on exact output without a terminal. `tests/mockdevice.h` gives the graphics, audio +and input verbs recording backends that log every call — which is how the device verbs are +tested at all, since they produce no stdout for a golden file to compare. + +Two more builds, when the answer is not in the logic: + +```sh norun +cmake -S . -B build-asan -DAKBASIC_SANITIZE=ON # ASan + UBSan +cmake -S . -B build-cov -DAKBASIC_COVERAGE=ON # gcovr, and the 90% gate +``` + +`MAINTENANCE.md` covers the mutation harness and the discipline that goes with a fix: +revert it, confirm the test fails, restore it. + +### Symptom to cause + +| Symptom | Look at | +|---|---| +| `Unknown command PRINT` | The verb table's sort order. `tests/verbs_table.c` | +| `peek() returned nil token!` | The line ended before the parser expected it to — a missing operand or an unbalanced paren | +| A line ran that should have been skipped | `waitingForCommand` on the active scope, and `skiprestofline` | +| A line was skipped that should have run | The same two, in the other direction | +| A variable reads `0` right after a loop | It lived in the loop's scope. Use `akbasic_runtime_global()` from a host | +| `Environment pool exhausted` | A scope pushed and never popped — usually a `GOSUB` with no `RETURN`, or recursion | +| `Line N has more than 32 tokens` | The per-environment token pool. Split the line | +| Every duration expires instantly | The host never called `akbasic_runtime_settime()` | +| A verb refuses with "needs a ... device" | That backend is `NULL`, or the entry point it wanted is | +| The driver exits 1 with a stack trace | An *interpreter* error, not a script error. Read the top frame | + +## Changing it + +### Adding a verb + +1. **One row in `src/verbs.c`**, in sort order. Pick `AKBASIC_TOK_COMMAND`, or + `AKBASIC_TOK_COMMAND_IMMEDIATE` if the REPL should run it against a numbered line. +2. **A parse handler**, only if the syntax is not "the verb and one expression". Put it in + `src/parser_commands.c` and leave the table cell `NULL` otherwise. +3. **An exec handler** in the `src/runtime_*.c` file for its verb group, with the + signature from `include/akbasic/verbs.h` — `(runtime, expr, lval, rval, dest)` — and a + prototype in the *private* `src/verbs.h`, where the table gets its declarations. +4. **A unit test** in `tests/`, registered in `AKBASIC_TESTS` in `CMakeLists.txt`, built as + `akbasic_test_`. +5. **A `.bas`/`.txt` pair** in `tests/language/`. Both kinds of test, because they answer + different questions: one pins the C contract, the other pins what a user sees. +6. **A row in [Chapter 11](11-verb-reference.md)**, and an example somewhere — every fenced + block in `docs/` is executed by `ctest`. + +Adding a *function* is the same, with `AKBASIC_TOK_FUNCTION`, a real `arity`, and +[Chapter 12](12-function-reference.md). + +### Adding a device capability + +Add the entry point to the backend record in the relevant `include/akbasic/*.h`, implement +it in `src/*_akgl.c`, and — this is the part that is easy to skip — **make the verb refuse +by name when the pointer is `NULL`**, so an older host or a different backend gets an +error rather than silence. + +If `libakgl` cannot supply what the verb needs, **do not work around it here**. File it in +`deps/libakgl/TODO.md`: what the BASIC verb requires, what the `akgl_*` entry point should +look like, and what tests would cover it. Four gaps have gone upstream that way and all +four landed. + +### The rules a change has to keep + +These are not style preferences. Each one is load-bearing for goal 3, and each one has a +test or a build failure behind it: + +- **Nothing in the library terminates the process.** `FINISH_NORETURN` belongs only in a + `main()`. +- **Nothing calls `malloc`.** Add a pool, or a layer to one. +- **No file-scope mutable state.** It goes on `akbasic_Runtime`. +- **Nothing blocks.** A verb that waits sets state and returns; `step()` checks it next + time round. `SLEEP`, `WAIT`, `GETKEY` and `PLAY` are all built that way. +- **The interpreter owns no window, renderer or loop.** + +## Where to read next + +`MAINTENANCE.md` is the conventions: the three test lists and why two of them invert +"passed", the documentation-example harness, the error-code map, the build collisions, the +style rules. `TODO.md` is the defect list, with file, line and consequence — including the +ones inherited from the Go reference and the ones found writing these chapters. The +headers under `include/akbasic/` are the API, and `doxygen Doxyfile` renders them into +`build/docs/html`. + +And `examples/embed.c` and `examples/hostvars.c` are two complete hosts, built and run by +every build, so neither can rot. diff --git a/docs/README.md b/docs/README.md index 245950c..16bb9b1 100644 --- a/docs/README.md +++ b/docs/README.md @@ -9,7 +9,9 @@ These chapters follow the shape of the [C128 Programmer's Reference Guide](http://www.jbrain.com/pub/cbm/manuals/128/C128PRG.pdf): the language first, then each hardware area, then the reference sections. If you know BASIC 7.0 you can skip to **[Chapter 13](13-differences.md)**, which is the list of -everything that behaves differently here and why. +everything that behaves differently here and why. **[Chapter 14](14-architecture.md)** is +the odd one out: it is about the interpreter rather than the language, for anyone +embedding it, debugging it or changing it. ## Chapters @@ -28,6 +30,7 @@ everything that behaves differently here and why. | **[11. Verb reference](11-verb-reference.md)** | Every statement, alphabetically | | **[12. Function reference](12-function-reference.md)** | Every function, alphabetically | | **[13. Differences from BASIC 7.0](13-differences.md)** | What a C128 programmer needs to know | +| **[14. Architecture](14-architecture.md)** | How the interpreter is put together, how to debug it, how to change it | ## The shortest possible start diff --git a/tests/docs_examples.sh b/tests/docs_examples.sh index 3c9818f..0bd5bb7 100755 --- a/tests/docs_examples.sh +++ b/tests/docs_examples.sh @@ -33,6 +33,7 @@ # ```sh setup=NAME the same, after tests/docs_setups/NAME.sh # ```sh norun destructive, networked, or re-enters this suite # ```cmake never executed; hand-maintained, by decision +# ```text a diagram or a captured dump; there is nothing to execute # # A block with **no** info string is a hard error. That is deliberate: the # failure mode this whole harness exists to avoid is passing because it quietly @@ -76,7 +77,19 @@ done [ -n "${BASIC}" ] || usage [ -x "${BASIC}" ] || { echo "FAIL: no interpreter at ${BASIC}" >&2; exit 2; } +# Both of these are used from inside a sandbox directory this script cd's into, +# so a relative one silently resolves against the wrong place: `--basic +# ./build/basic` fails every example with "exited 127", and `--root .` cannot +# find tests/docs_setups. CTest passes absolute paths and never saw it; a person +# running one document by hand hits it immediately. Resolve them here instead of +# documenting the trap. +case "${BASIC}" in +/*) ;; +*) BASIC="${PWD}/${BASIC}" ;; +esac + cd "${ROOT}" || exit 2 +ROOT="${PWD}" DOCS=("$@") if [ ${#DOCS[@]} -eq 0 ]; then @@ -116,7 +129,7 @@ CC="${CC:-cc}" FAILURES=0 declare -A RAN=([basic]=0 [repl]=0 [output]=0 [c]=0 [excerpt]=0 [sh]=0) -declare -A SKIPPED=([norun]=0 [akgl]=0 [cmake]=0 [nocc]=0) +declare -A SKIPPED=([norun]=0 [akgl]=0 [cmake]=0 [text]=0 [nocc]=0) # Every failure names the file and line of the block, so the message points at # the thing to edit rather than at this script. @@ -490,6 +503,12 @@ for DOC in "${DOCS[@]}"; do cmake) SKIPPED[cmake]=$((SKIPPED[cmake] + 1)) ;; + # A block diagram, or a stack trace captured from a real run. There is no + # executable claim in either, so the tag exists to say that out loud rather + # than to let one sit here untagged and unnoticed. + text) + SKIPPED[text]=$((SKIPPED[text] + 1)) + ;; "") fail "${WHERE}" "fenced block with no info string; tag it (see MAINTENANCE.md)" ;; @@ -504,7 +523,7 @@ done # a chapter stopped matching the extractor looks exactly like a harness that # passes because the documentation is correct, and this is what tells them apart. echo "ran: ${RAN[basic]} programs, ${RAN[repl]} transcripts, ${RAN[output]} output comparisons, ${RAN[c]} C snippets, ${RAN[excerpt]} excerpts, ${RAN[sh]} shell blocks" -echo "skipped: ${SKIPPED[norun]} norun, ${SKIPPED[akgl]} needing akgl, ${SKIPPED[cmake]} cmake, ${SKIPPED[nocc]} C (no compiler flags given)" +echo "skipped: ${SKIPPED[norun]} norun, ${SKIPPED[akgl]} needing akgl, ${SKIPPED[cmake]} cmake, ${SKIPPED[text]} text, ${SKIPPED[nocc]} C (no compiler flags given)" if [ "${FAILURES}" -eq 0 ]; then echo "PASS: every documented example does what the documentation says"