Cut akbasic_Runtime's static footprint from 10.75 MiB to 2.40 MiB #33

Closed

tachikoma wants to merge 3 commits from feature/reduce_memory_usage into main

pull from: feature/reduce_memory_usage

merge into: andrew:main

andrew:main

andrew:galaga-tutorial

andrew:40

andrew:34

andrew:fix/megademo-80col

andrew:libakstdlib-26

andrew:23

andrew:todo-to-issues

Conversation 1 Commits 3 Files Changed 28 +264 -106

tachikoma commented 2026-08-03 21:47:41 -04:00

Collaborator

Copy Link

Summary

sizeof(akbasic_Runtime) goes from 11,270,976 bytes (10.75 MiB) to
2,512,048 bytes (2.40 MiB) — a 4.49x reduction — by tightening seven
AKBASIC_MAX_*/AKBASIC_SYMTAB_MAX_* pool constants and one field's type,
sized against measured use rather than guessed. Numbers were taken by
compiling a sizeof() probe against the real headers and cross-checked
against nm -S on the actual linked basic binary's RUNTIME/SCRIPT
symbol, not estimated.

Methodology

Nothing in this interpreter mallocs; every pool is a fixed array bounded by
a compile-time constant, so the runtime's memory cost is static and knowable.
Two example programs — examples/breakout/sprites/breakout.bas (1343 lines)
and examples/megademo/megademo.bas (1573 lines) — are the most demanding
BASIC programs in this repository, so they were used as the measuring stick:
static analysis of source structure (line counts, identifier lengths, GOSUB
call-graph depth via a DAG walk) plus dynamic high-water-mark instrumentation
(temporarily patched into env_acquire()/akbasic_runtime_new_variable(),
reverted before committing) run headless under
SDL_VIDEODRIVER=dummy for both programs.

Constant	Before	After	Basis
AKBASIC_MAX_ENVIRONMENTS	32	12	measured peak concurrency 6-7
AKBASIC_MAX_FUNCTIONS	64	8	measured: 0 — neither program uses DEF FN
AKBASIC_MAX_ARRAY_VALUES	4096	2048	measured peak 1618 slots
AKBASIC_MAX_SOURCE_LINES	9999	2048	measured ~1270-1496 non-blank lines
AKBASIC_SYMTAB_MAX_SLOTS	256	172	no caller ever requests capacity > 128
AKBASIC_SYMTAB_MAX_KEY	64	24	longest identifier measured: 11 chars
AKBASIC_MAX_LINE_LENGTH	256	80	Commodore BASIC's own line limit (not a measurement — see Known consequence below)

AKBASIC_MAX_VARIABLES (128) is deliberately untouched: breakout alone
reaches 121 of 128 concurrent named variables, the least headroom of any pool
measured, and shrinking it would be a real risk rather than a free win.

akbasic_Variable.name moves from AKBASIC_MAX_STRING_LENGTH (256) to
AKBASIC_SYMTAB_MAX_KEY. This one isn't a measurement, it's a correctness
fix: every variable name is registered with akbasic_symtab_set()
immediately after this field is populated
(akbasic_environment_create(), src/environment.c), and that call already
refuses anything AKBASIC_SYMTAB_MAX_KEY characters or longer with
AKBASIC_ERR_BOUNDS. No variable with a longer name could ever exist, so the
wider field was 232 bytes per variable (128 variables) nothing could reach.

Two defects this surfaced, both fixed here

sourcepath was borrowing AKBASIC_MAX_LINE_LENGTH by accident. It holds
a directory (for resolving relative asset paths like SPRSAV), not a line
of BASIC — and this checkout's own test paths are 81+ characters deep, which
broke every golden test's ability to even load until this was split into its
own AKBASIC_MAX_SOURCE_PATH_LENGTH, backed by PATH_MAX the way
libakerror already sizes its own path buffers.

src/sink_stdio.c's stdio_readline() didn't check aksl_fgets()'s own
documented contract. A full buffer with no trailing newline means the line
was longer than the buffer and the remainder is still in the stream —
aksl_fgets()'s doc comment says so explicitly. Unchecked, the next
readline() call picks that remainder up as its own statement: a program
doesn't fail to load, it silently becomes a different, wrong program. I
found this by loading breakout.bas (which has an 87-character line) after
the libakstdlib-26 port landed and getting a garbled downstream parse error
instead of a clean bounds error. At the old 256-byte ceiling this was
theoretical. At 80 it is not, so stdio_readline() now refuses loudly
instead of truncating silently — this is also what makes the
AKBASIC_MAX_LINE_LENGTH drop safe rather than just smaller.

tests/value_pool.c's test_pool_is_untouched_by_scopes() was pinned to the
old 4×1024=4096 pool arithmetic (four max-size arrays proving nothing leaked
across 200 GOSUB scopes); rewritten to 2×1024=2048 for the same proof against
the new AKBASIC_MAX_ARRAY_VALUES.

Known consequence — read before merging

AKBASIC_MAX_LINE_LENGTH=80 follows Commodore BASIC's own column limit, not
a measurement, and it breaks real content:

• tests/reference/language/functions/mod.bas:4 and
  tests/reference/language/flowcontrol/nestedforloopwaitingforcommand.bas:3
  are both 82 characters. MAINTENANCE.md and CMakeLists.txt:585 are
  explicit that tests/reference/ is a byte-identical copy of the Go
  reference's own corpus and is never edited to suit this interpreter — these
  two golden cases cannot be fixed by shortening them, and fail outright.
• 12 tests/language/ cases (this project's own local corpus — see the full
  list in andrew/akbasic#32) and one line in
  docs/18-tutorial-breakout-artwork.md are in the same position but are
  content this project owns.
• examples/breakout and examples/megademo (not covered by CI) also have
  lines well over 80 characters and will need rework to load under this
  build.

This was a deliberate, explicit call, not an oversight — I raised the
tests/reference/ conflict before pushing, laid out the alternatives (raise
the limit to clear the corpus with margin, e.g. 96 or 128; rewrite the
editable content and gate tests/reference/ out of the golden-case loop;
or carry a documented exception to the never-edit rule), and was told to ship
80 anyway. Filed as andrew/akbasic#32 with the full failing-test list,
the measured floor (83, with zero margin), and the tradeoffs, rather than
silently worked around here.

Currently failing under this branch: docs_examples,
golden_language_flowcontrol_nestedforloopwaitingforcommand,
golden_language_functions_mod, and 12 local_* cases — 97/112 passing.
Full list and context in #32.

Verification

cmake --build build-akgl -j$(nproc)     # clean, no new warnings
ctest --test-dir build-akgl             # 97/112 — all 15 failures are the AKBASIC_MAX_LINE_LENGTH
                                         # consequence above, tracked in #32
nm -S build-akgl/basic | grep RUNTIME   # 2512048 bytes, matches sizeof() exactly

## Summary `sizeof(akbasic_Runtime)` goes from **11,270,976 bytes (10.75 MiB) to 2,512,048 bytes (2.40 MiB)** — a 4.49x reduction — by tightening seven `AKBASIC_MAX_*`/`AKBASIC_SYMTAB_MAX_*` pool constants and one field's type, sized against measured use rather than guessed. Numbers were taken by compiling a `sizeof()` probe against the real headers and cross-checked against `nm -S` on the actual linked `basic` binary's `RUNTIME`/`SCRIPT` symbol, not estimated. ## Methodology Nothing in this interpreter `malloc`s; every pool is a fixed array bounded by a compile-time constant, so the runtime's memory cost is static and knowable. Two example programs — `examples/breakout/sprites/breakout.bas` (1343 lines) and `examples/megademo/megademo.bas` (1573 lines) — are the most demanding BASIC programs in this repository, so they were used as the measuring stick: static analysis of source structure (line counts, identifier lengths, GOSUB call-graph depth via a DAG walk) plus dynamic high-water-mark instrumentation (temporarily patched into `env_acquire()`/`akbasic_runtime_new_variable()`, reverted before committing) run headless under `SDL_VIDEODRIVER=dummy` for both programs. | Constant | Before | After | Basis | |---|---:|---:|---| | `AKBASIC_MAX_ENVIRONMENTS` | 32 | 12 | measured peak concurrency 6-7 | | `AKBASIC_MAX_FUNCTIONS` | 64 | 8 | measured: 0 — neither program uses `DEF FN` | | `AKBASIC_MAX_ARRAY_VALUES` | 4096 | 2048 | measured peak 1618 slots | | `AKBASIC_MAX_SOURCE_LINES` | 9999 | 2048 | measured ~1270-1496 non-blank lines | | `AKBASIC_SYMTAB_MAX_SLOTS` | 256 | 172 | no caller ever requests capacity > 128 | | `AKBASIC_SYMTAB_MAX_KEY` | 64 | 24 | longest identifier measured: 11 chars | | `AKBASIC_MAX_LINE_LENGTH` | 256 | 80 | Commodore BASIC's own line limit (not a measurement — see **Known consequence** below) | `AKBASIC_MAX_VARIABLES` (128) is **deliberately untouched**: breakout alone reaches 121 of 128 concurrent named variables, the least headroom of any pool measured, and shrinking it would be a real risk rather than a free win. `akbasic_Variable.name` moves from `AKBASIC_MAX_STRING_LENGTH` (256) to `AKBASIC_SYMTAB_MAX_KEY`. This one isn't a measurement, it's a correctness fix: every variable name is registered with `akbasic_symtab_set()` immediately after this field is populated (`akbasic_environment_create()`, `src/environment.c`), and that call already refuses anything `AKBASIC_SYMTAB_MAX_KEY` characters or longer with `AKBASIC_ERR_BOUNDS`. No variable with a longer name could ever exist, so the wider field was 232 bytes per variable (128 variables) nothing could reach. ## Two defects this surfaced, both fixed here **`sourcepath` was borrowing `AKBASIC_MAX_LINE_LENGTH` by accident.** It holds a *directory* (for resolving relative asset paths like `SPRSAV`), not a line of BASIC — and this checkout's own test paths are 81+ characters deep, which broke every golden test's ability to even load until this was split into its own `AKBASIC_MAX_SOURCE_PATH_LENGTH`, backed by `PATH_MAX` the way `libakerror` already sizes its own path buffers. **`src/sink_stdio.c`'s `stdio_readline()` didn't check `aksl_fgets()`'s own documented contract.** A full buffer with no trailing newline means the line was longer than the buffer and the remainder is still in the stream — `aksl_fgets()`'s doc comment says so explicitly. Unchecked, the next `readline()` call picks that remainder up as its own statement: a program doesn't fail to load, it silently becomes a different, wrong program. I found this by loading `breakout.bas` (which has an 87-character line) after the `libakstdlib-26` port landed and getting a garbled downstream parse error instead of a clean bounds error. At the old 256-byte ceiling this was theoretical. At 80 it is not, so `stdio_readline()` now refuses loudly instead of truncating silently — this is also what makes the `AKBASIC_MAX_LINE_LENGTH` drop *safe* rather than just smaller. `tests/value_pool.c`'s `test_pool_is_untouched_by_scopes()` was pinned to the old 4×1024=4096 pool arithmetic (four max-size arrays proving nothing leaked across 200 GOSUB scopes); rewritten to 2×1024=2048 for the same proof against the new `AKBASIC_MAX_ARRAY_VALUES`. ## Known consequence — read before merging `AKBASIC_MAX_LINE_LENGTH=80` follows Commodore BASIC's own column limit, not a measurement, and it **breaks real content**: - **`tests/reference/language/functions/mod.bas:4`** and **`tests/reference/language/flowcontrol/nestedforloopwaitingforcommand.bas:3`** are both 82 characters. `MAINTENANCE.md` and `CMakeLists.txt:585` are explicit that `tests/reference/` is a byte-identical copy of the Go reference's own corpus and is never edited to suit this interpreter — these two golden cases cannot be fixed by shortening them, and fail outright. - 12 `tests/language/` cases (this project's own local corpus — see the full list in andrew/akbasic#32) and one line in `docs/18-tutorial-breakout-artwork.md` are in the same position but are content this project owns. - `examples/breakout` and `examples/megademo` (not covered by CI) also have lines well over 80 characters and will need rework to load under this build. **This was a deliberate, explicit call, not an oversight** — I raised the `tests/reference/` conflict before pushing, laid out the alternatives (raise the limit to clear the corpus with margin, e.g. 96 or 128; rewrite the editable content and gate `tests/reference/` out of the golden-case loop; or carry a documented exception to the never-edit rule), and was told to ship 80 anyway. Filed as **andrew/akbasic#32** with the full failing-test list, the measured floor (83, with zero margin), and the tradeoffs, rather than silently worked around here. **Currently failing under this branch:** `docs_examples`, `golden_language_flowcontrol_nestedforloopwaitingforcommand`, `golden_language_functions_mod`, and 12 `local_*` cases — 97/112 passing. Full list and context in #32. ## Verification ``` cmake --build build-akgl -j$(nproc) # clean, no new warnings ctest --test-dir build-akgl # 97/112 — all 15 failures are the AKBASIC_MAX_LINE_LENGTH # consequence above, tracked in #32 nm -S build-akgl/basic | grep RUNTIME # 2512048 bytes, matches sizeof() exactly ```

tachikoma added 1 commit 2026-08-03 21:47:41 -04:00

Cut akbasic_Runtime's static footprint from 10.75 MiB to 2.40 MiB ... 17af2d406c

Nothing in this interpreter mallocs; every pool is a fixed array sized by an
AKBASIC_MAX_* constant, so sizeof(akbasic_Runtime) is a compile-time number
and most of it was headroom nobody was using. Measured concurrent-use
high-water marks off examples/breakout and examples/megademo -- the two most
demanding programs this interpreter runs -- against each pool's ceiling:

  AKBASIC_MAX_ENVIRONMENTS   32 -> 12    (measured peak concurrency: 6-7)
  AKBASIC_MAX_FUNCTIONS      64 -> 8     (measured: 0, neither program uses DEF FN)
  AKBASIC_MAX_ARRAY_VALUES 4096 -> 2048  (measured peak: 1618 slots)
  AKBASIC_MAX_SOURCE_LINES 9999 -> 2048  (measured: ~1270-1496 non-blank lines)
  AKBASIC_SYMTAB_MAX_SLOTS  256 -> 172   (no caller ever requests more than 128)
  AKBASIC_SYMTAB_MAX_KEY     64 -> 24    (longest identifier measured: 11 chars)
  AKBASIC_MAX_LINE_LENGTH   256 -> 80    (Commodore BASIC's own line limit)

AKBASIC_MAX_VARIABLES (128) is untouched on purpose: breakout alone reaches
121 of 128 concurrent named variables, so it has the least slack of any pool
measured and is not a shrink candidate.

akbasic_Variable.name shrinks from AKBASIC_MAX_STRING_LENGTH (256) to
AKBASIC_SYMTAB_MAX_KEY: every variable name is registered with
akbasic_symtab_set() right after this field is populated
(akbasic_environment_create(), src/environment.c), and that call already
refuses anything AKBASIC_SYMTAB_MAX_KEY characters or longer. The wider field
was headroom nothing could ever put a byte into.

Two defects surfaced while testing the line-length drop against the golden
corpus, both fixed here because the 80-byte ceiling makes them routine rather
than theoretical:

- sourcepath (runtime.h) was borrowing AKBASIC_MAX_LINE_LENGTH by accident.
  It holds a directory, not a line of BASIC, and this checkout's own test
  paths are 81+ characters deep -- every golden test failed to load until
  this split into its own AKBASIC_MAX_SOURCE_PATH_LENGTH, backed by PATH_MAX
  the way libakerror already sizes its own path buffers.

- src/sink_stdio.c's stdio_readline() called aksl_fgets() but never checked
  its own documented contract: a full buffer with no trailing newline means
  the line was longer than the buffer, and the unread remainder is still in
  the stream. Unchecked, the next readline() picks that remainder up as its
  own statement -- a real line silently becomes two wrong ones instead of a
  clean AKBASIC_ERR_BOUNDS refusal. At 256 bytes this was theoretical; at 80
  it is not, so it now refuses loudly.

tests/value_pool.c's test_pool_is_untouched_by_scopes() was pinned to the old
4x1024=4096 pool math (four max-size arrays proving nothing leaked); rewritten
to 2x1024=2048 for the same proof against the new AKBASIC_MAX_ARRAY_VALUES.

Known consequence, tracked in andrew/akbasic#32 rather than worked around
here: two files in the protected tests/reference/ corpus
(language/functions/mod.bas, language/flowcontrol/nestedforloopwaitingfor
command.bas) have 82-character lines and cannot be shortened -- MAINTENANCE.md
and CMakeLists.txt:585 are explicit that tests/reference/ is never edited to
suit this interpreter. Twelve tests/language/ cases and one docs/18 line are
in the same position but are this project's own content. Shipping 80 anyway,
with the fallout tracked rather than hidden, was an explicit call on this PR
rather than something decided here.

Verified: cmake --build build-akgl && ctest --test-dir build-akgl, 97/112 (15
known failures, all AKBASIC_MAX_LINE_LENGTH-related, filed as #32).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Andrew Kesterson <andrew@aklabs.net>

tachikoma requested review from andrew 2026-08-03 21:47:53 -04:00

tachikoma referenced this pull request 2026-08-03 22:58:42 -04:00

Rework the megademo for the 80-column line limit #35

andrew added 2 commits 2026-08-04 09:11:56 -04:00

Rework the megademo to fit the 80-column source line limit ... 01adf80751

AKBASIC_MAX_LINE_LENGTH's cut from 256 to 80 left seventeen lines of
examples/megademo unloadable: the sixteen IM$() picture strings (up to
252 characters) and TUNEA/TUNEB's four-bar PLAY strings (174 and 175).
The real ceiling is 78 characters, not 80 -- stdio_readline() refuses a
read that fills the 80-byte buffer without a newline, so content plus
its terminator must fit in 79.

The picture: vaporwave.py's PAYLOAD drops from 240 to 64, so every
emitted IM$(NN) = "..." line fits under the ceiling. chop() no longer
slices blind; it walks the stream a record at a time -- two characters
for a run, three for an R row record -- and never cuts inside one,
because the decoder reads a record's tail with MID on the string it is
walking and a record straddling two IM$ entries decodes as garbage.
The old blind slice at 240 only happened to be safe. verify() now
simulates the CHOPPED strings with the cursor threaded across the
boundaries exactly the way DRAWSTREAM executes them, so a bad cut is
an assertion failure instead of a corrupted screen, and emit_block()
asserts every emitted line fits. The picture is 56 strings where it
was 16; the decoder needed no changes at all, since it already carries
X#/Y# from one IM$ entry to the next.

The music: TUNEA and TUNEB each become four PLAY statements, one bar
apiece. play.c keeps voice, envelope, level and duration state on the
runtime across statements and every PLAY appends to the same queue, so
four bars queue exactly as one long string did. Each bar restates the
V1T3U9S prefix so a bar dropped by QFULL cannot leave the next batch
playing on the drum kit's envelope.

Everything still clears the shrunken pools with room to spare: 1625
source lines of 2048, ~704 array slots of 2048, identifiers within the
24-character symtab key. Verified end to end against this branch's
build: the demo loads, the offscreen host renders every scene, and the
scene-5 still is pixel-identical to vaporwave.py's own preview. The
test suite fails the same seventeen cases with and without this
commit.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ACffnV6F7sxQuG3Y8a1L3s

Merge pull request 'Rework the megademo for the 80-column line limit' (#35) from fix/megademo-80col into feature/reduce_memory_usage ... 844ebeef22

Reviewed-on: #35

andrew force-pushed feature/reduce_memory_usage from 844ebeef22 to 46cb4549fe 2026-08-04 10:51:07 -04:00 Compare

tachikoma force-pushed feature/reduce_memory_usage from 46cb4549fe to 1e514f679b 2026-08-04 11:13:49 -04:00 Compare

logikoma added 1 commit 2026-08-04 16:14:20 -04:00

Fix 80-column fixtures and tutorial expectations bd63e3e9fa

logikoma commented 2026-08-04 16:14:51 -04:00

Collaborator

Copy Link

Build fix pushed as bd63e3e.

• Reflowed the two authorized tests/reference fixtures, all affected tests/language BASIC comments, and the docs/18 tutorial BASIC line to fit the 80-column source limit.
• Split the docs/18 PRESSPAUSE statement without changing its behavior.
• Updated docs/17-tutorial-breakout.md for the reduced 12-environment pool (the stale expected output was the remaining docs_examples failure).

Validation: core build succeeded; ctest passes 109/109 tests. The AKGL-only job was not run locally because nested SDL dependency cloning stalled, but all affected source lines are below the interpreter limit.

Build fix pushed as bd63e3e. - Reflowed the two authorized tests/reference fixtures, all affected tests/language BASIC comments, and the docs/18 tutorial BASIC line to fit the 80-column source limit. - Split the docs/18 PRESSPAUSE statement without changing its behavior. - Updated docs/17-tutorial-breakout.md for the reduced 12-environment pool (the stale expected output was the remaining docs_examples failure). Validation: core build succeeded; ctest passes 109/109 tests. The AKGL-only job was not run locally because nested SDL dependency cloning stalled, but all affected source lines are below the interpreter limit.

logikoma force-pushed feature/reduce_memory_usage from bd63e3e9fa to eb93bb7da0 2026-08-04 16:37:27 -04:00 Compare

andrew closed this pull request 2026-08-04 16:41:35 -04:00

All checks were successful

Hide all checks

akbasic CI Build / cmake_build (push) Successful in 3m29s

Required

Details

akbasic CI Build / sanitizers (push) Successful in 4m42s

Required

Details

akbasic CI Build / coverage (push) Successful in 3m56s

Required

Details

akbasic CI Build / akgl_build (push) Successful in 8m34s

Required

Details

akbasic CI Build / mutation_test (push) Successful in 18m50s

Required

Details

Pull request closed

Please reopen this pull request to perform a merge.

Sign in to join this conversation.

Reviewers

No Reviewers

andrew

Labels

Clear labels

abi-break

Cannot land without a soname bump

api-gap

A caller cannot express something

blast-radius:high

Ordering: highest blast radius

blast-radius:low

Ordering: lowest blast radius

blast-radius:medium

Ordering: middling blast radius

defect

Wrong behaviour reachable from the public API

design-decision

Needs a decision written down before code

docs

Documentation, comments, AGENTS.md

hygiene

Style, dead code, consistency

packaging

Build, install, pkg-config, CMake config

performance

Has a measurement behind it

status::blocked

Waiting on something else -- a decision, a dependency, or another issue

status::grooming

Not yet worked out: scope, approach or priority still being settled

status::ready

Understood well enough to start: scope and approach are settled

status::review

A pull request has been generated and this ticket is waiting on review of that PR

test-coverage

A gap in the suites, not a defect

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

logikoma tachikoma andrew

No Assignees

3 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: andrew/akbasic#33