Implement generators (GEN/EMIT/END GEN/FOR EACH/DO EACH) - closes #57 #61

Closed

tachikoma wants to merge 1 commits from 57-implement-generators into main

pull from: 57-implement-generators

merge into: andrew:main

andrew:main

andrew:10

andrew:47-prepush

andrew:36

andrew:46

andrew:deps-update

andrew:56-doxygen-gate-fixes

andrew:16

andrew:47

andrew:galaga-tutorial

andrew:feature/reduce_memory_usage

andrew:40

andrew:34

andrew:fix/megademo-80col

andrew:libakstdlib-26

andrew:23

andrew:todo-to-issues

Conversation 1 Commits 1 Files Changed 17 +426 -13

tachikoma commented 2026-08-06 10:03:58 -04:00

Collaborator

Copy Link

Implements generators for issue #57: GEN / END GEN / EMIT / FOR EACH / DO EACH.

Closes #57

Summary of changes:

• environment.h/environment.c: new isGenerator/generatorFn fields on a GEN call's own environment, and isEachLoop/forGeneratorEnv on a FOR EACH/DO EACH loop's own environment.
• runtime.c: split akbasic_runtime_prev_environment() into akbasic_runtime_detach_environment() (return to parent without releasing) and akbasic_runtime_release_environment() (release variables/pool slot on any environment); prev_environment() is the two in sequence. akbasic_runtime_call_function() refuses to call a GEN like an ordinary function.
• verbs.c/verbs.h/scanner: new keywords GEN, EMIT, EACH, IN and the compound verb "END GEN".
• parser_commands.c: akbasic_parse_gen(), akbasic_parse_end(), and EACH branches in akbasic_parse_for()/akbasic_parse_do().
• runtime_generator.c (new): akbasic_cmd_gen, akbasic_cmd_emit, akbasic_cmd_end_gen, plus shared invoke/pump/release machinery used by FOR EACH, DO EACH, NEXT and LOOP.
• runtime_commands.c/runtime_structure.c: EACH branches in cmd_for/cmd_do/cmd_next/cmd_loop, generator release on abandonment paths (EXIT, mismatched-NEXT pop).
• tests/generators.c (new) plus 9 new .bas/.txt language test pairs under tests/language/flowcontrol/, covering the ROOMOBJECTS example, DO EACH, empty generators, EXIT/pool-leak, non-numeric EMIT, nested/sibling generators, self-recursion rejection, EMIT-outside-GEN, and calling a GEN like a function.
• docs/04-control-flow.md, docs/11-verb-reference.md, docs/14-architecture.md: new documentation sections.

Build is clean; ctest is 123/123 passing, including the docs example checker.

Deviations from the plan (both noted inline in the code):

1. Added an isGenerator flag to akbasic_FunctionDef so calling a GEN like a regular function fails cleanly with a proper BASIC error, since a BASIC-level error raised inside akbasic_runtime_call_function()'s own step loop is otherwise swallowed.
2. Discovered the shared parse_def_parameters() helper doesn't support zero-argument parameter lists (a pre-existing DEF/GEN limitation); every generator in the tests takes at least one parameter as a result.

Implements generators for issue #57: GEN / END GEN / EMIT / FOR EACH / DO EACH. Closes #57 Summary of changes: - environment.h/environment.c: new isGenerator/generatorFn fields on a GEN call's own environment, and isEachLoop/forGeneratorEnv on a FOR EACH/DO EACH loop's own environment. - runtime.c: split akbasic_runtime_prev_environment() into akbasic_runtime_detach_environment() (return to parent without releasing) and akbasic_runtime_release_environment() (release variables/pool slot on any environment); prev_environment() is the two in sequence. akbasic_runtime_call_function() refuses to call a GEN like an ordinary function. - verbs.c/verbs.h/scanner: new keywords GEN, EMIT, EACH, IN and the compound verb "END GEN". - parser_commands.c: akbasic_parse_gen(), akbasic_parse_end(), and EACH branches in akbasic_parse_for()/akbasic_parse_do(). - runtime_generator.c (new): akbasic_cmd_gen, akbasic_cmd_emit, akbasic_cmd_end_gen, plus shared invoke/pump/release machinery used by FOR EACH, DO EACH, NEXT and LOOP. - runtime_commands.c/runtime_structure.c: EACH branches in cmd_for/cmd_do/cmd_next/cmd_loop, generator release on abandonment paths (EXIT, mismatched-NEXT pop). - tests/generators.c (new) plus 9 new .bas/.txt language test pairs under tests/language/flowcontrol/, covering the ROOMOBJECTS example, DO EACH, empty generators, EXIT/pool-leak, non-numeric EMIT, nested/sibling generators, self-recursion rejection, EMIT-outside-GEN, and calling a GEN like a function. - docs/04-control-flow.md, docs/11-verb-reference.md, docs/14-architecture.md: new documentation sections. Build is clean; ctest is 123/123 passing, including the docs example checker. Deviations from the plan (both noted inline in the code): 1. Added an isGenerator flag to akbasic_FunctionDef so calling a GEN like a regular function fails cleanly with a proper BASIC error, since a BASIC-level error raised inside akbasic_runtime_call_function()'s own step loop is otherwise swallowed. 2. Discovered the shared parse_def_parameters() helper doesn't support zero-argument parameter lists (a pre-existing DEF/GEN limitation); every generator in the tests takes at least one parameter as a result.

tachikoma added 1 commit 2026-08-06 10:03:59 -04:00

Implement generators: GEN, EMIT, END GEN, FOR EACH and DO EACH (issue 57) ... f7e8d4b82b

Adds generator support per the plan in issue 57:

- environment.h: isGenerator/generatorFn on a GEN call's own environment,
  isEachLoop and forGeneratorEnv on a FOR EACH/DO EACH loop's own
  environment.
- runtime.c: splits akbasic_runtime_prev_environment() into
  akbasic_runtime_detach_environment() (return to parent without releasing)
  and akbasic_runtime_release_environment() (give variables and the pool
  slot back, on any environment); prev_environment() is now the two in
  sequence. akbasic_runtime_call_function() refuses to call a GEN like an
  ordinary function.
- verbs.c/verbs.h/scanner: new keywords GEN, EMIT, EACH, IN and the compound
  verb "END GEN" (built by a new akbasic_parse_end(), the same trick
  akbasic_parse_print() uses for PRINT #).
- parser_commands.c: akbasic_parse_gen() (modelled on multi-line DEF),
  akbasic_parse_end(), and EACH branches in akbasic_parse_for()/
  akbasic_parse_do().
- runtime_generator.c (new): akbasic_cmd_gen, akbasic_cmd_emit,
  akbasic_cmd_end_gen, and the invoke/pump/release machinery FOR EACH, DO
  EACH, NEXT and LOOP share. EMIT walks up to the nearest isGenerator
  ancestor rather than assuming it is standing directly in the GEN's own
  call frame, because a GEN body may nest its own FOR/DO/GOSUB around an
  EMIT -- the issue's own ROOMOBJECTS example does exactly that.
- runtime_commands.c/runtime_structure.c: EACH branches in cmd_for/cmd_do,
  matching EACH branches in cmd_next/cmd_loop, and forGeneratorEnv release
  on every path that can abandon a live generator (EXIT, a NEXT that pops
  for a mismatched loop variable).

Deviates from the plan in one place: FunctionDef gained an isGenerator flag
(not in the plan's field list) because refusing a GEN called like a
function has to happen before anything is pushed. Relying on EMIT's own
isGenerator check for that case doesn't work: akbasic_runtime_call_function()
drives its own step loop the same way akbasic_runtime_pump_generator() does,
and a BASIC-level error inside that loop is swallowed by process_line_run()
as reported-but-not-propagated, so the call would silently "succeed" with a
meaningless return value instead of failing.

Also: a zero-argument parameter list is not supported by the DEF/GEN
parameter parser this reuses (a pre-existing limitation, not
generator-specific); every generator in the tests takes at least one
parameter as a result.

Tests: tests/generators.c (pool exhaustion under repeated EXIT, calling a
GEN like a function, EMIT outside a GEN, self-recursion, sibling/nested
invocations) and tests/language/flowcontrol/generators_*.bas -- the
issue's own ROOMOBJECTS example in both loop shapes, an empty generator,
non-numeric EMIT, nested/interleaved invocations, and three error-path
golden cases. Docs: control-flow chapter 4 gets a GEN/EMIT/FOR EACH/DO EACH
section, the verb reference gets GEN/EMIT/END GEN entries and updated
FOR/DO/NEXT/LOOP/EXIT rows, and architecture chapter 14 documents the
detach/release split and the two-environment generator invocation.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

tachikoma requested review from andrew 2026-08-06 10:04:13 -04:00

tachikoma referenced this pull request 2026-08-06 10:04:23 -04:00

Implement generators #57

andrew referenced this issue from a commit 2026-08-06 11:29:32 -04:00

Fix generator teardown leaks, add RETURN-in-GEN and LOOP conditions on DO EACH

andrew added 1 commit 2026-08-06 11:29:32 -04:00

Fix generator teardown leaks, add RETURN-in-GEN and LOOP conditions on DO EACH ... 2cb68665d0

Review findings and follow-ups from PR #61 review:

- runtime_generator.c: akbasic_runtime_release_generator() now releases the
  forGeneratorEnv of every scope it walks through. Abandoning a generator
  that was itself suspended inside a FOR EACH over another generator
  stranded the inner generator's pool slot; a loop doing so exhausted the
  twelve-slot pool and died far from the cause.
- runtime.c/runtime.h: new akbasic_runtime_unwind_to_environment(), the
  shared teardown for the error unwinds in pump_generator() and
  call_function() -- both previously bare prev_environment() loops with the
  same suspended-generator blindness.
- runtime_commands.c: bare RETURN standing in a GEN's own frame ends the
  generator exactly as END GEN does -- a GEN is a function at heart. RETURN
  with a value there is refused (values leave a GEN only through EMIT). The
  no-frame error message now says "GOSUB, DEF, or GEN".
- runtime_structure.c: LOOP WHILE/UNTIL composes with DO EACH -- checked
  after each trip with the loop variable still holding that trip's value; a
  condition that stops the loop abandons the generator exactly as EXIT
  does. Previously the condition was silently ignored, while the verb
  reference documented it as working.
- parser_commands.c: trailing tokens after the generator call on a FOR
  EACH/DO EACH line are refused at parse. Previously they sat unparsed and
  blew up only after the loop completed, when the parent scope resumed the
  line mid-statement -- an error at the loop's end pointing at its start.
- tests/generators.c: pool-exhaustion tests for the nested-abandonment and
  LOOP-condition paths, RETURN semantics tests, and a direct test of the
  unwind primitive. Three new golden pairs cover RETURN, LOOP conditions
  and the misplaced-condition parse error.
- docs: RETURN and LOOP-condition semantics in 04-control-flow.md and
  11-verb-reference.md; corrected the self-recursion analogy (functions
  are re-entrant here). TODO.md 1.10 records the generator design
  decisions the code comments were already citing, plus the zero-arg
  parameter-list limitation. MAINTENANCE.md gains the abandoned-generators
  invariant those comments also cited.

Co-Authored-By: Andrew Kesterson <andrew@aklabs.net>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

andrew commented 2026-08-06 11:33:04 -04:00

Owner

Copy Link

Review of this PR complete (conducted by Ishikawa, posting via Andrew's account). The design is solid — the detach/release split, the parent-chain self-recursion check, and EMIT's resume-point handling all verified correct under tracing. One confirmed bug and several semantics gaps were found; fixes are pushed to this branch as 2cb6866. All 126 tests pass, including 3 new golden cases and 5 new C tests.

What was found and fixed:

1. Pool leak on nested-generator abandonment (confirmed bug). akbasic_runtime_release_generator() walked the parent chain but never released a walked scope's own suspended forGeneratorEnv — a child, off the parent chain. Abandoning a generator that was itself suspended inside a FOR EACH over another generator stranded one pool slot per abandonment; a 40-iteration repro died with "Environment pool exhausted at line 60 (12 in use)". Fixed with a guarded recursion; repro and pool-exhaustion tests added.

2. Same blindness in two error-unwind paths. pump_generator()'s and call_function()'s CLEANUP blocks unwound with bare prev_environment() loops. Both now use a new shared primitive, akbasic_runtime_unwind_to_environment(), which releases each popped scope's suspended generator on the way down. MAINTENANCE.md now documents the invariant ("Abandoned generators") — the note this PR's comments cited but which didn't exist yet; same for the TODO.md namespace-decision citations (now TODO.md §1.10).

3. RETURN inside a GEN was a runtime error. Per review discussion with Andrew: a GEN is a function at heart, so bare RETURN standing in the GEN's own frame now ends it exactly as END GEN does. RETURN expr there is refused — values leave a GEN only through EMIT. Inherits the interpreter-wide restriction that RETURN doesn't unwind nested FOR/DO scopes (same as GOSUB/DEF; lifting that everywhere is filed separately).

4. LOOP WHILE c | UNTIL c on a DO EACH was silently ignored — while docs/11-verb-reference.md documented it as working. It now composes: the condition is checked after each trip with the loop variable still holding that trip's value; stopping abandons the generator exactly as EXIT does (released, leak-tested).

5. Trailing tokens on the DO EACH/FOR EACH header were a time bomb. DO EACH V# IN G(0) WHILE V# < 2 ran the whole loop, then died with "Unknown command WHILE" pointing at the DO line — the leftover tokens surfaced only when the parent scope resumed the line mid-statement. Now a parse error, with the DO variant's message naming the likely fix ("takes its WHILE/UNTIL on the LOOP").

6. Docs: RETURN and LOOP-condition semantics added to 04-control-flow.md (examples run by the docs checker); the self-recursion analogy corrected (this interpreter's functions are re-entrant — the old text claimed otherwise); zero-arg parameter-list limitation recorded in TODO.md §1.10.

Deferred to their own issues (opening next): unwinding RETURN through nested scopes (C64-reference behavior; touches GOSUB/DEF/handlers), and zero-argument parameter lists in parse_def_parameters().

Review of this PR complete (conducted by **Ishikawa**, posting via Andrew's account). The design is solid — the detach/release split, the parent-chain self-recursion check, and EMIT's resume-point handling all verified correct under tracing. One confirmed bug and several semantics gaps were found; fixes are pushed to this branch as `2cb6866`. All 126 tests pass, including 3 new golden cases and 5 new C tests. **What was found and fixed:** 1. **Pool leak on nested-generator abandonment (confirmed bug).** `akbasic_runtime_release_generator()` walked the parent chain but never released a walked scope's own suspended `forGeneratorEnv` — a *child*, off the parent chain. Abandoning a generator that was itself suspended inside a `FOR EACH` over another generator stranded one pool slot per abandonment; a 40-iteration repro died with "Environment pool exhausted at line 60 (12 in use)". Fixed with a guarded recursion; repro and pool-exhaustion tests added. 2. **Same blindness in two error-unwind paths.** `pump_generator()`'s and `call_function()`'s CLEANUP blocks unwound with bare `prev_environment()` loops. Both now use a new shared primitive, `akbasic_runtime_unwind_to_environment()`, which releases each popped scope's suspended generator on the way down. MAINTENANCE.md now documents the invariant ("Abandoned generators") — the note this PR's comments cited but which didn't exist yet; same for the TODO.md namespace-decision citations (now TODO.md §1.10). 3. **`RETURN` inside a GEN was a runtime error.** Per review discussion with Andrew: a GEN is a function at heart, so bare `RETURN` standing in the GEN's own frame now ends it exactly as `END GEN` does. `RETURN expr` there is refused — values leave a GEN only through `EMIT`. Inherits the interpreter-wide restriction that RETURN doesn't unwind nested FOR/DO scopes (same as GOSUB/DEF; lifting that everywhere is filed separately). 4. **`LOOP WHILE c | UNTIL c` on a `DO EACH` was silently ignored** — while `docs/11-verb-reference.md` documented it as working. It now composes: the condition is checked after each trip with the loop variable still holding that trip's value; stopping abandons the generator exactly as EXIT does (released, leak-tested). 5. **Trailing tokens on the `DO EACH`/`FOR EACH` header were a time bomb.** `DO EACH V# IN G(0) WHILE V# < 2` ran the whole loop, then died with "Unknown command WHILE" pointing at the DO line — the leftover tokens surfaced only when the parent scope resumed the line mid-statement. Now a parse error, with the DO variant's message naming the likely fix ("takes its WHILE/UNTIL on the LOOP"). 6. **Docs**: RETURN and LOOP-condition semantics added to 04-control-flow.md (examples run by the docs checker); the self-recursion analogy corrected (this interpreter's functions *are* re-entrant — the old text claimed otherwise); zero-arg parameter-list limitation recorded in TODO.md §1.10. **Deferred to their own issues** (opening next): unwinding RETURN through nested scopes (C64-reference behavior; touches GOSUB/DEF/handlers), and zero-argument parameter lists in `parse_def_parameters()`.

andrew referenced this pull request 2026-08-06 11:34:16 -04:00

RETURN should unwind nested scopes to the nearest call frame, as the C64 reference does #62

andrew referenced this pull request 2026-08-06 11:34:17 -04:00

Accept zero-argument parameter lists in DEF and GEN #63

andrew referenced this issue from a commit 2026-08-06 11:58:15 -04:00

Fix generator teardown leaks, add RETURN-in-GEN and LOOP conditions on DO EACH

andrew referenced this issue from a commit 2026-08-06 11:58:15 -04:00

Merge pull request 'Implement generators: GEN, EMIT, END GEN, FOR EACH and DO EACH' (#61) from 57-implement-generators into main

andrew closed this pull request 2026-08-06 12:07:00 -04:00

All checks were successful

Hide all checks

akbasic CI Build / cmake_build (push) Successful in 3m26s

Required

Details

akbasic CI Build / coverage (push) Successful in 4m0s

Required

Details

akbasic CI Build / sanitizers (push) Successful in 6m33s

Required

Details

akbasic CI Build / akgl_build (push) Successful in 9m32s

Required

Details

akbasic CI Build / mutation_test (push) Successful in 18m47s

Required

Details

Pull request closed

Please reopen this pull request to perform a merge.

Sign in to join this conversation.

Reviewers

No Reviewers

andrew

Labels

Clear labels

abi-break

Cannot land without a soname bump

api-gap

A caller cannot express something

blast-radius:high

Ordering: highest blast radius

blast-radius:low

Ordering: lowest blast radius

blast-radius:medium

Ordering: middling blast radius

defect

Wrong behaviour reachable from the public API

design-decision

Needs a decision written down before code

docs

Documentation, comments, AGENTS.md

hygiene

Style, dead code, consistency

packaging

Build, install, pkg-config, CMake config

performance

Has a measurement behind it

status::blocked

Waiting on something else -- a decision, a dependency, or another issue

status::grooming

Not yet worked out: scope, approach or priority still being settled

status::ready

Understood well enough to start: scope and approach are settled

status::review

A pull request has been generated and this ticket is waiting on review of that PR

test-coverage

A gap in the suites, not a defect

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

logikoma tachikoma andrew

No Assignees

2 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: andrew/akbasic#61