Implement generators #57

New Issue

Open

opened 2026-08-05 23:13:23 -04:00 by andrew · 8 comments

andrew commented 2026-08-05 23:13:23 -04:00

Owner

Copy Link

Consider this code

GEN ROOMOBJECTS(R%)
    FOR I% = 0 TO OBJCOUNT%(R%) - 1
        IF VISIBLE(R%, I%) THEN EMIT OBJ%(R%, I%)
    NEXT I%
END GEN

FOR EACH O% IN ROOMOBJECTS(CURROOM%)
    HIGHLIGHT O%
NEXT O%

Plan

1. Data model (include/akbasic/environment.h)

Add to akbasic_Environment:

• bool isGenerator — set on the environment a GEN call pushes.
• struct akbasic_Environment *forGeneratorEnv — lives on the FOR EACH/DO EACH
  loop's own environment, pointing at the detached-but-alive generator environment
  between iterations. NULL means "not an EACH loop" or "generator exhausted."
• Reuse forNextVariable for the EACH variable.

No new field needed to resume a generator: each environment already tracks its own
nextline.

2. Split akbasic_runtime_prev_environment() (src/runtime.c)

• akbasic_runtime_detach_environment(obj) — returns control to ->parent without
  freeing (no variable-release, no used = false).
• akbasic_runtime_release_environment(obj, env) — the variable-release + used = false
  part, callable on an environment that isn't necessarily obj->environment right now.
• akbasic_runtime_prev_environment(obj) becomes detach + release in sequence —
  every existing GOSUB/RETURN/DEF/FOR/NEXT caller is unaffected.

3. Grammar / scanner

New keywords: GEN, END GEN, EMIT, EACH — add to verb table (src/verbs.c/
verbs.h) and scanner keyword list (src/scanner.c). GEN gets an AST shape mirroring
multi-line DEF (name + parameter list; body scanned to END GEN). EACH must be
parseable after both FOR and DO.

4. GEN / END GEN

Modeled on multi-line DEF/RETURN: arms
akbasic_environment_wait_for_command(env, "END GEN") so the body is skipped on first
pass, only running when invoked via FOR EACH/DO EACH (not the expression-call path
used by AKBASIC_MAX_FUNCTIONS). The EACH branch of cmd_for/cmd_do pushes an
environment (parent = the loop's own environment), marks isGenerator = true, binds
params like a function call, sets nextline to the first line of the GEN body.

5. EMIT (akbasic_cmd_emit, structurally close to akbasic_cmd_return)

• Requires obj->environment->isGenerator (error otherwise).
• Evaluates its expression, assigns into obj->environment->parent->forNextVariable.
• Sets obj->environment->parent->nextline to the FOR EACH/DO EACH loop body start.
• Calls akbasic_runtime_detach_environment(obj) (not prev_environment) so the
  generator environment survives, still referenced by the loop's forGeneratorEnv.

6. FOR EACH ... IN ... parsing (akbasic_parse_for)

• Leaf right of FOR must be a variable or EACH, else syntax error.
• On EACH: parse EACH <var> IN <generator-call-expr>, stash the call expression on
  the pushed environment instead of forToLeaf/forStepLeaf.
• EACH accepts any emitted type (STRING, structure element, etc.) — assignment
  bypasses evaluate_for_condition's arithmetic machinery. The numeric-only type-check
  for NEXT <var> stays on the plain FOR path only.

akbasic_cmd_for EACH branch:

• Look up the GEN by name in functions (shared table with DEF; one lookup, one
  "unknown function/generator" error path; names still can't collide across GEN/DEF).
• Before pushing the call's environment, walk parent from the current environment
  looking for an active, reachable isGenerator environment with matching function
  identity. If found, error (self-recursion). A detached environment sitting in another
  loop's forGeneratorEnv is a sibling, not an ancestor, so independent/nested FOR EACH/DO EACH over the same GEN is unaffected.
• Push and run the generator body until the first EMIT (detaches back here) or END GEN with no EMIT (empty generator, zero iterations).
• Zero-iteration case: akbasic_environment_wait_for_command(obj->environment, "NEXT")
  on the loop environment.

7. NEXT <var> — EACH branch (akbasic_cmd_next)

Ahead of the existing numeric-condition logic: if obj->environment->forGeneratorEnv != NULL:

• Set obj->environment = forGeneratorEnv, resume at forGeneratorEnv->nextline
  (parent is unchanged since EMIT only detached).
• Resumes until next EMIT (repeat) or real END GEN.
• On real END GEN: akbasic_runtime_release_environment() the generator env, clear
  forGeneratorEnv on the loop environment, pop the loop environment too, hand
  nextline to whatever follows NEXT — same as today's EXIT-into-NEXT path.

8. DO ... LOOP EACH ... IN ...

Reuses sections 1–7 directly:

• forGeneratorEnv lives on the DO's own environment the same way.
• akbasic_parse_do gets the same EACH <var> IN <generator-call-expr> branch as
  akbasic_parse_for, stashing the call expression alongside/instead of
  doConditionLeaf/doConditionKind. DO EACH and DO WHILE/UNTIL are mutually
  exclusive on the same DO.
• akbasic_cmd_do gains the same zero-iteration check as cmd_for: push generator, run
  to first EMIT or immediate END GEN → wait_for_command(obj->environment, "LOOP").
• akbasic_cmd_loop gains the same check as cmd_next: if
  obj->environment->forGeneratorEnv != NULL, reactivate and resume at its nextline,
  ahead of (not blended with) the WHILE/UNTIL/bare-LOOP logic (never reached by DO EACH).
• Natural exhaustion releases the generator environment and pops the DO environment
  exactly as section 7, landing on whatever follows LOOP.
• EXIT needs no new branch — already dispatches on obj->environment->isDoLoop, which
  DO EACH sets and FOR EACH doesn't.

Net new work: parser plumbing (akbasic_parse_do accepting EACH) plus the two
forGeneratorEnv checks in cmd_do/cmd_loop mirroring cmd_for/cmd_next.

9. Abandoned generators (early EXIT, GOTO out of the loop, error unwind)

Wherever a loop environment is popped/released (EXIT's path in cmd_next, the
equivalent path in cmd_loop, any error-unwind path walking ->parent releasing
scopes): check forGeneratorEnv != NULL and release it too via
akbasic_runtime_release_environment(). Otherwise it leaks in the
AKBASIC_MAX_ENVIRONMENTS pool. This is the main new failure mode — needs dedicated
test cases for both loop shapes.

10. Design decisions

1. Namespace: GEN and DEF share the functions table, at least to start.
2. DO ... LOOP EACH: in scope for v1.
3. Emitted type: FOR EACH/DO EACH can emit any type; numeric-only check stays on
   plain NEXT only.
4. Self-recursion: disallowed (section 6 ancestor-chain walk). Nested/sibling FOR EACH/DO EACH over the same GEN (even same name, different args) is explicitly
   allowed — each invocation is a fresh pool environment.
5. EXIT: unchanged, dispatches on isDoLoop exactly as today.

11. Testing

• GEN ROOMOBJECTS / FOR EACH snippet from this issue, checked against expected
  emitted order.
• Same generator via DO EACH ... LOOP, same expected order.
• Empty generator (END GEN, no EMIT) → zero iterations, no crash, both loop shapes.
• EXIT out of FOR EACH and DO EACH partway through → generator env released, pool
  not leaked (loop AKBASIC_MAX_ENVIRONMENTS+1 such constructs in one test per shape).
• GEN emitting a non-numeric value (e.g. STRING) via FOR EACH/DO EACH → succeeds.
• Nested FOR EACH/DO EACH (legitimate): outer loop body containing inner FOR EACH/DO EACH, including both invoking the same GEN with different args → both run
  to completion, correct interleaved emitted values, no false rejection, no pool
  corruption.
• Self-recursion (rejected): a GEN that FOR EACH/DO EACHs back over itself from its
  own still-running invocation → clean error, no corrupted environment stack.
• EMIT outside a GEN body, and mismatched-NEXT variable case → errors like the
  existing mismatched-NEXT case.
• GEN invoked without being consumed via FOR EACH/DO EACH (e.g. called like a
  function) → clean error, no stack corruption.

12. Documentation

• Architecture chapter: how environments work in this paradigm.
• Control flow chapter: generators.
• Verbs chapter: generator verbs.

Undefined behavior

• Assigning to a generator.
• Calling a generator like a regular function.

Consider this code ``` GEN ROOMOBJECTS(R%) FOR I% = 0 TO OBJCOUNT%(R%) - 1 IF VISIBLE(R%, I%) THEN EMIT OBJ%(R%, I%) NEXT I% END GEN FOR EACH O% IN ROOMOBJECTS(CURROOM%) HIGHLIGHT O% NEXT O% ``` ## Plan ### 1. Data model (`include/akbasic/environment.h`) Add to `akbasic_Environment`: - `bool isGenerator` — set on the environment a `GEN` call pushes. - `struct akbasic_Environment *forGeneratorEnv` — lives on the `FOR EACH`/`DO EACH` loop's own environment, pointing at the detached-but-alive generator environment between iterations. `NULL` means "not an EACH loop" or "generator exhausted." - Reuse `forNextVariable` for the `EACH` variable. No new field needed to resume a generator: each environment already tracks its own `nextline`. ### 2. Split `akbasic_runtime_prev_environment()` (`src/runtime.c`) - `akbasic_runtime_detach_environment(obj)` — returns control to `->parent` without freeing (no variable-release, no `used = false`). - `akbasic_runtime_release_environment(obj, env)` — the variable-release + `used = false` part, callable on an environment that isn't necessarily `obj->environment` right now. - `akbasic_runtime_prev_environment(obj)` becomes `detach` + `release` in sequence — every existing GOSUB/RETURN/DEF/FOR/NEXT caller is unaffected. ### 3. Grammar / scanner New keywords: `GEN`, `END GEN`, `EMIT`, `EACH` — add to verb table (`src/verbs.c`/ `verbs.h`) and scanner keyword list (`src/scanner.c`). `GEN` gets an AST shape mirroring multi-line `DEF` (name + parameter list; body scanned to `END GEN`). `EACH` must be parseable after both `FOR` and `DO`. ### 4. `GEN` / `END GEN` Modeled on multi-line `DEF`/`RETURN`: arms `akbasic_environment_wait_for_command(env, "END GEN")` so the body is skipped on first pass, only running when invoked via `FOR EACH`/`DO EACH` (not the expression-call path used by `AKBASIC_MAX_FUNCTIONS`). The EACH branch of `cmd_for`/`cmd_do` pushes an environment (parent = the loop's own environment), marks `isGenerator = true`, binds params like a function call, sets `nextline` to the first line of the `GEN` body. ### 5. `EMIT` (`akbasic_cmd_emit`, structurally close to `akbasic_cmd_return`) - Requires `obj->environment->isGenerator` (error otherwise). - Evaluates its expression, assigns into `obj->environment->parent->forNextVariable`. - Sets `obj->environment->parent->nextline` to the `FOR EACH`/`DO EACH` loop body start. - Calls `akbasic_runtime_detach_environment(obj)` (not `prev_environment`) so the generator environment survives, still referenced by the loop's `forGeneratorEnv`. ### 6. `FOR EACH ... IN ...` parsing (`akbasic_parse_for`) - Leaf right of `FOR` must be a variable or `EACH`, else syntax error. - On `EACH`: parse `EACH <var> IN <generator-call-expr>`, stash the call expression on the pushed environment instead of `forToLeaf`/`forStepLeaf`. - `EACH` accepts any emitted type (`STRING`, structure element, etc.) — assignment bypasses `evaluate_for_condition`'s arithmetic machinery. The numeric-only type-check for `NEXT <var>` stays on the plain `FOR` path only. `akbasic_cmd_for` EACH branch: - Look up the `GEN` by name in `functions` (shared table with `DEF`; one lookup, one "unknown function/generator" error path; names still can't collide across `GEN`/`DEF`). - Before pushing the call's environment, walk `parent` from the current environment looking for an active, reachable `isGenerator` environment with matching function identity. If found, error (self-recursion). A detached environment sitting in another loop's `forGeneratorEnv` is a sibling, not an ancestor, so independent/nested `FOR EACH`/`DO EACH` over the same `GEN` is unaffected. - Push and run the generator body until the first `EMIT` (detaches back here) or `END GEN` with no `EMIT` (empty generator, zero iterations). - Zero-iteration case: `akbasic_environment_wait_for_command(obj->environment, "NEXT")` on the loop environment. ### 7. `NEXT <var>` — EACH branch (`akbasic_cmd_next`) Ahead of the existing numeric-condition logic: if `obj->environment->forGeneratorEnv != NULL`: - Set `obj->environment = forGeneratorEnv`, resume at `forGeneratorEnv->nextline` (`parent` is unchanged since `EMIT` only detached). - Resumes until next `EMIT` (repeat) or real `END GEN`. - On real `END GEN`: `akbasic_runtime_release_environment()` the generator env, clear `forGeneratorEnv` on the loop environment, pop the loop environment too, hand `nextline` to whatever follows `NEXT` — same as today's `EXIT`-into-`NEXT` path. ### 8. `DO ... LOOP EACH ... IN ...` Reuses sections 1–7 directly: - `forGeneratorEnv` lives on the `DO`'s own environment the same way. - `akbasic_parse_do` gets the same `EACH <var> IN <generator-call-expr>` branch as `akbasic_parse_for`, stashing the call expression alongside/instead of `doConditionLeaf`/`doConditionKind`. `DO EACH` and `DO WHILE/UNTIL` are mutually exclusive on the same `DO`. - `akbasic_cmd_do` gains the same zero-iteration check as `cmd_for`: push generator, run to first `EMIT` or immediate `END GEN` → `wait_for_command(obj->environment, "LOOP")`. - `akbasic_cmd_loop` gains the same check as `cmd_next`: if `obj->environment->forGeneratorEnv != NULL`, reactivate and resume at its `nextline`, ahead of (not blended with) the `WHILE`/`UNTIL`/bare-`LOOP` logic (never reached by `DO EACH`). - Natural exhaustion releases the generator environment and pops the `DO` environment exactly as section 7, landing on whatever follows `LOOP`. - `EXIT` needs no new branch — already dispatches on `obj->environment->isDoLoop`, which `DO EACH` sets and `FOR EACH` doesn't. Net new work: parser plumbing (`akbasic_parse_do` accepting `EACH`) plus the two `forGeneratorEnv` checks in `cmd_do`/`cmd_loop` mirroring `cmd_for`/`cmd_next`. ### 9. Abandoned generators (early `EXIT`, `GOTO` out of the loop, error unwind) Wherever a loop environment is popped/released (`EXIT`'s path in `cmd_next`, the equivalent path in `cmd_loop`, any error-unwind path walking `->parent` releasing scopes): check `forGeneratorEnv != NULL` and release it too via `akbasic_runtime_release_environment()`. Otherwise it leaks in the `AKBASIC_MAX_ENVIRONMENTS` pool. This is the main new failure mode — needs dedicated test cases for both loop shapes. ### 10. Design decisions 1. **Namespace**: `GEN` and `DEF` share the `functions` table, at least to start. 2. **`DO ... LOOP EACH`**: in scope for v1. 3. **Emitted type**: `FOR EACH`/`DO EACH` can emit any type; numeric-only check stays on plain `NEXT` only. 4. **Self-recursion**: disallowed (section 6 ancestor-chain walk). Nested/sibling `FOR EACH`/`DO EACH` over the same `GEN` (even same name, different args) is explicitly allowed — each invocation is a fresh pool environment. 5. **`EXIT`**: unchanged, dispatches on `isDoLoop` exactly as today. ### 11. Testing - `GEN ROOMOBJECTS` / `FOR EACH` snippet from this issue, checked against expected emitted order. - Same generator via `DO EACH ... LOOP`, same expected order. - Empty generator (`END GEN`, no `EMIT`) → zero iterations, no crash, both loop shapes. - `EXIT` out of `FOR EACH` and `DO EACH` partway through → generator env released, pool not leaked (loop `AKBASIC_MAX_ENVIRONMENTS`+1 such constructs in one test per shape). - `GEN` emitting a non-numeric value (e.g. `STRING`) via `FOR EACH`/`DO EACH` → succeeds. - Nested `FOR EACH`/`DO EACH` (legitimate): outer loop body containing inner `FOR EACH`/`DO EACH`, including both invoking the same `GEN` with different args → both run to completion, correct interleaved emitted values, no false rejection, no pool corruption. - Self-recursion (rejected): a `GEN` that `FOR EACH`/`DO EACH`s back over itself from its own still-running invocation → clean error, no corrupted environment stack. - `EMIT` outside a `GEN` body, and mismatched-`NEXT` variable case → errors like the existing mismatched-`NEXT` case. - `GEN` invoked without being consumed via `FOR EACH`/`DO EACH` (e.g. called like a function) → clean error, no stack corruption. ### 12. Documentation - Architecture chapter: how environments work in this paradigm. - Control flow chapter: generators. - Verbs chapter: generator verbs. ### Undefined behavior - Assigning to a generator. - Calling a generator like a regular function.

andrew added the api-gapdesign-decisionblast-radius:highstatus::grooming labels 2026-08-05 23:13:23 -04:00

andrew commented 2026-08-05 23:13:36 -04:00

Author

Owner

Copy Link

@tachikoma give me a plan to implement this

@tachikoma give me a plan to implement this

👀 1

andrew commented 2026-08-06 06:17:11 -04:00

Author

Owner

Copy Link

@tachikoma update the plan for a few things:

1. account for ‘DO … LOOP’ constructs. It seems EACH should work there as well, and it seems like the work is basically free once we get everything else working
2. Let’s try GEN and DEF using the same function table space at first (question 1)
3. for each can emit any type (question 2)
4. let’s not allow recursive generators for now
5. lets leave EXIT unambiguous - same meaning wherever it is used

@tachikoma update the plan for a few things: 1. account for ‘DO … LOOP’ constructs. It seems EACH should work there as well, and it seems like the work is basically free once we get everything else working 2. Let’s try GEN and DEF using the same function table space at first (question 1) 3. for each can emit any type (question 2) 4. let’s not allow recursive generators for now 5. lets leave EXIT unambiguous - same meaning wherever it is used

👀 1

andrew commented 2026-08-06 07:02:23 -04:00

Author

Owner

Copy Link

@tachikoma i disagree here

This also means
the "nested FOR EACH" test case from the original testing list, section 11 below,
changes from "exercises recursive chaining" to "confirms recursive chaining is
rejected.")

I don’t want to allow a generator to call back into itself (recursion). I do want to allow nested ‘FOR EACH’ or ‘DO … LOOP EACH’ constructs. Even if each of those constructs call the same generator (hopefully with different arguments - but that’s a documentation issue), each invocation is a new instance with a new environment. So the environment chaining still must be accounted for.

@tachikoma i disagree here > This also means the "nested FOR EACH" test case from the original testing list, section 11 below, changes from "exercises recursive chaining" to "confirms recursive chaining is rejected.") I don’t want to allow a generator to call back into itself (recursion). I do want to allow nested ‘FOR EACH’ or ‘DO … LOOP EACH’ constructs. Even if each of those constructs call the same generator (hopefully with different arguments - but that’s a documentation issue), each invocation is a new instance with a new environment. So the environment chaining still must be accounted for.

👀 1

andrew commented 2026-08-06 07:23:41 -04:00

Author

Owner

Copy Link

@tachikoma compact the plan please, ditch the history and focus on the implementation.

@tachikoma compact the plan please, ditch the history and focus on the implementation.

👀 1

andrew added status::ready and removed status::grooming labels 2026-08-06 08:33:39 -04:00

tachikoma was assigned by andrew 2026-08-06 08:33:42 -04:00

tachikoma commented 2026-08-06 08:48:20 -04:00

Collaborator

Copy Link

Compacted the plan in the issue body — trimmed the narrative/rationale asides and "today's behavior" explanations, kept all 12 implementation sections (data model, environment split, grammar, GEN/END GEN, EMIT, FOR EACH, NEXT, DO LOOP EACH, abandoned-generator cleanup, design decisions, testing, docs) plus the undefined-behavior notes. No code changed.

Compacted the plan in the issue body — trimmed the narrative/rationale asides and "today's behavior" explanations, kept all 12 implementation sections (data model, environment split, grammar, GEN/END GEN, EMIT, FOR EACH, NEXT, DO LOOP EACH, abandoned-generator cleanup, design decisions, testing, docs) plus the undefined-behavior notes. No code changed.

andrew commented 2026-08-06 08:53:51 -04:00

Author

Owner

Copy Link

@tachikoma execute this ticket please, it's assigned to you and ready

@tachikoma execute this ticket please, it's assigned to you and ready

tachikoma commented 2026-08-06 09:16:52 -04:00

Collaborator

Copy Link

Beginning work on this ticket: implementing generators (GEN/END GEN/EMIT/FOR EACH/DO EACH) per the plan in the ticket body.

Beginning work on this ticket: implementing generators (GEN/END GEN/EMIT/FOR EACH/DO EACH) per the plan in the ticket body.

tachikoma referenced a pull request that will close this issue 2026-08-06 10:03:58 -04:00

Implement generators (GEN/EMIT/END GEN/FOR EACH/DO EACH) - closes #57 #61

tachikoma added status::review and removed status::ready labels 2026-08-06 10:04:18 -04:00

tachikoma commented 2026-08-06 10:04:23 -04:00

Collaborator

Copy Link

Work is done. Implemented GEN/EMIT/END GEN/FOR EACH/DO EACH per the plan above, with full test coverage (123/123 ctest passing) and documentation updates.

PR: #61

Work is done. Implemented GEN/EMIT/END GEN/FOR EACH/DO EACH per the plan above, with full test coverage (123/123 ctest passing) and documentation updates. PR: https://source.starfort.tech/andrew/akbasic/pulls/61

Sign in to join this conversation.

No Branch/Tag Specified

Branches Tags

main

10

47-prepush

36

46

57-implement-generators

deps-update

56-doxygen-gate-fixes

16

47

galaga-tutorial

feature/reduce_memory_usage

40

34

fix/megademo-80col

libakstdlib-26

23

todo-to-issues

No results found.

Labels

Clear labels

abi-break

Cannot land without a soname bump

api-gap

A caller cannot express something

blast-radius:high

Ordering: highest blast radius

blast-radius:low

Ordering: lowest blast radius

blast-radius:medium

Ordering: middling blast radius

defect

Wrong behaviour reachable from the public API

design-decision

Needs a decision written down before code

docs

Documentation, comments, AGENTS.md

hygiene

Style, dead code, consistency

packaging

Build, install, pkg-config, CMake config

performance

Has a measurement behind it

status::blocked

Waiting on something else -- a decision, a dependency, or another issue

status::grooming

Not yet worked out: scope, approach or priority still being settled

status::ready

Understood well enough to start: scope and approach are settled

status::review

A pull request has been generated and this ticket is waiting on review of that PR

test-coverage

A gap in the suites, not a defect

No Label api-gap blast-radius:high design-decision status::review

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

logikoma tachikoma andrew

No Assignees tachikoma

2 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: andrew/akbasic#57