Record how to find out whether a tutorial teaches
Some checks failed
libakgl CI Build / cmake_build (push) Failing after 21s
libakgl CI Build / performance (push) Failing after 20s
libakgl CI Build / memory_check (push) Failing after 17s
libakgl CI Build / mutation_test (push) Failing after 19s

docs_examples proves every listing in docs/ compiles and still matches the file
it was quoted from. It cannot prove a chapter teaches anything -- a document can
be made entirely of verified excerpts and still be unfollowable, because what a
reader needs is the glue between them.

So AGENTS.md gains "Testing a Tutorial": hand the chapter to a subagent on a
weaker model with the library, its headers and the art but not the finished
program, and make them build and run the game. The weaker model is the point; it
will not paper over a gap with knowledge the document did not give it.

The section carries the rules that make the result mean something -- copy the
tree with examples/ and docs/ removed rather than asking them not to look, let
them read the public headers because a real reader has them, do not make them
hand-author a .tmj, give them a screenshot helper, and verify their binary
yourself -- plus the sandbox recipe, and the warning to prove the build path
works before handing it over.

It also records how to read the report, because a weaker model states its own
mistakes as documentation defects with total confidence. Reproduce before
believing; and a rejected finding usually still leaves something behind.

The evidence for doing any of this is in the table at the end: four read-only
reviews of chapters 20 and 21 found real gaps and missed all three of the
defects the first build-and-run found, including a published example output the
chapter could not produce.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KzBDV2fqgnUAcqCKqKvc71
This commit is contained in:
2026-08-02 09:45:28 -04:00
parent cf0142ea85
commit 75da766724
2 changed files with 102 additions and 0 deletions

View File

@@ -682,6 +682,101 @@ of a hundred thousand times, and the timings a checked run prints are labelled
as meaningless. **Do not add memory-check suites**: a new path worth checking as meaningless. **Do not add memory-check suites**: a new path worth checking
belongs in a benchmark, where it gets both. belongs in a benchmark, where it gets both.
## Testing a Tutorial
`ctest -R docs_examples` proves every listing in `docs/` still compiles and
still matches the file it was quoted from. It cannot prove the chapter *teaches*
anything: a document can be composed entirely of verified excerpts and still be
unfollowable, because what a reader needs is the glue between them.
**So test a tutorial by making somebody follow it.** Hand it to a subagent on a
weaker model with no context beyond the chapter, and have them build the thing it
describes. The weaker model is the point -- it will not paper over a gap with
knowledge the document did not give it, which is exactly the failure mode a
capable reviewer has.
### The rules that make the result mean something
- **Give them what a real reader has, and nothing more.** The chapter, the
library source, its public headers, and the art. **Not the finished program.**
Copy the tree with `examples/` and `docs/` removed rather than telling them not
to look -- a rule they can break is not a control.
- **Headers are fair game.** The manual defers signatures to Doxygen and a real
user has `include/akgl/*.h` in front of them. Forbidding those tests a reader
who does not exist.
- **They must compile it and run it.** This is the whole thing. A design nobody
built proves nothing, and the defects that matter most do not fail to compile.
- **Verify their work yourself.** Run the binary they produced. Look at the
screenshot they took. Do not take a report's word for what it built --
see "Reading the report" below.
- **Do not make them author assets by hand.** A real reader draws a map in Tiled;
hand-writing a `.tmj` tests nothing about the chapter and eats the whole
session. Give them `docs/tutorials/assets/`. The asset *formats* are already
covered by `docs_examples`, which runs the chapter's own JSON blocks through
`akgl_sprite_load_json` and `akgl_character_load_json`.
- **Give them a screenshot helper**, marked as scaffolding and not part of the
tutorial, so there is a picture to check. Screen output is evidence; an exit
status of 0 is not.
### Setting the sandbox up
One directory per reader, because two of them building at once in the same tree
collide:
```sh
rsync -a --exclude='.git' --exclude='build' --exclude='examples' --exclude='docs' \
. "$SB/reader/libakgl/"
cp docs/20-tutorial-sidescroller.md "$SB/reader/TUTORIAL.md"
cp -r docs/tutorials/assets/sidescroller "$SB/reader/art"
mkdir -p "$SB/reader/game"
```
A consumer using the CMake the chapter itself teaches --
`add_subdirectory(../libakgl libakgl)` plus the documented
`target_link_libraries` line -- configures and builds in about a minute from
cold, and seconds after that. Tell them to use
`cmake --build build --target <theirs> --parallel`, or they will also build
every test suite in the tree.
**Prove the path works before you hand it over.** Write a throwaway consumer
that opens a window and takes a screenshot, build it, run it, and delete it. A
reader who cannot build is a reader who finds nothing, and the fault will be
yours -- the screenshot helper's first draft included `akgl/renderer.h` for
`akgl_renderer`, which is declared in `akgl/game.h`, and would have cost them
the session.
### Reading the report
**Every finding is a hypothesis until you check it.** A weaker model reports its
own mistakes as documentation defects with total confidence, and both kinds are
worth having -- but only one is worth acting on.
- Reproduce the failure before believing it. One reader reported the dialogue
freeze broken; it had drawn its own map with the NPC out of reach and never
used the one it was given. Running the reference showed the player moving zero
pixels through the whole freeze window.
- A rejected finding usually still leaves something. That same map failure was
not a code defect, but it did show the chapter stated `TALK_RANGE` without
saying what it means when you are placing NPCs -- and that a too-distant NPC
produces *success*, not an error.
- Fix the document, not the reader. If they guessed a header name, the chapter
never named it.
### What this catches that review does not
Four read-only passes over the two tutorials in `docs/` found real gaps and
missed all three of these, which the first build-and-run found immediately:
| Defect | Why reading missed it |
|---|---|
| libakstdlib's header was never named -- `aksl_*` functions, `<akstdlib.h>` file | Every reviewer already knew, or did not have to write the `#include` |
| The asset directory defaulted to `"."` instead of the compiled-in macro | Prose said "falls back", and nobody had to run it from another directory |
| A published example output the chapter could not produce | The capture happened after teardown released the pool. Nothing fails to compile |
The third is the shape to remember: **a tutorial that prints an expected result
is making a claim, and a claim nobody executed is a claim that is probably
wrong.**
## Commit & Pull Request Guidelines ## Commit & Pull Request Guidelines
Recent commits use concise, imperative summaries such as `Fix a scale bug...` and often explain related changes in one sentence. Keep each commit scoped and describe user-visible behavior. Pull requests should summarize the change, identify affected modules, list the CMake/CTest commands run, and link relevant issues. Include screenshots only for rendering, map, or `charviewer` changes. Recent commits use concise, imperative summaries such as `Fix a scale bug...` and often explain related changes in one sentence. Keep each commit scoped and describe user-visible behavior. Pull requests should summarize the change, identify affected modules, list the CMake/CTest commands run, and link relevant issues. Include screenshots only for rendering, map, or `charviewer` changes.

View File

@@ -4,6 +4,13 @@ This file documents the harness that keeps `docs/` honest. It is not a chapter
a reader learning libakgl never needs it — and it is checked by the same harness a reader learning libakgl never needs it — and it is checked by the same harness
it describes, so the file that publishes the convention is held to it. it describes, so the file that publishes the convention is held to it.
**The harness checks that every listing is true. It cannot check that a chapter
teaches.** For that, the tutorials get a second test: the chapter is handed to a
subagent on a weaker model with the library, the headers and the art but *not*
the finished program, and that reader has to build and run the game. See
`AGENTS.md`, "Testing a Tutorial". Four read-only reviews of chapters 20 and 21
missed three defects the first build-and-run found in minutes.
## Documentation examples ## Documentation examples
**Every fenced block in `docs/*.md` is compiled, linked, run, or matched against **Every fenced block in `docs/*.md` is compiled, linked, run, or matched against