Record how to find out whether a tutorial teaches
docs_examples proves every listing in docs/ compiles and still matches the file it was quoted from. It cannot prove a chapter teaches anything -- a document can be made entirely of verified excerpts and still be unfollowable, because what a reader needs is the glue between them. So AGENTS.md gains "Testing a Tutorial": hand the chapter to a subagent on a weaker model with the library, its headers and the art but not the finished program, and make them build and run the game. The weaker model is the point; it will not paper over a gap with knowledge the document did not give it. The section carries the rules that make the result mean something -- copy the tree with examples/ and docs/ removed rather than asking them not to look, let them read the public headers because a real reader has them, do not make them hand-author a .tmj, give them a screenshot helper, and verify their binary yourself -- plus the sandbox recipe, and the warning to prove the build path works before handing it over. It also records how to read the report, because a weaker model states its own mistakes as documentation defects with total confidence. Reproduce before believing; and a rejected finding usually still leaves something behind. The evidence for doing any of this is in the table at the end: four read-only reviews of chapters 20 and 21 found real gaps and missed all three of the defects the first build-and-run found, including a published example output the chapter could not produce. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KzBDV2fqgnUAcqCKqKvc71
This commit is contained in:
95
AGENTS.md
95
AGENTS.md
@@ -682,6 +682,101 @@ of a hundred thousand times, and the timings a checked run prints are labelled
|
|||||||
as meaningless. **Do not add memory-check suites**: a new path worth checking
|
as meaningless. **Do not add memory-check suites**: a new path worth checking
|
||||||
belongs in a benchmark, where it gets both.
|
belongs in a benchmark, where it gets both.
|
||||||
|
|
||||||
|
## Testing a Tutorial
|
||||||
|
|
||||||
|
`ctest -R docs_examples` proves every listing in `docs/` still compiles and
|
||||||
|
still matches the file it was quoted from. It cannot prove the chapter *teaches*
|
||||||
|
anything: a document can be composed entirely of verified excerpts and still be
|
||||||
|
unfollowable, because what a reader needs is the glue between them.
|
||||||
|
|
||||||
|
**So test a tutorial by making somebody follow it.** Hand it to a subagent on a
|
||||||
|
weaker model with no context beyond the chapter, and have them build the thing it
|
||||||
|
describes. The weaker model is the point -- it will not paper over a gap with
|
||||||
|
knowledge the document did not give it, which is exactly the failure mode a
|
||||||
|
capable reviewer has.
|
||||||
|
|
||||||
|
### The rules that make the result mean something
|
||||||
|
|
||||||
|
- **Give them what a real reader has, and nothing more.** The chapter, the
|
||||||
|
library source, its public headers, and the art. **Not the finished program.**
|
||||||
|
Copy the tree with `examples/` and `docs/` removed rather than telling them not
|
||||||
|
to look -- a rule they can break is not a control.
|
||||||
|
- **Headers are fair game.** The manual defers signatures to Doxygen and a real
|
||||||
|
user has `include/akgl/*.h` in front of them. Forbidding those tests a reader
|
||||||
|
who does not exist.
|
||||||
|
- **They must compile it and run it.** This is the whole thing. A design nobody
|
||||||
|
built proves nothing, and the defects that matter most do not fail to compile.
|
||||||
|
- **Verify their work yourself.** Run the binary they produced. Look at the
|
||||||
|
screenshot they took. Do not take a report's word for what it built --
|
||||||
|
see "Reading the report" below.
|
||||||
|
- **Do not make them author assets by hand.** A real reader draws a map in Tiled;
|
||||||
|
hand-writing a `.tmj` tests nothing about the chapter and eats the whole
|
||||||
|
session. Give them `docs/tutorials/assets/`. The asset *formats* are already
|
||||||
|
covered by `docs_examples`, which runs the chapter's own JSON blocks through
|
||||||
|
`akgl_sprite_load_json` and `akgl_character_load_json`.
|
||||||
|
- **Give them a screenshot helper**, marked as scaffolding and not part of the
|
||||||
|
tutorial, so there is a picture to check. Screen output is evidence; an exit
|
||||||
|
status of 0 is not.
|
||||||
|
|
||||||
|
### Setting the sandbox up
|
||||||
|
|
||||||
|
One directory per reader, because two of them building at once in the same tree
|
||||||
|
collide:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
rsync -a --exclude='.git' --exclude='build' --exclude='examples' --exclude='docs' \
|
||||||
|
. "$SB/reader/libakgl/"
|
||||||
|
cp docs/20-tutorial-sidescroller.md "$SB/reader/TUTORIAL.md"
|
||||||
|
cp -r docs/tutorials/assets/sidescroller "$SB/reader/art"
|
||||||
|
mkdir -p "$SB/reader/game"
|
||||||
|
```
|
||||||
|
|
||||||
|
A consumer using the CMake the chapter itself teaches --
|
||||||
|
`add_subdirectory(../libakgl libakgl)` plus the documented
|
||||||
|
`target_link_libraries` line -- configures and builds in about a minute from
|
||||||
|
cold, and seconds after that. Tell them to use
|
||||||
|
`cmake --build build --target <theirs> --parallel`, or they will also build
|
||||||
|
every test suite in the tree.
|
||||||
|
|
||||||
|
**Prove the path works before you hand it over.** Write a throwaway consumer
|
||||||
|
that opens a window and takes a screenshot, build it, run it, and delete it. A
|
||||||
|
reader who cannot build is a reader who finds nothing, and the fault will be
|
||||||
|
yours -- the screenshot helper's first draft included `akgl/renderer.h` for
|
||||||
|
`akgl_renderer`, which is declared in `akgl/game.h`, and would have cost them
|
||||||
|
the session.
|
||||||
|
|
||||||
|
### Reading the report
|
||||||
|
|
||||||
|
**Every finding is a hypothesis until you check it.** A weaker model reports its
|
||||||
|
own mistakes as documentation defects with total confidence, and both kinds are
|
||||||
|
worth having -- but only one is worth acting on.
|
||||||
|
|
||||||
|
- Reproduce the failure before believing it. One reader reported the dialogue
|
||||||
|
freeze broken; it had drawn its own map with the NPC out of reach and never
|
||||||
|
used the one it was given. Running the reference showed the player moving zero
|
||||||
|
pixels through the whole freeze window.
|
||||||
|
- A rejected finding usually still leaves something. That same map failure was
|
||||||
|
not a code defect, but it did show the chapter stated `TALK_RANGE` without
|
||||||
|
saying what it means when you are placing NPCs -- and that a too-distant NPC
|
||||||
|
produces *success*, not an error.
|
||||||
|
- Fix the document, not the reader. If they guessed a header name, the chapter
|
||||||
|
never named it.
|
||||||
|
|
||||||
|
### What this catches that review does not
|
||||||
|
|
||||||
|
Four read-only passes over the two tutorials in `docs/` found real gaps and
|
||||||
|
missed all three of these, which the first build-and-run found immediately:
|
||||||
|
|
||||||
|
| Defect | Why reading missed it |
|
||||||
|
|---|---|
|
||||||
|
| libakstdlib's header was never named -- `aksl_*` functions, `<akstdlib.h>` file | Every reviewer already knew, or did not have to write the `#include` |
|
||||||
|
| The asset directory defaulted to `"."` instead of the compiled-in macro | Prose said "falls back", and nobody had to run it from another directory |
|
||||||
|
| A published example output the chapter could not produce | The capture happened after teardown released the pool. Nothing fails to compile |
|
||||||
|
|
||||||
|
The third is the shape to remember: **a tutorial that prints an expected result
|
||||||
|
is making a claim, and a claim nobody executed is a claim that is probably
|
||||||
|
wrong.**
|
||||||
|
|
||||||
## Commit & Pull Request Guidelines
|
## Commit & Pull Request Guidelines
|
||||||
|
|
||||||
Recent commits use concise, imperative summaries such as `Fix a scale bug...` and often explain related changes in one sentence. Keep each commit scoped and describe user-visible behavior. Pull requests should summarize the change, identify affected modules, list the CMake/CTest commands run, and link relevant issues. Include screenshots only for rendering, map, or `charviewer` changes.
|
Recent commits use concise, imperative summaries such as `Fix a scale bug...` and often explain related changes in one sentence. Keep each commit scoped and describe user-visible behavior. Pull requests should summarize the change, identify affected modules, list the CMake/CTest commands run, and link relevant issues. Include screenshots only for rendering, map, or `charviewer` changes.
|
||||||
|
|||||||
@@ -4,6 +4,13 @@ This file documents the harness that keeps `docs/` honest. It is not a chapter
|
|||||||
a reader learning libakgl never needs it — and it is checked by the same harness
|
a reader learning libakgl never needs it — and it is checked by the same harness
|
||||||
it describes, so the file that publishes the convention is held to it.
|
it describes, so the file that publishes the convention is held to it.
|
||||||
|
|
||||||
|
**The harness checks that every listing is true. It cannot check that a chapter
|
||||||
|
teaches.** For that, the tutorials get a second test: the chapter is handed to a
|
||||||
|
subagent on a weaker model with the library, the headers and the art but *not*
|
||||||
|
the finished program, and that reader has to build and run the game. See
|
||||||
|
`AGENTS.md`, "Testing a Tutorial". Four read-only reviews of chapters 20 and 21
|
||||||
|
missed three defects the first build-and-run found in minutes.
|
||||||
|
|
||||||
## Documentation examples
|
## Documentation examples
|
||||||
|
|
||||||
**Every fenced block in `docs/*.md` is compiled, linked, run, or matched against
|
**Every fenced block in `docs/*.md` is compiled, linked, run, or matched against
|
||||||
|
|||||||
Reference in New Issue
Block a user