Execute every documented example as a test
Some checks failed
akbasic CI Build / cmake_build (push) Successful in 3m2s
akbasic CI Build / sanitizers (push) Successful in 3m52s
akbasic CI Build / coverage (push) Failing after 3m24s
akbasic CI Build / akgl_build (push) Failing after 20s
akbasic CI Build / mutation_test (push) Has been cancelled

docs/ and README.md carry 85 fenced blocks. Every one was checked by hand
exactly once, when it was written, which is not a standard that survives a
changing interpreter -- and four were already wrong: two transcripts showing a
leading space PRINT does not emit, akbasic_TextSink in README.md missing the
two members it had grown hours earlier, and FILTER's refusal quoted with
wording the code does not use.

tests/docs_examples.sh reads a fence-tag vocabulary and runs what it finds.
BASIC programs and transcripts run and are byte-compared against an `output`
block; C snippets compile with -fsyntax-only against the real include path,
which CMake writes out because it is transitive through akerror, akstdlib and
akgl; shell blocks run in a sandbox. Anything that would reconfigure the build
tree, hit the network or re-enter the suite is tagged norun with the reason in
MAINTENANCE.md, and the two cmake blocks stay hand-maintained by decision.

An untagged block is a failure rather than a default, and the pass line
reports what it executed by kind. Both exist because the way a harness like
this dies is by quietly matching nothing and passing -- which it duly did on
the first CTest run, where a generator expression evaluating to nothing still
contributed an empty argument that the script read as a filename. The count is
what caught it.

The excerpt check earns its own mention: a block tagged
`c excerpt=include/akbasic/sink.h` must still appear in that header, comments
and whitespace ignored. Compiling it would only redefine the type, so a
compile check could not have found the stale struct, and did not.

Registered as the CTest case docs_examples in both configurations. Fixing the
four wrong examples turned up two interpreter defects, fixed in the previous
commit and recorded in TODO.md section 8.

MAINTENANCE.md is new: the fence-tag reference, what to do when the case
fails, and the conventions that until now only existed inside source comments
-- the three test lists and how two of them invert "passed", the sorted verb
table, that a golden file is never edited to suit this interpreter, and that a
fix gets mutation-checked with a file copy rather than git checkout.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-07-31 22:41:36 -04:00
parent 6f49f6a7f2
commit 342e4c07da
29 changed files with 1303 additions and 103 deletions

View File

@@ -372,6 +372,59 @@ if(AKBASIC_WITH_AKGL)
SKIP_RETURN_CODE 77)
endif()
# ---------------------------------------------------------------------------
# The documentation's own examples.
#
# docs/ and README.md are full of programs, transcripts, C snippets and shell
# commands, and every one of them was checked by hand exactly once -- when it
# was written. Three were already wrong the day this was added, one of them a
# struct that had grown two members hours earlier. Documentation goes stale
# because the *code* moved, so this is an ordinary test case rather than
# something a docs-only CI job runs: it fails when the interpreter changes under
# the chapter, which is the case a path filter would miss.
#
# tests/docs_examples.sh reads the fence info strings; MAINTENANCE.md documents
# them.
#
# The C snippets compile against the real include path, which is transitive
# through akerror, akstdlib and -- in this configuration -- akgl. Writing it out
# for the script to read keeps that one source of truth: a hardcoded -I list
# here would rot exactly the way the documentation does.
set(AKBASIC_DOCS_CFLAGS_FILE "${CMAKE_CURRENT_BINARY_DIR}/docs_cflags.txt")
if(AKBASIC_WITH_AKGL)
set(AKBASIC_DOCS_CFLAGS_TARGET akbasic_akgl)
else()
set(AKBASIC_DOCS_CFLAGS_TARGET akbasic)
endif()
file(GENERATE
OUTPUT "${AKBASIC_DOCS_CFLAGS_FILE}"
CONTENT "-I$<JOIN:$<TARGET_PROPERTY:${AKBASIC_DOCS_CFLAGS_TARGET},INCLUDE_DIRECTORIES>,\n-I>\n"
)
# Spelled out with if() rather than $<$<BOOL:...>:--akgl>, because a
# generator expression that evaluates to nothing still contributes an *empty
# argument*. The script read that empty string as the first filename, checked
# no documents at all, and passed -- caught only because it reports what it ran.
set(AKBASIC_DOCS_ARGS
--root "${CMAKE_CURRENT_SOURCE_DIR}"
--basic $<TARGET_FILE:basic>
--cflags-file "${AKBASIC_DOCS_CFLAGS_FILE}")
if(AKBASIC_WITH_AKGL)
list(APPEND AKBASIC_DOCS_ARGS --akgl)
endif()
_add_test(NAME docs_examples
COMMAND ${CMAKE_CURRENT_SOURCE_DIR}/tests/docs_examples.sh ${AKBASIC_DOCS_ARGS})
_set_tests_properties(docs_examples PROPERTIES
WORKING_DIRECTORY "${CMAKE_CURRENT_SOURCE_DIR}"
TIMEOUT 300)
if(AKBASIC_WITH_AKGL)
# Same reason as the golden cases: an AKGL `basic` opens a window, and the
# chapters on graphics, sound and sprites are most of what runs here.
_set_tests_properties(docs_examples PROPERTIES
ENVIRONMENT "SDL_VIDEODRIVER=dummy;SDL_AUDIODRIVER=dummy;SDL_RENDER_DRIVER=software")
endif()
if(AKBASIC_TESTS OR AKBASIC_WILL_FAIL_TESTS OR AKBASIC_KNOWN_FAILING_TESTS)
_set_tests_properties(
${AKBASIC_TESTS} ${AKBASIC_WILL_FAIL_TESTS} ${AKBASIC_KNOWN_FAILING_TESTS}