Execute every documented example as a test
Some checks failed
akbasic CI Build / cmake_build (push) Successful in 3m2s
akbasic CI Build / sanitizers (push) Successful in 3m52s
akbasic CI Build / coverage (push) Failing after 3m24s
akbasic CI Build / akgl_build (push) Failing after 20s
akbasic CI Build / mutation_test (push) Has been cancelled
Some checks failed
akbasic CI Build / cmake_build (push) Successful in 3m2s
akbasic CI Build / sanitizers (push) Successful in 3m52s
akbasic CI Build / coverage (push) Failing after 3m24s
akbasic CI Build / akgl_build (push) Failing after 20s
akbasic CI Build / mutation_test (push) Has been cancelled
docs/ and README.md carry 85 fenced blocks. Every one was checked by hand exactly once, when it was written, which is not a standard that survives a changing interpreter -- and four were already wrong: two transcripts showing a leading space PRINT does not emit, akbasic_TextSink in README.md missing the two members it had grown hours earlier, and FILTER's refusal quoted with wording the code does not use. tests/docs_examples.sh reads a fence-tag vocabulary and runs what it finds. BASIC programs and transcripts run and are byte-compared against an `output` block; C snippets compile with -fsyntax-only against the real include path, which CMake writes out because it is transitive through akerror, akstdlib and akgl; shell blocks run in a sandbox. Anything that would reconfigure the build tree, hit the network or re-enter the suite is tagged norun with the reason in MAINTENANCE.md, and the two cmake blocks stay hand-maintained by decision. An untagged block is a failure rather than a default, and the pass line reports what it executed by kind. Both exist because the way a harness like this dies is by quietly matching nothing and passing -- which it duly did on the first CTest run, where a generator expression evaluating to nothing still contributed an empty argument that the script read as a filename. The count is what caught it. The excerpt check earns its own mention: a block tagged `c excerpt=include/akbasic/sink.h` must still appear in that header, comments and whitespace ignored. Compiling it would only redefine the type, so a compile check could not have found the stale struct, and did not. Registered as the CTest case docs_examples in both configurations. Fixing the four wrong examples turned up two interpreter defects, fixed in the previous commit and recorded in TODO.md section 8. MAINTENANCE.md is new: the fence-tag reference, what to do when the case fails, and the conventions that until now only existed inside source comments -- the three test lists and how two of them invert "passed", the sorted verb table, that a golden file is never edited to suit this interpreter, and that a fix gets mutation-checked with a file copy rather than git checkout. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -372,6 +372,59 @@ if(AKBASIC_WITH_AKGL)
|
||||
SKIP_RETURN_CODE 77)
|
||||
endif()
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# The documentation's own examples.
|
||||
#
|
||||
# docs/ and README.md are full of programs, transcripts, C snippets and shell
|
||||
# commands, and every one of them was checked by hand exactly once -- when it
|
||||
# was written. Three were already wrong the day this was added, one of them a
|
||||
# struct that had grown two members hours earlier. Documentation goes stale
|
||||
# because the *code* moved, so this is an ordinary test case rather than
|
||||
# something a docs-only CI job runs: it fails when the interpreter changes under
|
||||
# the chapter, which is the case a path filter would miss.
|
||||
#
|
||||
# tests/docs_examples.sh reads the fence info strings; MAINTENANCE.md documents
|
||||
# them.
|
||||
#
|
||||
# The C snippets compile against the real include path, which is transitive
|
||||
# through akerror, akstdlib and -- in this configuration -- akgl. Writing it out
|
||||
# for the script to read keeps that one source of truth: a hardcoded -I list
|
||||
# here would rot exactly the way the documentation does.
|
||||
set(AKBASIC_DOCS_CFLAGS_FILE "${CMAKE_CURRENT_BINARY_DIR}/docs_cflags.txt")
|
||||
if(AKBASIC_WITH_AKGL)
|
||||
set(AKBASIC_DOCS_CFLAGS_TARGET akbasic_akgl)
|
||||
else()
|
||||
set(AKBASIC_DOCS_CFLAGS_TARGET akbasic)
|
||||
endif()
|
||||
file(GENERATE
|
||||
OUTPUT "${AKBASIC_DOCS_CFLAGS_FILE}"
|
||||
CONTENT "-I$<JOIN:$<TARGET_PROPERTY:${AKBASIC_DOCS_CFLAGS_TARGET},INCLUDE_DIRECTORIES>,\n-I>\n"
|
||||
)
|
||||
|
||||
# Spelled out with if() rather than $<$<BOOL:...>:--akgl>, because a
|
||||
# generator expression that evaluates to nothing still contributes an *empty
|
||||
# argument*. The script read that empty string as the first filename, checked
|
||||
# no documents at all, and passed -- caught only because it reports what it ran.
|
||||
set(AKBASIC_DOCS_ARGS
|
||||
--root "${CMAKE_CURRENT_SOURCE_DIR}"
|
||||
--basic $<TARGET_FILE:basic>
|
||||
--cflags-file "${AKBASIC_DOCS_CFLAGS_FILE}")
|
||||
if(AKBASIC_WITH_AKGL)
|
||||
list(APPEND AKBASIC_DOCS_ARGS --akgl)
|
||||
endif()
|
||||
|
||||
_add_test(NAME docs_examples
|
||||
COMMAND ${CMAKE_CURRENT_SOURCE_DIR}/tests/docs_examples.sh ${AKBASIC_DOCS_ARGS})
|
||||
_set_tests_properties(docs_examples PROPERTIES
|
||||
WORKING_DIRECTORY "${CMAKE_CURRENT_SOURCE_DIR}"
|
||||
TIMEOUT 300)
|
||||
if(AKBASIC_WITH_AKGL)
|
||||
# Same reason as the golden cases: an AKGL `basic` opens a window, and the
|
||||
# chapters on graphics, sound and sprites are most of what runs here.
|
||||
_set_tests_properties(docs_examples PROPERTIES
|
||||
ENVIRONMENT "SDL_VIDEODRIVER=dummy;SDL_AUDIODRIVER=dummy;SDL_RENDER_DRIVER=software")
|
||||
endif()
|
||||
|
||||
if(AKBASIC_TESTS OR AKBASIC_WILL_FAIL_TESTS OR AKBASIC_KNOWN_FAILING_TESTS)
|
||||
_set_tests_properties(
|
||||
${AKBASIC_TESTS} ${AKBASIC_WILL_FAIL_TESTS} ${AKBASIC_KNOWN_FAILING_TESTS}
|
||||
|
||||
Reference in New Issue
Block a user