Move BASIC fixtures into the editable language corpus
All checks were successful
akbasic CI Build / cmake_build (push) Successful in 3m34s
akbasic CI Build / coverage (push) Successful in 4m4s
akbasic CI Build / sanitizers (push) Successful in 6m59s
akbasic CI Build / akgl_build (push) Successful in 7m57s
akbasic CI Build / mutation_test (push) Successful in 23m28s
All checks were successful
akbasic CI Build / cmake_build (push) Successful in 3m34s
akbasic CI Build / coverage (push) Successful in 4m4s
akbasic CI Build / sanitizers (push) Successful in 6m59s
akbasic CI Build / akgl_build (push) Successful in 7m57s
akbasic CI Build / mutation_test (push) Successful in 23m28s
Move every program and expectation out of tests/reference and register the unified tests/language corpus as local cases. Remove the old immutable-corpus protections from build, maintenance, and documentation paths. Co-authored-by: andrew <andrew@aklabs.net>
This commit is contained in:
@@ -448,7 +448,7 @@ if(AKBASIC_WITH_AKGL)
|
||||
# against.
|
||||
#
|
||||
# **A byte comparison of a rendered PNG is a deliberate bet**, the same bet
|
||||
# tests/reference/ already makes about golden output: that the dummy video
|
||||
# the language corpus makes about golden output: that the dummy video
|
||||
# driver and the software renderer are reproducible. They are, run to run and
|
||||
# build to build. What is untested is an SDL upgrade that shifts one pixel of
|
||||
# a diagonal, and the answer to that is to regenerate the figures in the same
|
||||
@@ -531,63 +531,20 @@ if(AKBASIC_WILL_FAIL_TESTS OR AKBASIC_KNOWN_FAILING_TESTS)
|
||||
)
|
||||
endif()
|
||||
|
||||
# The reference's own corpus, byte-compared against the sibling .txt. One CTest
|
||||
# case per .bas so a failure names the file.
|
||||
# The editable language corpus. One CTest case per .bas so a failure names the
|
||||
# file. Each program is paired with a sibling .txt expectation; see
|
||||
# tests/language/README.md for the editing rule.
|
||||
#
|
||||
# It used to be driven in place out of deps/basicinterpret, on the reasoning that
|
||||
# copying a submodule's corpus guarantees drift. That reasoning was sound and it
|
||||
# has been overruled deliberately: the Go dependency is being deprecated, and a
|
||||
# build that cannot run its acceptance suite without cloning the implementation
|
||||
# it replaced is not finished. The copy is byte-identical to
|
||||
# basicinterpreter@d76162c and tests/reference/README.md records
|
||||
# where it came from and what the drift now costs.
|
||||
file(GLOB_RECURSE AKBASIC_GOLDEN_CASES
|
||||
RELATIVE "${CMAKE_CURRENT_SOURCE_DIR}/tests/reference"
|
||||
"${CMAKE_CURRENT_SOURCE_DIR}/tests/reference/*.bas"
|
||||
)
|
||||
|
||||
foreach(_case IN LISTS AKBASIC_GOLDEN_CASES)
|
||||
string(REGEX REPLACE "^tests/" "" _name "${_case}")
|
||||
string(REGEX REPLACE "\\.bas$" "" _name "${_name}")
|
||||
string(REPLACE "/" "_" _name "${_name}")
|
||||
_add_test(
|
||||
NAME golden_${_name}
|
||||
COMMAND ${CMAKE_COMMAND}
|
||||
-DBASIC=$<TARGET_FILE:basic>
|
||||
-DCASE=${CMAKE_CURRENT_SOURCE_DIR}/tests/reference/${_case}
|
||||
-P ${CMAKE_CURRENT_SOURCE_DIR}/tests/golden.cmake
|
||||
)
|
||||
endforeach()
|
||||
|
||||
if(AKBASIC_GOLDEN_CASES)
|
||||
set(AKBASIC_GOLDEN_NAMES)
|
||||
foreach(_case IN LISTS AKBASIC_GOLDEN_CASES)
|
||||
string(REGEX REPLACE "^tests/" "" _name "${_case}")
|
||||
string(REGEX REPLACE "\\.bas$" "" _name "${_name}")
|
||||
string(REPLACE "/" "_" _name "${_name}")
|
||||
list(APPEND AKBASIC_GOLDEN_NAMES golden_${_name})
|
||||
endforeach()
|
||||
_set_tests_properties(${AKBASIC_GOLDEN_NAMES} PROPERTIES TIMEOUT 30)
|
||||
# An AKGL build of `basic` opens a window, and forty-one of them is not what
|
||||
# anybody running the suite wanted. The dummy driver produces the same stdout,
|
||||
# which is the only thing a golden case compares.
|
||||
if(AKBASIC_WITH_AKGL)
|
||||
_set_tests_properties(${AKBASIC_GOLDEN_NAMES} PROPERTIES
|
||||
ENVIRONMENT "SDL_VIDEODRIVER=dummy;SDL_AUDIODRIVER=dummy;SDL_RENDER_DRIVER=software")
|
||||
endif()
|
||||
endif()
|
||||
|
||||
# The local golden corpus, for verbs the reference never implemented.
|
||||
# The corpus includes programs carried over from the deprecated Go implementation
|
||||
# and cases written for this interpreter. Their provenance is useful when
|
||||
# investigating a regression, but it does not make any case immutable.
|
||||
#
|
||||
# Still separate now that the reference's corpus lives in this repository too,
|
||||
# and the reason changed rather than went away: tests/reference/ is a *record* of
|
||||
# what the Go implementation did and nothing in it should ever be edited to suit
|
||||
# this one, while tests/language/ is ours to change. Registered under local_ so a
|
||||
# failure says at a glance which of the two it came from -- and so a diff that
|
||||
# touches tests/reference/ stands out as the thing it is.
|
||||
# Programs carried over from the deprecated Go implementation and cases written
|
||||
# for this interpreter use the same editable `.bas`/`.txt` contract. Registered
|
||||
# under local_ so every failure identifies the program that produced it.
|
||||
#
|
||||
# Note what this can and cannot cover. The graphics and sound verbs draw and play
|
||||
# rather than print, so what a golden file sees of them is their *refusals* and
|
||||
# rather than print, so what a golden file sees of them is their refusals and
|
||||
# whatever a program can PRINT about the state they changed. The behaviour that
|
||||
# reaches a device is asserted against tests/mockdevice.h instead.
|
||||
file(GLOB_RECURSE AKBASIC_LOCAL_CASES
|
||||
|
||||
Reference in New Issue
Block a user