Ports libakstdlib's mutation harness (the newest of the three) to scripts/
mutation_test.py, adds the namespaced `mutation` CMake target the sibling
libraries have, and splits CI in two.
.gitea/workflows/ci.yaml keeps the push path: suite, ASan+UBSan, coverage, and a
mutation run bounded to src/convert.c and src/symtab.c -- about four minutes,
scoring 77.8% against a gate of 65.
.gitea/workflows/release.yaml is new and manual (workflow_dispatch). It carries
the doxygen gate, moved out of ci.yaml, and a whole-tree mutation run: 3675
mutants and hours of runner time, which is a release cost rather than a
per-commit one. Both artifacts a release wants -- api-documentation and
mutation-report -- come out of it. Two optional inputs narrow the run or change
the threshold; they arrive through the environment rather than being
interpolated into the shell, because ${{ }} substitution happens before the
shell sees the line.
The harness paid for itself immediately, which is the point of it. Three real
gaps, each checked to be a genuine bug rather than an equivalent mutant:
- errno was never asserted clear before a strtoll. Confirmed with a standalone
probe that strtoll leaves a stale errno untouched on success, so without the
`errno = 0` a valid conversion raises ERANGE.
- Nothing exercised a maximum-length symbol-table key, so every MAX_KEY - 1
off-by-one in a strncpy and its NUL terminator survived. The same hole exists
for strings in src/value.c and is filed.
- Nothing asserted a freshly initialised table was actually zeroed.
Closing the first two took the measured score from 73.1% to 77.8%.
I set the push-path threshold to 75 first, on an estimate. Measuring gave 73.1%
and the job would have failed on its first run -- the earlier per-file figure was
too high because the captured output had been truncated to its last lines and I
counted fewer survivors than there were. It is 65 now, and the gate was run as
written and confirmed to exit 0.
src/value.c is the file most worth mutating and is deliberately off the push
path: 368 mutants at ~11s each is about 70 minutes, because almost everything
links against it. A partial run over it found the same maximum-length-string
hole plus two genuinely equivalent mutants that only exist because of reference
defect section 6 item 5 -- adding both of the right operand's numeric fields
works only while the unused one is zero, so + and - are interchangeable there.
Recorded in TODO.md.
ctest 61/61; doxygen exits 0; both workflows' steps were executed locally, both
input paths included.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
309 lines
12 KiB
CMake
309 lines
12 KiB
CMake
cmake_minimum_required(VERSION 3.10)
|
|
# The single source of truth for the version, in the shape libakstdlib and
|
|
# libakgl both use. It flows into the library SOVERSION and nothing else spells
|
|
# a version number.
|
|
#
|
|
# 0.x on purpose: TODO.md section 12 records eleven defects carried over from the
|
|
# Go reference that are deliberately reproduced and not yet fixed, so the
|
|
# language surface is not being promised yet.
|
|
project(akbasic VERSION 0.1.0 LANGUAGES C)
|
|
|
|
# Pre-1.0 the ABI may break on a minor bump, so the soname carries MAJOR.MINOR.
|
|
# At 1.0 this becomes ${PROJECT_VERSION_MAJOR} alone.
|
|
if(PROJECT_VERSION_MAJOR EQUAL 0)
|
|
set(AKBASIC_SOVERSION "${PROJECT_VERSION_MAJOR}.${PROJECT_VERSION_MINOR}")
|
|
else()
|
|
set(AKBASIC_SOVERSION "${PROJECT_VERSION_MAJOR}")
|
|
endif()
|
|
|
|
include(CTest)
|
|
include(GNUInstallDirs)
|
|
|
|
option(AKBASIC_WITH_AKGL "Build the libakgl-backed text sink and link SDL3" OFF)
|
|
option(AKBASIC_BUILD_EXAMPLES "Build the embedding example in examples/" ON)
|
|
option(AKBASIC_COVERAGE "Instrument the build with gcov coverage counters" OFF)
|
|
option(AKBASIC_SANITIZE "Build with ASan + UBSan" OFF)
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Dependencies
|
|
#
|
|
# libakgl vendors its own copies of libakerror and libakstdlib and guards every
|
|
# dependency with if(NOT TARGET ...), so akerror::akerror and akstdlib::akstdlib
|
|
# must exist *before* add_subdirectory(deps/libakgl) or the targets are declared
|
|
# twice.
|
|
#
|
|
# All three dependencies register their CTest tests unconditionally. Pulled in
|
|
# EXCLUDE_FROM_ALL their test binaries are never built, so each one would land in
|
|
# our suite as "Not Run" and fail. CMake has no way to un-register a test and
|
|
# set_tests_properties cannot reach across directory scopes, so add_test() and
|
|
# set_tests_properties() are shadowed for the duration of the add_subdirectory()
|
|
# calls. Note libakstdlib carries the same shadow but only arms it when *it* is
|
|
# top-level, so it does nothing for us -- this one has to wrap all three.
|
|
#
|
|
# libakerror additionally namespaces its `mutation` target when embedded but not
|
|
# its `coverage` target (deps/libakerror/CMakeLists.txt:194 vs :172), so a
|
|
# coverage build collides on the `coverage` target and fails to configure at all.
|
|
# Rename the dependency's on the way past. Remove this once libakerror applies
|
|
# the same CMAKE_SOURCE_DIR test to `coverage` that it already applies to
|
|
# `mutation` -- filed in deps/libakstdlib/TODO.md section 2.3.
|
|
# ---------------------------------------------------------------------------
|
|
set(AKBASIC_SUPPRESS_ADD_TEST TRUE)
|
|
|
|
function(add_test)
|
|
if(NOT AKBASIC_SUPPRESS_ADD_TEST)
|
|
_add_test(${ARGV})
|
|
endif()
|
|
endfunction()
|
|
|
|
function(set_tests_properties)
|
|
if(NOT AKBASIC_SUPPRESS_ADD_TEST)
|
|
_set_tests_properties(${ARGV})
|
|
endif()
|
|
endfunction()
|
|
|
|
function(add_custom_target _name)
|
|
if(AKBASIC_SUPPRESS_ADD_TEST AND _name STREQUAL "coverage")
|
|
_add_custom_target(akerror_coverage ${ARGN})
|
|
else()
|
|
_add_custom_target(${ARGV})
|
|
endif()
|
|
endfunction()
|
|
|
|
add_subdirectory(deps/libakerror EXCLUDE_FROM_ALL)
|
|
add_subdirectory(deps/libakstdlib EXCLUDE_FROM_ALL)
|
|
if(AKBASIC_WITH_AKGL)
|
|
add_subdirectory(deps/libakgl EXCLUDE_FROM_ALL)
|
|
endif()
|
|
|
|
set(AKBASIC_SUPPRESS_ADD_TEST FALSE)
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Instrumentation, applied per target so it never leaks into an exported
|
|
# interface.
|
|
# ---------------------------------------------------------------------------
|
|
function(akbasic_instrument _target)
|
|
if(AKBASIC_COVERAGE)
|
|
target_compile_options(${_target} PRIVATE --coverage -O0 -g)
|
|
target_link_options(${_target} PRIVATE --coverage)
|
|
endif()
|
|
if(AKBASIC_SANITIZE)
|
|
target_compile_options(${_target} PRIVATE -fsanitize=address,undefined -fno-omit-frame-pointer -g)
|
|
target_link_options(${_target} PRIVATE -fsanitize=address,undefined)
|
|
endif()
|
|
endfunction()
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# The interpreter library. Everything here must be free of SDL, free of any
|
|
# process-terminating call, and buildable with no libakgl present.
|
|
# ---------------------------------------------------------------------------
|
|
set(AKBASIC_SOURCES
|
|
src/convert.c
|
|
src/environment.c
|
|
src/error.c
|
|
src/grammar.c
|
|
src/parser.c
|
|
src/parser_commands.c
|
|
src/runtime.c
|
|
src/runtime_commands.c
|
|
src/runtime_functions.c
|
|
src/scanner.c
|
|
src/sink_stdio.c
|
|
src/symtab.c
|
|
src/value.c
|
|
src/variable.c
|
|
src/verbs.c
|
|
)
|
|
|
|
add_library(akbasic ${AKBASIC_SOURCES})
|
|
add_library(akbasic::akbasic ALIAS akbasic)
|
|
|
|
target_include_directories(akbasic PUBLIC
|
|
$<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/include>
|
|
$<INSTALL_INTERFACE:${CMAKE_INSTALL_INCLUDEDIR}/>
|
|
)
|
|
|
|
target_compile_options(akbasic PRIVATE -Wall -Wextra)
|
|
target_link_libraries(akbasic PUBLIC akstdlib::akstdlib akerror::akerror)
|
|
target_link_libraries(akbasic PRIVATE m)
|
|
|
|
set_target_properties(akbasic PROPERTIES
|
|
VERSION ${PROJECT_VERSION}
|
|
SOVERSION ${AKBASIC_SOVERSION}
|
|
)
|
|
|
|
akbasic_instrument(akbasic)
|
|
|
|
# The libakgl-backed text sink, kept in its own target so the core library and
|
|
# the whole golden suite build with no SDL present.
|
|
if(AKBASIC_WITH_AKGL)
|
|
add_library(akbasic_akgl src/sink_akgl.c)
|
|
target_compile_options(akbasic_akgl PRIVATE -Wall -Wextra)
|
|
target_link_libraries(akbasic_akgl PUBLIC akbasic akgl)
|
|
akbasic_instrument(akbasic_akgl)
|
|
endif()
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# The standalone driver. This is the only place FINISH_NORETURN may appear.
|
|
# ---------------------------------------------------------------------------
|
|
add_executable(basic src/main.c)
|
|
target_compile_options(basic PRIVATE -Wall -Wextra)
|
|
target_link_libraries(basic PRIVATE akbasic)
|
|
if(AKBASIC_WITH_AKGL)
|
|
target_link_libraries(basic PRIVATE akbasic_akgl)
|
|
target_compile_definitions(basic PRIVATE AKBASIC_HAVE_AKGL=1)
|
|
endif()
|
|
akbasic_instrument(basic)
|
|
|
|
# The embedding example. It is built by default and registered as a test so the
|
|
# code README.md quotes cannot rot: a signature change breaks the build.
|
|
if(AKBASIC_BUILD_EXAMPLES)
|
|
foreach(_example IN ITEMS embed hostvars)
|
|
add_executable(akbasic_example_${_example} examples/${_example}.c)
|
|
target_compile_options(akbasic_example_${_example} PRIVATE -Wall -Wextra)
|
|
target_link_libraries(akbasic_example_${_example} PRIVATE akbasic)
|
|
akbasic_instrument(akbasic_example_${_example})
|
|
_add_test(NAME example_${_example} COMMAND akbasic_example_${_example})
|
|
endforeach()
|
|
endif()
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Tests.
|
|
#
|
|
# Three lists, and two of them invert the meaning of "Passed", exactly as
|
|
# libakstdlib does:
|
|
# AKBASIC_TESTS must exit 0
|
|
# AKBASIC_WILL_FAIL_TESTS abort by design
|
|
# AKBASIC_KNOWN_FAILING_TESTS assert the *correct* contract for a defect
|
|
# recorded in TODO.md and are expected to fail.
|
|
# When one starts passing CTest reports
|
|
# "unexpectedly passed" -- that is the cue to move
|
|
# it into AKBASIC_TESTS along with the fix.
|
|
#
|
|
# Test executables are named akbasic_test_<name>, never test_<name>: a
|
|
# dependency's targets are created even under EXCLUDE_FROM_ALL, and libakstdlib
|
|
# already ships test_version. The CTest name stays bare, so `ctest -R value`
|
|
# still selects it.
|
|
# ---------------------------------------------------------------------------
|
|
set(AKBASIC_TESTS
|
|
convert
|
|
environment_scope
|
|
error_codes
|
|
grammar_leaves
|
|
parser_commands
|
|
parser_expressions
|
|
runtime_evaluate
|
|
runtime_verbs
|
|
scanner_tokens
|
|
sink_stdio
|
|
symtab
|
|
value_arithmetic
|
|
value_bitwise
|
|
value_compare
|
|
variable_subscript
|
|
verbs_table
|
|
version_check
|
|
)
|
|
|
|
set(AKBASIC_WILL_FAIL_TESTS
|
|
)
|
|
|
|
set(AKBASIC_KNOWN_FAILING_TESTS
|
|
known_reference_defects
|
|
)
|
|
|
|
foreach(_test IN LISTS AKBASIC_TESTS AKBASIC_WILL_FAIL_TESTS AKBASIC_KNOWN_FAILING_TESTS)
|
|
add_executable(akbasic_test_${_test} tests/${_test}.c)
|
|
target_compile_options(akbasic_test_${_test} PRIVATE -Wall -Wextra)
|
|
target_link_libraries(akbasic_test_${_test} PRIVATE akbasic)
|
|
target_include_directories(akbasic_test_${_test} PRIVATE "${CMAKE_CURRENT_SOURCE_DIR}/tests")
|
|
akbasic_instrument(akbasic_test_${_test})
|
|
_add_test(NAME ${_test} COMMAND akbasic_test_${_test})
|
|
endforeach()
|
|
|
|
if(AKBASIC_TESTS OR AKBASIC_WILL_FAIL_TESTS OR AKBASIC_KNOWN_FAILING_TESTS)
|
|
_set_tests_properties(
|
|
${AKBASIC_TESTS} ${AKBASIC_WILL_FAIL_TESTS} ${AKBASIC_KNOWN_FAILING_TESTS}
|
|
PROPERTIES WORKING_DIRECTORY "${CMAKE_CURRENT_SOURCE_DIR}/tests" TIMEOUT 30
|
|
)
|
|
endif()
|
|
|
|
if(AKBASIC_WILL_FAIL_TESTS OR AKBASIC_KNOWN_FAILING_TESTS)
|
|
_set_tests_properties(
|
|
${AKBASIC_WILL_FAIL_TESTS} ${AKBASIC_KNOWN_FAILING_TESTS}
|
|
PROPERTIES WILL_FAIL TRUE
|
|
)
|
|
endif()
|
|
|
|
# The golden-file suite. It drives deps/basicinterpret/tests/**/*.bas in place --
|
|
# the corpus is a submodule and copying it guarantees drift -- and byte-compares
|
|
# stdout against the sibling .txt. One CTest case per .bas so a failure names the
|
|
# file.
|
|
file(GLOB_RECURSE AKBASIC_GOLDEN_CASES
|
|
RELATIVE "${CMAKE_CURRENT_SOURCE_DIR}/deps/basicinterpret"
|
|
"${CMAKE_CURRENT_SOURCE_DIR}/deps/basicinterpret/tests/*.bas"
|
|
)
|
|
|
|
foreach(_case IN LISTS AKBASIC_GOLDEN_CASES)
|
|
string(REGEX REPLACE "^tests/" "" _name "${_case}")
|
|
string(REGEX REPLACE "\\.bas$" "" _name "${_name}")
|
|
string(REPLACE "/" "_" _name "${_name}")
|
|
_add_test(
|
|
NAME golden_${_name}
|
|
COMMAND ${CMAKE_COMMAND}
|
|
-DBASIC=$<TARGET_FILE:basic>
|
|
-DCASE=${CMAKE_CURRENT_SOURCE_DIR}/deps/basicinterpret/${_case}
|
|
-P ${CMAKE_CURRENT_SOURCE_DIR}/tests/golden.cmake
|
|
)
|
|
endforeach()
|
|
|
|
if(AKBASIC_GOLDEN_CASES)
|
|
set(AKBASIC_GOLDEN_NAMES)
|
|
foreach(_case IN LISTS AKBASIC_GOLDEN_CASES)
|
|
string(REGEX REPLACE "^tests/" "" _name "${_case}")
|
|
string(REGEX REPLACE "\\.bas$" "" _name "${_name}")
|
|
string(REPLACE "/" "_" _name "${_name}")
|
|
list(APPEND AKBASIC_GOLDEN_NAMES golden_${_name})
|
|
endforeach()
|
|
_set_tests_properties(${AKBASIC_GOLDEN_NAMES} PROPERTIES TIMEOUT 30)
|
|
endif()
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Mutation testing.
|
|
#
|
|
# Coverage says which lines ran; mutation testing says whether anything would
|
|
# have noticed if they were wrong. That distinction matters more here than in an
|
|
# ordinary C library: the akerror control-flow macros expand at their call sites,
|
|
# so gcov attributes ATTEMPT/CATCH/PASS to the caller and cannot really see them.
|
|
# Mutation testing is the only thing that checks them at all.
|
|
#
|
|
# This target runs the whole akbasic-owned src/ tree and is slow -- upwards of an
|
|
# hour. CI runs a narrower, faster set with a --threshold gate; see
|
|
# .gitea/workflows/ci.yaml.
|
|
#
|
|
# Namespaced when embedded in another project, for the same reason the coverage
|
|
# targets in the dependencies are: a sibling may well ship a `mutation` target.
|
|
find_package(Python3 COMPONENTS Interpreter)
|
|
if(Python3_FOUND)
|
|
if(CMAKE_SOURCE_DIR STREQUAL CMAKE_CURRENT_SOURCE_DIR)
|
|
set(AKBASIC_MUTATION_TARGET mutation)
|
|
else()
|
|
set(AKBASIC_MUTATION_TARGET akbasic_mutation)
|
|
endif()
|
|
add_custom_target(${AKBASIC_MUTATION_TARGET}
|
|
COMMAND ${Python3_EXECUTABLE}
|
|
${CMAKE_CURRENT_SOURCE_DIR}/scripts/mutation_test.py
|
|
--source-root ${CMAKE_CURRENT_SOURCE_DIR}
|
|
WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}
|
|
USES_TERMINAL
|
|
COMMENT "Running mutation tests (breaks the library, expects tests to fail)"
|
|
)
|
|
endif()
|
|
|
|
install(TARGETS akbasic basic
|
|
ARCHIVE DESTINATION ${CMAKE_INSTALL_LIBDIR}
|
|
LIBRARY DESTINATION ${CMAKE_INSTALL_LIBDIR}
|
|
RUNTIME DESTINATION ${CMAKE_INSTALL_BINDIR}
|
|
INCLUDES DESTINATION ${CMAKE_INSTALL_INCLUDEDIR}
|
|
)
|
|
install(DIRECTORY include/akbasic DESTINATION ${CMAKE_INSTALL_INCLUDEDIR})
|