andrew/akbasic
1
0
Fork 0
Code Issues 26 Pull Requests 1 Actions 1 Packages Projects Releases Wiki Activity

Add mutation testing, and split the release gates into their own workflow

Browse Source 
Ports libakstdlib's mutation harness (the newest of the three) to scripts/
mutation_test.py, adds the namespaced `mutation` CMake target the sibling
libraries have, and splits CI in two.

.gitea/workflows/ci.yaml keeps the push path: suite, ASan+UBSan, coverage, and a
mutation run bounded to src/convert.c and src/symtab.c -- about four minutes,
scoring 77.8% against a gate of 65.

.gitea/workflows/release.yaml is new and manual (workflow_dispatch). It carries
the doxygen gate, moved out of ci.yaml, and a whole-tree mutation run: 3675
mutants and hours of runner time, which is a release cost rather than a
per-commit one. Both artifacts a release wants -- api-documentation and
mutation-report -- come out of it. Two optional inputs narrow the run or change
the threshold; they arrive through the environment rather than being
interpolated into the shell, because ${{ }} substitution happens before the
shell sees the line.

The harness paid for itself immediately, which is the point of it. Three real
gaps, each checked to be a genuine bug rather than an equivalent mutant:

  - errno was never asserted clear before a strtoll. Confirmed with a standalone
    probe that strtoll leaves a stale errno untouched on success, so without the
    `errno = 0` a valid conversion raises ERANGE.
  - Nothing exercised a maximum-length symbol-table key, so every MAX_KEY - 1
    off-by-one in a strncpy and its NUL terminator survived. The same hole exists
    for strings in src/value.c and is filed.
  - Nothing asserted a freshly initialised table was actually zeroed.

Closing the first two took the measured score from 73.1% to 77.8%.

I set the push-path threshold to 75 first, on an estimate. Measuring gave 73.1%
and the job would have failed on its first run -- the earlier per-file figure was
too high because the captured output had been truncated to its last lines and I
counted fewer survivors than there were. It is 65 now, and the gate was run as
written and confirmed to exit 0.

src/value.c is the file most worth mutating and is deliberately off the push
path: 368 mutants at ~11s each is about 70 minutes, because almost everything
links against it. A partial run over it found the same maximum-length-string
hole plus two genuinely equivalent mutants that only exist because of reference
defect section 6 item 5 -- adding both of the right operand's numeric fields
works only while the unused one is zero, so + and - are interchangeable there.
Recorded in TODO.md.

ctest 61/61; doxygen exits 0; both workflows' steps were executed locally, both
input paths included.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Andrew Kesterson
2026-07-31 07:46:53 -04:00
parent 2ddee571a1
commit 506af093a1
8 changed files with 828 additions and 48 deletions
Download Patch File Download Diff File Expand all files Collapse all files

105
.gitea/workflows/ci.yaml
View File

@@ -2,6 +2,10 @@ name: akbasic CI Build
run-name: ${{ gitea.actor }} akbasic test
on: [push]
# Push-triggered checks only. The doxygen gate and the whole-tree mutation run
# live in release.yaml, which is manual (workflow_dispatch) because between them
# they cost hours of runner time and are release gates rather than per-commit
# ones.
jobs:
cmake_build:
runs-on: ubuntu-latest
@@ -52,40 +56,6 @@ jobs:
fail_on_failure: 'true'
- run: echo "🍏 This job's status is ${{ job.status }}."
docs:
runs-on: ubuntu-latest
steps:
- name: Check out repository code
uses: actions/checkout@v4
# No submodules at all. Doxyfile's INPUT is include/akbasic, src and
# examples, none of which live under deps/, and doxygen does not need to
# resolve <akerror.h> to parse a declaration. Verified by running it
# against a tree with no deps/ present.
- name: dependencies
run: |
sudo apt-get update -y
sudo apt-get install -y doxygen graphviz
# This is a gate, not a convenience. The Doxyfile sets
# WARN_AS_ERROR = FAIL_ON_WARNINGS, so a doc block that documents some of a
# function's parameters but not all of them fails the build. All 114 public
# declarations under include/akbasic carry a @brief, a @param per
# parameter, a @return and their @throws; keep it that way.
#
# OUTPUT_DIRECTORY is build/docs and doxygen will not create a missing
# parent, so make it first -- there is no cmake step in this job to do it.
- name: build API documentation
run: |
mkdir -p build
doxygen Doxyfile
- name: upload API documentation
if: always()
uses: actions/upload-artifact@v4
with:
name: api-documentation
path: build/docs/html/
if-no-files-found: error
- run: echo "🍏 This job's status is ${{ job.status }}."
sanitizers:
runs-on: ubuntu-latest
steps:
@@ -159,3 +129,70 @@ jobs:
path: build-coverage/coverage/
if-no-files-found: warn
- run: echo "🍏 This job's status is ${{ job.status }}."
mutation_test:
runs-on: ubuntu-latest
steps:
- name: Check out repository code
uses: actions/checkout@v4
with:
# The harness copies the repo and configures a build inside the copy,
# so it needs the same submodules the main build does -- including
# deps/basicinterpret, because the golden corpus is part of what kills
# mutants.
submodules: true
- name: dependencies
run: |
sudo apt-get update -y
sudo apt-get install -y cmake gcc python3
# Verify the tests actually catch bugs: break the library many ways and
# confirm the suite fails. This matters more here than in an ordinary C
# library, because the akerror control-flow macros expand at their call
# sites -- gcov attributes ATTEMPT/CATCH/PASS to the caller, so coverage
# cannot see them and mutation testing is the only thing that checks them.
#
# Bounded to two small, fast, deterministic files. src/value.c is the file
# most worth mutating and is deliberately *not* here: 368 mutants at ~11s
# each is about 70 minutes, because almost everything links against it.
# Run the default target locally for the whole tree:
# cmake --build build --target mutation
#
# The threshold is a ratchet, not a quality bar. The measured score for
# these two files is 77.8% (84 killed, 24 survived, 108 total); 65 leaves
# headroom for runner variance while still failing on a real regression --
# a test deleted, or new untested code added.
#
# It was 73.1% before writing this job. The run's own findings closed the
# gap: nothing exercised a maximum-length symbol-table key, so every
# `MAX_KEY - 1` off-by-one in a strncpy survived, and nothing asserted a
# freshly initialised table was actually zeroed. Adding those two
# assertions to tests/symtab.c killed five mutants. The same run found that
# errno was never asserted clear before a strtoll, which is what stops a
# stale ERANGE from failing a valid conversion -- that is now in
# tests/convert.c.
#
# The 24 remaining survivors are listed in the published report. Most are
# ICR mutants on loop and accumulator initialisers that a stronger
# placement assertion would catch; three in convert.c are genuinely
# equivalent and cannot be killed. Recorded in TODO.md.
- name: mutation testing
run: |
python3 scripts/mutation_test.py \
--target src/convert.c \
--target src/symtab.c \
--junit mutation-junit.xml \
--threshold 65
# Publish even when the threshold gate fails, so survivors are visible --
# each one is a missing test. Display-only (fail_on_failure: false); the
# --threshold above is the gate. annotate_only avoids the Checks API 404
# on Gitea (mikepenz/action-junit-report#23).
- name: publish mutation results
if: always()
uses: mikepenz/action-junit-report@v4
with:
report_paths: 'mutation-junit.xml'
annotate_only: true
detailed_summary: true
include_passed: true
fail_on_failure: 'false'
- run: echo "🍏 This job's status is ${{ job.status }}."

136
.gitea/workflows/release.yaml Normal file
View File

@@ -0,0 +1,136 @@
name: akbasic Release Build
run-name: ${{ gitea.actor }} akbasic release checks
# Manual only. Nothing here runs on push: the full mutation set is thousands of
# mutants and hours of runner time, which is a release-gate cost, not a
# per-commit one. Trigger it from the Actions tab.
on:
workflow_dispatch:
inputs:
mutation_threshold:
description: "Fail if the mutation score falls below this percentage"
required: false
default: "65"
mutation_targets:
description: "Space-separated files to mutate; empty means the whole src/ tree"
required: false
default: ""
jobs:
# Moved here from ci.yaml. The docs are wanted for a release, not on every
# push, and building them beside the release mutation run keeps the two
# artefacts a release needs in one place.
docs:
runs-on: ubuntu-latest
steps:
- name: Check out repository code
uses: actions/checkout@v4
# No submodules at all. Doxyfile's INPUT is include/akbasic, src and
# examples, none of which live under deps/, and doxygen does not need to
# resolve <akerror.h> to parse a declaration. Verified by running it
# against a tree with no deps/ present.
- name: dependencies
run: |
sudo apt-get update -y
sudo apt-get install -y doxygen graphviz
# This is a gate, not a convenience. The Doxyfile sets
# WARN_AS_ERROR = FAIL_ON_WARNINGS, so a doc block that documents some of a
# function's parameters but not all of them fails the build. All 114 public
# declarations under include/akbasic carry a @brief, a @param per
# parameter, a @return and their @throws; keep it that way.
#
# OUTPUT_DIRECTORY is build/docs and doxygen will not create a missing
# parent, so make it first -- there is no cmake step in this job to do it.
- name: build API documentation
run: |
mkdir -p build
doxygen Doxyfile
- name: upload API documentation
if: always()
uses: actions/upload-artifact@v4
with:
name: api-documentation
path: build/docs/html/
if-no-files-found: error
- run: echo "🍏 This job's status is ${{ job.status }}."
full_mutation:
runs-on: ubuntu-latest
# 3675 mutants across the whole src/ tree. Cost per mutant is one incremental
# rebuild plus one full ctest run, which measures at roughly 2s for a leaf
# file and 11s for src/value.c, where almost everything links against it.
# Budget most of a day and expect it to finish well inside that; the ceiling
# exists so a mutant that wedges the runner cannot hold it forever.
timeout-minutes: 720
steps:
- name: Check out repository code
uses: actions/checkout@v4
with:
# The harness copies the repo and configures a build inside the copy,
# so it needs the same submodules the main build does -- including
# deps/basicinterpret, because the golden corpus is part of what kills
# mutants.
submodules: true
- name: dependencies
run: |
sudo apt-get update -y
sudo apt-get install -y cmake gcc python3
# The whole akbasic-owned src/ tree. ci.yaml runs a two-file subset on every
# push; this is the one that actually covers the interpreter.
#
# Mutation testing is the only gate that sees the error-handling control
# flow at all: libakerror's ATTEMPT/CATCH/PASS macros expand at their call
# sites, so gcov attributes them to the caller and line coverage cannot
# measure them.
#
# The default threshold matches ci.yaml's rather than being stricter.
# Whole-tree coverage is uneven -- src/convert.c and src/symtab.c are the
# two measured files, at 77.8% between them, and the rest is unmeasured, so
# a tighter number here would be a guess. Raise it with the workflow input
# once a full run has established a real baseline.
#
# The two inputs arrive through the environment rather than being
# interpolated straight into the script. ${{ }} substitution happens before
# the shell sees the line, so a value containing shell metacharacters would
# otherwise run as code. This workflow is manual and owner-triggered, but
# the safe form costs nothing.
- name: mutation testing (full tree)
env:
MUTATION_TARGETS: ${{ gitea.event.inputs.mutation_targets }}
MUTATION_THRESHOLD: ${{ gitea.event.inputs.mutation_threshold }}
run: |
set -eu
# Word-splitting is the point here: the input is a space-separated
# list. An empty input leaves $targets empty and the harness falls
# through to its own default, which is the whole src/ tree.
targets=""
for f in ${MUTATION_TARGETS:-}; do
targets="$targets --target $f"
done
# shellcheck disable=SC2086
python3 scripts/mutation_test.py \
$targets \
--junit mutation-junit.xml \
--threshold "${MUTATION_THRESHOLD:-65}"
# Publish even when the threshold gate fails, so survivors are visible --
# each one is a missing test. Display-only (fail_on_failure: false); the
# --threshold above is the gate. annotate_only avoids the Checks API 404
# on Gitea (mikepenz/action-junit-report#23).
- name: publish mutation results
if: always()
uses: mikepenz/action-junit-report@v4
with:
report_paths: 'mutation-junit.xml'
annotate_only: true
detailed_summary: true
include_passed: true
fail_on_failure: 'false'
# Keep the raw report as well as the annotations: a release wants the
# survivor list on file, and the job summary is not durable.
- name: upload mutation report
if: always()
uses: actions/upload-artifact@v4
with:
name: mutation-report
path: mutation-junit.xml
if-no-files-found: warn
- run: echo "🍏 This job's status is ${{ job.status }}."