andrew/akbasic
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add mutation testing, and split the release gates into their own workflow

Browse Source 
Ports libakstdlib's mutation harness (the newest of the three) to scripts/
mutation_test.py, adds the namespaced `mutation` CMake target the sibling
libraries have, and splits CI in two.

.gitea/workflows/ci.yaml keeps the push path: suite, ASan+UBSan, coverage, and a
mutation run bounded to src/convert.c and src/symtab.c -- about four minutes,
scoring 77.8% against a gate of 65.

.gitea/workflows/release.yaml is new and manual (workflow_dispatch). It carries
the doxygen gate, moved out of ci.yaml, and a whole-tree mutation run: 3675
mutants and hours of runner time, which is a release cost rather than a
per-commit one. Both artifacts a release wants -- api-documentation and
mutation-report -- come out of it. Two optional inputs narrow the run or change
the threshold; they arrive through the environment rather than being
interpolated into the shell, because ${{ }} substitution happens before the
shell sees the line.

The harness paid for itself immediately, which is the point of it. Three real
gaps, each checked to be a genuine bug rather than an equivalent mutant:

  - errno was never asserted clear before a strtoll. Confirmed with a standalone
    probe that strtoll leaves a stale errno untouched on success, so without the
    `errno = 0` a valid conversion raises ERANGE.
  - Nothing exercised a maximum-length symbol-table key, so every MAX_KEY - 1
    off-by-one in a strncpy and its NUL terminator survived. The same hole exists
    for strings in src/value.c and is filed.
  - Nothing asserted a freshly initialised table was actually zeroed.

Closing the first two took the measured score from 73.1% to 77.8%.

I set the push-path threshold to 75 first, on an estimate. Measuring gave 73.1%
and the job would have failed on its first run -- the earlier per-file figure was
too high because the captured output had been truncated to its last lines and I
counted fewer survivors than there were. It is 65 now, and the gate was run as
written and confirmed to exit 0.

src/value.c is the file most worth mutating and is deliberately off the push
path: 368 mutants at ~11s each is about 70 minutes, because almost everything
links against it. A partial run over it found the same maximum-length-string
hole plus two genuinely equivalent mutants that only exist because of reference
defect section 6 item 5 -- adding both of the right operand's numeric fields
works only while the unused one is zero, so + and - are interchangeable there.
Recorded in TODO.md.

ctest 61/61; doxygen exits 0; both workflows' steps were executed locally, both
input paths included.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Andrew Kesterson
2026-07-31 07:46:53 -04:00
parent 2ddee571a1
commit 506af093a1
8 changed files with 828 additions and 48 deletions
Download Patch File Download Diff File Expand all files Collapse all files

105
.gitea/workflows/ci.yaml
View File

@@ -2,6 +2,10 @@ name: akbasic CI Build
run-name: ${{ gitea.actor }} akbasic test
on: [push]
# Push-triggered checks only. The doxygen gate and the whole-tree mutation run
# live in release.yaml, which is manual (workflow_dispatch) because between them
# they cost hours of runner time and are release gates rather than per-commit
# ones.
jobs:
cmake_build:
runs-on: ubuntu-latest
@@ -52,40 +56,6 @@ jobs:
fail_on_failure: 'true'
- run: echo "🍏 This job's status is ${{ job.status }}."
docs:
runs-on: ubuntu-latest
steps:
- name: Check out repository code
uses: actions/checkout@v4
# No submodules at all. Doxyfile's INPUT is include/akbasic, src and
# examples, none of which live under deps/, and doxygen does not need to
# resolve <akerror.h> to parse a declaration. Verified by running it
# against a tree with no deps/ present.
- name: dependencies
run: |
sudo apt-get update -y
sudo apt-get install -y doxygen graphviz
# This is a gate, not a convenience. The Doxyfile sets
# WARN_AS_ERROR = FAIL_ON_WARNINGS, so a doc block that documents some of a
# function's parameters but not all of them fails the build. All 114 public
# declarations under include/akbasic carry a @brief, a @param per
# parameter, a @return and their @throws; keep it that way.
#
# OUTPUT_DIRECTORY is build/docs and doxygen will not create a missing
# parent, so make it first -- there is no cmake step in this job to do it.
- name: build API documentation
run: |
mkdir -p build
doxygen Doxyfile
- name: upload API documentation
if: always()
uses: actions/upload-artifact@v4
with:
name: api-documentation
path: build/docs/html/
if-no-files-found: error
- run: echo "🍏 This job's status is ${{ job.status }}."
sanitizers:
runs-on: ubuntu-latest
steps:
@@ -159,3 +129,70 @@ jobs:
path: build-coverage/coverage/
if-no-files-found: warn
- run: echo "🍏 This job's status is ${{ job.status }}."
mutation_test:
runs-on: ubuntu-latest
steps:
- name: Check out repository code
uses: actions/checkout@v4
with:
# The harness copies the repo and configures a build inside the copy,
# so it needs the same submodules the main build does -- including
# deps/basicinterpret, because the golden corpus is part of what kills
# mutants.
submodules: true
- name: dependencies
run: |
sudo apt-get update -y
sudo apt-get install -y cmake gcc python3
# Verify the tests actually catch bugs: break the library many ways and
# confirm the suite fails. This matters more here than in an ordinary C
# library, because the akerror control-flow macros expand at their call
# sites -- gcov attributes ATTEMPT/CATCH/PASS to the caller, so coverage
# cannot see them and mutation testing is the only thing that checks them.
#
# Bounded to two small, fast, deterministic files. src/value.c is the file
# most worth mutating and is deliberately *not* here: 368 mutants at ~11s
# each is about 70 minutes, because almost everything links against it.
# Run the default target locally for the whole tree:
# cmake --build build --target mutation
#
# The threshold is a ratchet, not a quality bar. The measured score for
# these two files is 77.8% (84 killed, 24 survived, 108 total); 65 leaves
# headroom for runner variance while still failing on a real regression --
# a test deleted, or new untested code added.
#
# It was 73.1% before writing this job. The run's own findings closed the
# gap: nothing exercised a maximum-length symbol-table key, so every
# `MAX_KEY - 1` off-by-one in a strncpy survived, and nothing asserted a
# freshly initialised table was actually zeroed. Adding those two
# assertions to tests/symtab.c killed five mutants. The same run found that
# errno was never asserted clear before a strtoll, which is what stops a
# stale ERANGE from failing a valid conversion -- that is now in
# tests/convert.c.
#
# The 24 remaining survivors are listed in the published report. Most are
# ICR mutants on loop and accumulator initialisers that a stronger
# placement assertion would catch; three in convert.c are genuinely
# equivalent and cannot be killed. Recorded in TODO.md.
- name: mutation testing
run: |
python3 scripts/mutation_test.py \
--target src/convert.c \
--target src/symtab.c \
--junit mutation-junit.xml \
--threshold 65
# Publish even when the threshold gate fails, so survivors are visible --
# each one is a missing test. Display-only (fail_on_failure: false); the
# --threshold above is the gate. annotate_only avoids the Checks API 404
# on Gitea (mikepenz/action-junit-report#23).
- name: publish mutation results
if: always()
uses: mikepenz/action-junit-report@v4
with:
report_paths: 'mutation-junit.xml'
annotate_only: true
detailed_summary: true
include_passed: true
fail_on_failure: 'false'
- run: echo "🍏 This job's status is ${{ job.status }}."

136
.gitea/workflows/release.yaml Normal file
View File

@@ -0,0 +1,136 @@
name: akbasic Release Build
run-name: ${{ gitea.actor }} akbasic release checks
# Manual only. Nothing here runs on push: the full mutation set is thousands of
# mutants and hours of runner time, which is a release-gate cost, not a
# per-commit one. Trigger it from the Actions tab.
on:
workflow_dispatch:
inputs:
mutation_threshold:
description: "Fail if the mutation score falls below this percentage"
required: false
default: "65"
mutation_targets:
description: "Space-separated files to mutate; empty means the whole src/ tree"
required: false
default: ""
jobs:
# Moved here from ci.yaml. The docs are wanted for a release, not on every
# push, and building them beside the release mutation run keeps the two
# artefacts a release needs in one place.
docs:
runs-on: ubuntu-latest
steps:
- name: Check out repository code
uses: actions/checkout@v4
# No submodules at all. Doxyfile's INPUT is include/akbasic, src and
# examples, none of which live under deps/, and doxygen does not need to
# resolve <akerror.h> to parse a declaration. Verified by running it
# against a tree with no deps/ present.
- name: dependencies
run: |
sudo apt-get update -y
sudo apt-get install -y doxygen graphviz
# This is a gate, not a convenience. The Doxyfile sets
# WARN_AS_ERROR = FAIL_ON_WARNINGS, so a doc block that documents some of a
# function's parameters but not all of them fails the build. All 114 public
# declarations under include/akbasic carry a @brief, a @param per
# parameter, a @return and their @throws; keep it that way.
#
# OUTPUT_DIRECTORY is build/docs and doxygen will not create a missing
# parent, so make it first -- there is no cmake step in this job to do it.
- name: build API documentation
run: |
mkdir -p build
doxygen Doxyfile
- name: upload API documentation
if: always()
uses: actions/upload-artifact@v4
with:
name: api-documentation
path: build/docs/html/
if-no-files-found: error
- run: echo "🍏 This job's status is ${{ job.status }}."
full_mutation:
runs-on: ubuntu-latest
# 3675 mutants across the whole src/ tree. Cost per mutant is one incremental
# rebuild plus one full ctest run, which measures at roughly 2s for a leaf
# file and 11s for src/value.c, where almost everything links against it.
# Budget most of a day and expect it to finish well inside that; the ceiling
# exists so a mutant that wedges the runner cannot hold it forever.
timeout-minutes: 720
steps:
- name: Check out repository code
uses: actions/checkout@v4
with:
# The harness copies the repo and configures a build inside the copy,
# so it needs the same submodules the main build does -- including
# deps/basicinterpret, because the golden corpus is part of what kills
# mutants.
submodules: true
- name: dependencies
run: |
sudo apt-get update -y
sudo apt-get install -y cmake gcc python3
# The whole akbasic-owned src/ tree. ci.yaml runs a two-file subset on every
# push; this is the one that actually covers the interpreter.
#
# Mutation testing is the only gate that sees the error-handling control
# flow at all: libakerror's ATTEMPT/CATCH/PASS macros expand at their call
# sites, so gcov attributes them to the caller and line coverage cannot
# measure them.
#
# The default threshold matches ci.yaml's rather than being stricter.
# Whole-tree coverage is uneven -- src/convert.c and src/symtab.c are the
# two measured files, at 77.8% between them, and the rest is unmeasured, so
# a tighter number here would be a guess. Raise it with the workflow input
# once a full run has established a real baseline.
#
# The two inputs arrive through the environment rather than being
# interpolated straight into the script. ${{ }} substitution happens before
# the shell sees the line, so a value containing shell metacharacters would
# otherwise run as code. This workflow is manual and owner-triggered, but
# the safe form costs nothing.
- name: mutation testing (full tree)
env:
MUTATION_TARGETS: ${{ gitea.event.inputs.mutation_targets }}
MUTATION_THRESHOLD: ${{ gitea.event.inputs.mutation_threshold }}
run: |
set -eu
# Word-splitting is the point here: the input is a space-separated
# list. An empty input leaves $targets empty and the harness falls
# through to its own default, which is the whole src/ tree.
targets=""
for f in ${MUTATION_TARGETS:-}; do
targets="$targets --target $f"
done
# shellcheck disable=SC2086
python3 scripts/mutation_test.py \
$targets \
--junit mutation-junit.xml \
--threshold "${MUTATION_THRESHOLD:-65}"
# Publish even when the threshold gate fails, so survivors are visible --
# each one is a missing test. Display-only (fail_on_failure: false); the
# --threshold above is the gate. annotate_only avoids the Checks API 404
# on Gitea (mikepenz/action-junit-report#23).
- name: publish mutation results
if: always()
uses: mikepenz/action-junit-report@v4
with:
report_paths: 'mutation-junit.xml'
annotate_only: true
detailed_summary: true
include_passed: true
fail_on_failure: 'false'
# Keep the raw report as well as the annotations: a release wants the
# survivor list on file, and the job summary is not durable.
- name: upload mutation report
if: always()
uses: actions/upload-artifact@v4
with:
name: mutation-report
path: mutation-junit.xml
if-no-files-found: warn
- run: echo "🍏 This job's status is ${{ job.status }}."