Schedule mutation testing outside push CI
Some checks failed
akbasic CI Build / mutation_test (push) Waiting to run
akbasic CI Build / cmake_build (push) Successful in 3m41s
akbasic CI Build / akgl_build (push) Has been cancelled
akbasic CI Build / sanitizers (push) Has been cancelled
akbasic CI Build / coverage (push) Has been cancelled
Some checks failed
akbasic CI Build / mutation_test (push) Waiting to run
akbasic CI Build / cmake_build (push) Successful in 3m41s
akbasic CI Build / akgl_build (push) Has been cancelled
akbasic CI Build / sanitizers (push) Has been cancelled
akbasic CI Build / coverage (push) Has been cancelled
Move the full mutation suite into a dedicated daily 02:00 EST workflow so merge and release checks do not consume CI agents for hours.
This commit is contained in:
48
.gitea/workflows/mutation.yaml
Normal file
48
.gitea/workflows/mutation.yaml
Normal file
@@ -0,0 +1,48 @@
|
||||
name: akbasic Mutation Testing
|
||||
run-name: ${{ gitea.actor }} akbasic mutation test
|
||||
|
||||
# Mutation testing is intentionally isolated from push and merge checks. The
|
||||
# complete src/ tree takes hours and must not consume the shared CI capacity
|
||||
# during normal development.
|
||||
on:
|
||||
# Gitea schedules use UTC. 07:00 UTC is 02:00 EST (the requested fixed
|
||||
# Eastern Standard Time slot).
|
||||
schedule:
|
||||
- cron: '0 7 * * *'
|
||||
workflow_dispatch:
|
||||
|
||||
jobs:
|
||||
full_mutation:
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 720
|
||||
steps:
|
||||
- name: Check out repository code
|
||||
uses: actions/checkout@v4
|
||||
with:
|
||||
submodules: true
|
||||
- name: dependencies
|
||||
run: |
|
||||
sudo apt-get update -y
|
||||
sudo apt-get install -y cmake gcc python3 moreutils
|
||||
- name: mutation testing (full tree)
|
||||
run: |
|
||||
python3 scripts/mutation_test.py \
|
||||
--junit mutation-junit.xml \
|
||||
--threshold 65
|
||||
- name: publish mutation results
|
||||
if: always()
|
||||
uses: mikepenz/action-junit-report@v4
|
||||
with:
|
||||
report_paths: 'mutation-junit.xml'
|
||||
annotate_only: true
|
||||
detailed_summary: true
|
||||
include_passed: true
|
||||
fail_on_failure: 'false'
|
||||
- name: upload mutation report
|
||||
if: always()
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: mutation-report
|
||||
path: mutation-junit.xml
|
||||
if-no-files-found: warn
|
||||
- run: echo "🍏 This job's status is ${{ job.status }}."
|
||||
@@ -1,24 +1,13 @@
|
||||
name: akbasic Release Build
|
||||
run-name: ${{ gitea.actor }} akbasic release checks
|
||||
# Manual only. Nothing here runs on push: the full mutation set is thousands of
|
||||
# mutants and hours of runner time, which is a release-gate cost, not a
|
||||
# per-commit one. Trigger it from the Actions tab.
|
||||
# Manual only. Nothing here runs on push: documentation is a release gate, not
|
||||
# a per-commit check. The expensive mutation suite has its own daily schedule.
|
||||
on:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
mutation_threshold:
|
||||
description: "Fail if the mutation score falls below this percentage"
|
||||
required: false
|
||||
default: "65"
|
||||
mutation_targets:
|
||||
description: "Space-separated files to mutate; empty means the whole src/ tree"
|
||||
required: false
|
||||
default: ""
|
||||
|
||||
jobs:
|
||||
# Moved here from ci.yaml. The docs are wanted for a release, not on every
|
||||
# push, and building them beside the release mutation run keeps the two
|
||||
# artefacts a release needs in one place.
|
||||
# push.
|
||||
docs:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
@@ -52,89 +41,3 @@ jobs:
|
||||
path: build/docs/html/
|
||||
if-no-files-found: error
|
||||
- run: echo "🍏 This job's status is ${{ job.status }}."
|
||||
|
||||
full_mutation:
|
||||
runs-on: ubuntu-latest
|
||||
# 3675 mutants across the whole src/ tree. Cost per mutant is one incremental
|
||||
# rebuild plus one full ctest run, which measures at roughly 2s for a leaf
|
||||
# file and 11s for src/value.c, where almost everything links against it.
|
||||
# Budget most of a day and expect it to finish well inside that; the ceiling
|
||||
# exists so a mutant that wedges the runner cannot hold it forever.
|
||||
timeout-minutes: 720
|
||||
steps:
|
||||
- name: Check out repository code
|
||||
uses: actions/checkout@v4
|
||||
with:
|
||||
# The harness copies the repo and configures a build inside the copy,
|
||||
# so it needs the same submodules the main build does. The golden
|
||||
# corpus is part of what kills mutants and it is checked in now, so
|
||||
# that is libakerror and libakstdlib and nothing else.
|
||||
submodules: true
|
||||
# moreutils for errno(1), which deps/libakerror's scripts/generrno.sh needs
|
||||
# to generate its errno table. Without it the generated table is empty and
|
||||
# every AKERR_* code collapses onto a low integer -- see the long note on
|
||||
# ci.yaml's cmake_build job for what that breaks and how it presents.
|
||||
- name: dependencies
|
||||
run: |
|
||||
sudo apt-get update -y
|
||||
sudo apt-get install -y cmake gcc python3 moreutils
|
||||
# The whole akbasic-owned src/ tree. ci.yaml runs a two-file subset on every
|
||||
# push; this is the one that actually covers the interpreter.
|
||||
#
|
||||
# Mutation testing is the only gate that sees the error-handling control
|
||||
# flow at all: libakerror's ATTEMPT/CATCH/PASS macros expand at their call
|
||||
# sites, so gcov attributes them to the caller and line coverage cannot
|
||||
# measure them.
|
||||
#
|
||||
# The default threshold matches ci.yaml's rather than being stricter.
|
||||
# Whole-tree coverage is uneven -- src/symtab.c is the only file measured
|
||||
# on the push path, at 74.1%, and the rest is unmeasured, so a tighter
|
||||
# number here would be a guess. Raise it with the workflow input once a
|
||||
# full run has established a real baseline.
|
||||
#
|
||||
# The two inputs arrive through the environment rather than being
|
||||
# interpolated straight into the script. ${{ }} substitution happens before
|
||||
# the shell sees the line, so a value containing shell metacharacters would
|
||||
# otherwise run as code. This workflow is manual and owner-triggered, but
|
||||
# the safe form costs nothing.
|
||||
- name: mutation testing (full tree)
|
||||
env:
|
||||
MUTATION_TARGETS: ${{ gitea.event.inputs.mutation_targets }}
|
||||
MUTATION_THRESHOLD: ${{ gitea.event.inputs.mutation_threshold }}
|
||||
run: |
|
||||
set -eu
|
||||
# Word-splitting is the point here: the input is a space-separated
|
||||
# list. An empty input leaves $targets empty and the harness falls
|
||||
# through to its own default, which is the whole src/ tree.
|
||||
targets=""
|
||||
for f in ${MUTATION_TARGETS:-}; do
|
||||
targets="$targets --target $f"
|
||||
done
|
||||
# shellcheck disable=SC2086
|
||||
python3 scripts/mutation_test.py \
|
||||
$targets \
|
||||
--junit mutation-junit.xml \
|
||||
--threshold "${MUTATION_THRESHOLD:-65}"
|
||||
# Publish even when the threshold gate fails, so survivors are visible --
|
||||
# each one is a missing test. Display-only (fail_on_failure: false); the
|
||||
# --threshold above is the gate. annotate_only avoids the Checks API 404
|
||||
# on Gitea (mikepenz/action-junit-report#23).
|
||||
- name: publish mutation results
|
||||
if: always()
|
||||
uses: mikepenz/action-junit-report@v4
|
||||
with:
|
||||
report_paths: 'mutation-junit.xml'
|
||||
annotate_only: true
|
||||
detailed_summary: true
|
||||
include_passed: true
|
||||
fail_on_failure: 'false'
|
||||
# Keep the raw report as well as the annotations: a release wants the
|
||||
# survivor list on file, and the job summary is not durable.
|
||||
- name: upload mutation report
|
||||
if: always()
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: mutation-report
|
||||
path: mutation-junit.xml
|
||||
if-no-files-found: warn
|
||||
- run: echo "🍏 This job's status is ${{ job.status }}."
|
||||
|
||||
Reference in New Issue
Block a user