Andrew Kesterson
68009ea0e3
A separate mutation_test job runs scripts/mutation_test.py over src/stdlib.c with a --threshold gate and publishes a JUnit report, mirroring libakerror's CI. Checkout needs submodules: recursive -- the harness copies the repo and builds the copy top-level, so deps/libakerror has to be there. Measured score on the section 1.0 suite is 46.8% (81/173 killed). The gate is set at 40: a regression ratchet, not a quality bar. Survivors cluster in the wrappers that have no tests yet (printf family, ato* family, realpath, the memory and stream wrappers); the list and tree code that section 1.0 does cover leaves only 5 survivors between them. Also caps every test with TIMEOUT 30. Measuring the score turned this up: a mutant deleting `fast = fast->next->next` makes aksl_list_iterate spin forever, so ctest never returned, the harness killed ctest at its own 120s timeout, and the orphaned test binary kept burning a core while holding the pipe open -- the run wedged and needed killing by hand. With a per-test timeout ctest reaps its own child, those mutants are killed in 30s, and nothing is left behind. It guards the real suite against the same shape of bug too. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
78 lines
3.2 KiB
YAML
78 lines
3.2 KiB
YAML
name: libakstdlib CI Build
|
|
run-name: ${{ gitea.actor }} libakstdlib test
|
|
on: [push]
|
|
|
|
jobs:
|
|
cmake_build:
|
|
runs-on: ubuntu-latest
|
|
steps:
|
|
- run: echo "Triggered by ${{ gitea.event_name }} from ${{ gitea.repository }}@${{ gitea.ref }}. Building on ${{ runner.os }}."
|
|
- name: Check out repository code
|
|
uses: actions/checkout@v4
|
|
with:
|
|
# A top-level build uses deps/libakerror via add_subdirectory, so the
|
|
# submodule has to be present or the configure step fails outright.
|
|
submodules: recursive
|
|
- name: dependencies
|
|
run: |
|
|
sudo apt-get update -y
|
|
sudo apt-get install -y cmake gcc moreutils
|
|
# Depends on libakerror@main
|
|
git clone https://source.starfort.tech/andrew/libakerror.git
|
|
cd libakerror
|
|
cmake -S . -B build
|
|
cmake --build build
|
|
cmake --install build
|
|
- name: build and test
|
|
run: |
|
|
cmake -S . -B build
|
|
cmake --build build
|
|
sudo cmake --install build
|
|
ctest --test-dir build --output-on-failure
|
|
- run: echo "🍏 This job's status is ${{ job.status }}."
|
|
|
|
mutation_test:
|
|
runs-on: ubuntu-latest
|
|
steps:
|
|
- name: Check out repository code
|
|
uses: actions/checkout@v4
|
|
with:
|
|
# The harness copies the repo and builds the copy top-level, so it
|
|
# needs deps/libakerror present just like the main job does.
|
|
submodules: recursive
|
|
- name: dependencies
|
|
run: |
|
|
sudo apt-get update -y
|
|
sudo apt-get install -y cmake gcc moreutils python3
|
|
# Verify the tests actually catch bugs: break the library many ways and
|
|
# confirm the suite fails. Gated on src/stdlib.c (fast, deterministic);
|
|
# run the full default target locally for the macro header as well.
|
|
#
|
|
# The threshold is a ratchet, not a quality bar. The score on the section
|
|
# 1.0 suite is 46.8% (81/173 killed): the list and tree functions are
|
|
# covered, and everything else in src/stdlib.c -- the printf family, the
|
|
# ato* family, realpath, the memory and stream wrappers -- has no tests
|
|
# yet, so its mutants survive. 40 leaves headroom for the runner while
|
|
# still failing on a real regression (tests deleted, or new untested code
|
|
# added). Raise it as TODO.md sections 1.1-1.9 land.
|
|
- name: mutation testing
|
|
run: |
|
|
python3 scripts/mutation_test.py \
|
|
--target src/stdlib.c \
|
|
--junit mutation-junit.xml \
|
|
--threshold 40
|
|
# Publish even when the threshold gate fails, so survivors are visible --
|
|
# each one is a missing test. Display-only (fail_on_failure: false); the
|
|
# --threshold above is the gate. annotate_only avoids the Checks API 404
|
|
# on Gitea (mikepenz/action-junit-report#23).
|
|
- name: publish mutation results
|
|
if: always()
|
|
uses: mikepenz/action-junit-report@v4
|
|
with:
|
|
report_paths: 'mutation-junit.xml'
|
|
annotate_only: true
|
|
detailed_summary: true
|
|
include_passed: true
|
|
fail_on_failure: 'false'
|
|
- run: echo "🍏 This job's status is ${{ job.status }}."
|