Files
libakgl/.gitea/workflows/ci.yaml
Andrew Kesterson 80fdf2e098 Run the benchmarks and the memory check in CI
The perf suites were already running in CI, because they are ordinary CTest
tests -- but only in the coverage job, which is a Debug build with gcov
instrumentation. That is the one place their numbers cannot mean anything,
and it was spending full-scale iteration counts to prove it. They now run
there at AKGL_BENCH_SCALE=0.02, for path coverage alone, which takes the
whole coverage run to four seconds.

The `performance` job is where the timings are taken: RelWithDebInfo, full
scale, `ctest -L perf`, which is the only configuration in which
tests/benchutil.h enforces a budget. It keeps the tables as an artifact
whether it passes or fails -- a red run says which measurement moved, and a
green one is the next baseline for PERFORMANCE.md. The step uses --verbose
rather than --output-on-failure, because --output-on-failure prints nothing
at all for a passing benchmark, and sets pipefail, without which the status
reaching the runner is tee's and a blown budget reads as a pass.

The `memory_check` job runs scripts/memcheck.sh over every suite under
valgrind and keeps the logs. It carries continue-on-error, and that is a
deliberate, temporary lie: the six findings in TODO.md "Memory checking" are
real and all of them are libakgl's, so gating on it today would only teach
everyone to ignore a red build. The flag comes off with the commit that
closes the last item, and item 34 -- the missing json_decref in all four
asset loaders -- is four lines.

Both new jobs were run locally against a clean copy of the tree with the
exact commands the workflow uses.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-31 15:33:58 -04:00

212 lines
8.5 KiB
YAML

name: libakgl CI Build
run-name: ${{ gitea.actor }} libakgl test
on: [push]
jobs:
cmake_build:
runs-on: ubuntu-latest
steps:
- name: Check out repository code
uses: actions/checkout@v4
with:
submodules: recursive
- name: Install build dependencies
run: |
sudo apt-get update -y
sudo apt-get install -y \
cmake doxygen gcc gcovr pkg-config \
libasound2-dev libfreetype-dev libharfbuzz-dev \
libpng-dev libtiff-dev libwebp-dev \
libudev-dev libx11-dev libxcursor-dev libxext-dev \
libxfixes-dev libxi-dev libxrandr-dev libxrender-dev \
libxss-dev libxtst-dev
- name: Configure and build
run: |
cmake -S . -B build \
-DCMAKE_BUILD_TYPE=Debug \
-DAKGL_COVERAGE=ON
cmake --build build --parallel
- name: Build API documentation
run: doxygen Doxyfile
# The perf suites run here too -- they are ordinary CTest tests -- but this
# is a Debug build with coverage instrumentation, where a timing is a
# measurement of gcov. AKGL_BENCH_SCALE cuts their iteration counts so
# they contribute path coverage without spending ten minutes proving
# nothing; benchutil.h already refuses to enforce budgets in an
# unoptimized build. The performance job below is where the timings are
# taken and the budgets are checked.
- name: Test (JUnit)
env:
AKGL_BENCH_SCALE: '0.02'
run: |
export LD_LIBRARY_PATH="$PWD/build:$PWD/build/deps/SDL:$PWD/build/deps/SDL_image:$PWD/build/deps/SDL_mixer:$PWD/build/deps/SDL_ttf"
ctest \
--test-dir build \
-E '^character$' \
--output-on-failure \
--output-junit "$(pwd)/ctest-junit.xml"
- name: Publish test results
if: always()
uses: mikepenz/action-junit-report@v4
with:
report_paths: 'ctest-junit.xml'
annotate_only: true
detailed_summary: true
include_passed: true
fail_on_failure: 'true'
- name: Upload coverage reports
if: always()
uses: actions/upload-artifact@v4
with:
name: code-coverage
path: build/coverage/
if-no-files-found: warn
# The benchmarks, in the only build where their numbers mean anything.
# tests/benchutil.h enforces a budget per measurement only when it is compiled
# optimized and running at full scale, which is exactly this job and no other:
# the coverage job above is Debug, and the memory job below is under valgrind.
# A budget is roughly ten times the baseline recorded in PERFORMANCE.md, which
# is what makes a shared, slower runner viable -- it catches an algorithmic
# regression and ignores a busy machine. If a budget still proves flaky here,
# raise that one budget with a measurement behind it and re-record the
# baseline; do not drop the gate.
performance:
runs-on: ubuntu-latest
steps:
- name: Check out repository code
uses: actions/checkout@v4
with:
submodules: recursive
- name: Install build dependencies
run: |
sudo apt-get update -y
sudo apt-get install -y \
cmake gcc pkg-config \
libasound2-dev libfreetype-dev libharfbuzz-dev \
libpng-dev libtiff-dev libwebp-dev \
libudev-dev libx11-dev libxcursor-dev libxext-dev \
libxfixes-dev libxi-dev libxrandr-dev libxrender-dev \
libxss-dev libxtst-dev
- name: Configure and build
run: |
cmake -S . -B build \
-DCMAKE_BUILD_TYPE=RelWithDebInfo
cmake --build build --parallel
# --verbose rather than --output-on-failure: a passing benchmark's output
# is the entire point of running it, and --output-on-failure prints
# nothing at all for a green run. pipefail because the step ends in a
# pipe, whose status would otherwise be tee's -- always 0 -- and a blown
# budget would be reported as a passing step.
- name: Benchmarks (JUnit)
run: |
set -o pipefail
export LD_LIBRARY_PATH="$PWD/build:$PWD/build/deps/SDL:$PWD/build/deps/SDL_image:$PWD/build/deps/SDL_mixer:$PWD/build/deps/SDL_ttf"
ctest \
--test-dir build \
-L perf \
--verbose \
--output-junit "$(pwd)/perf-junit.xml" \
2>&1 | tee "$(pwd)/perf-baseline.txt"
- name: Publish benchmark results
if: always()
uses: mikepenz/action-junit-report@v4
with:
report_paths: 'perf-junit.xml'
annotate_only: true
detailed_summary: true
include_passed: true
fail_on_failure: 'true'
# Kept whether the job passed or failed: the tables are the point. A run
# that went red says which measurement moved, and a run that went green is
# the next baseline if somebody re-records PERFORMANCE.md.
- name: Upload benchmark tables
if: always()
uses: actions/upload-artifact@v4
with:
name: performance-baseline
path: perf-baseline.txt
if-no-files-found: warn
# Every suite under valgrind. No memory-check programs of their own: the perf
# suites scale themselves down when they detect valgrind, which turns them
# into the broadest path coverage in the tree at a cost this job can afford.
#
# continue-on-error until TODO.md "Memory checking" items 34-39 are closed.
# The findings are real and they are all in libakgl, so this job is red today
# by telling the truth; gating on it now would only teach everyone to ignore a
# red build. Remove the flag with the commit that closes the last item -- and
# the JSON one (item 34) is four lines, so that should not be a long wait.
memory_check:
runs-on: ubuntu-latest
continue-on-error: true
steps:
- name: Check out repository code
uses: actions/checkout@v4
with:
submodules: recursive
- name: Install memory-check dependencies
run: |
sudo apt-get update -y
sudo apt-get install -y \
cmake gcc pkg-config valgrind \
libasound2-dev libfreetype-dev libharfbuzz-dev \
libpng-dev libtiff-dev libwebp-dev \
libudev-dev libx11-dev libxcursor-dev libxext-dev \
libxfixes-dev libxi-dev libxrandr-dev libxrender-dev \
libxss-dev libxtst-dev
# RelWithDebInfo rather than Debug: valgrind needs the symbols, and -O0
# would leave every inlined frame in the stacks it prints.
- name: Configure and build
run: |
cmake -S . -B build \
-DCMAKE_BUILD_TYPE=RelWithDebInfo
cmake --build build --parallel
# The character suite is excluded here for the same reason as in the
# coverage job: it fails deliberately, and a failing suite would be
# reported as "valgrind found nothing but ctest exited non-zero".
- name: Memory check
run: scripts/memcheck.sh -E '^character$'
- name: Upload valgrind logs
if: always()
uses: actions/upload-artifact@v4
with:
name: valgrind-logs
path: build/Testing/Temporary/MemoryChecker.*.log
if-no-files-found: warn
mutation_test:
runs-on: ubuntu-latest
steps:
- name: Check out repository code
uses: actions/checkout@v4
with:
submodules: recursive
- name: Install mutation-test dependencies
run: |
sudo apt-get update -y
sudo apt-get install -y \
cmake gcc pkg-config python3 \
libasound2-dev libfreetype-dev libharfbuzz-dev \
libpng-dev libtiff-dev libwebp-dev \
libudev-dev libx11-dev libxcursor-dev libxext-dev \
libxfixes-dev libxi-dev libxrandr-dev libxrender-dev \
libxss-dev libxtst-dev
# Keep CI bounded to a focused source file. Run the default target locally
# for the complete libakgl-owned src/ tree.
- name: Mutation testing
run: |
python3 scripts/mutation_test.py \
--target src/staticstring.c \
--junit mutation-junit.xml \
--threshold 50
- name: Publish mutation results
if: always()
uses: mikepenz/action-junit-report@v4
with:
report_paths: 'mutation-junit.xml'
annotate_only: true
detailed_summary: true
include_passed: true
fail_on_failure: 'false'