The perf suites were already running in CI, because they are ordinary CTest tests -- but only in the coverage job, which is a Debug build with gcov instrumentation. That is the one place their numbers cannot mean anything, and it was spending full-scale iteration counts to prove it. They now run there at AKGL_BENCH_SCALE=0.02, for path coverage alone, which takes the whole coverage run to four seconds. The `performance` job is where the timings are taken: RelWithDebInfo, full scale, `ctest -L perf`, which is the only configuration in which tests/benchutil.h enforces a budget. It keeps the tables as an artifact whether it passes or fails -- a red run says which measurement moved, and a green one is the next baseline for PERFORMANCE.md. The step uses --verbose rather than --output-on-failure, because --output-on-failure prints nothing at all for a passing benchmark, and sets pipefail, without which the status reaching the runner is tee's and a blown budget reads as a pass. The `memory_check` job runs scripts/memcheck.sh over every suite under valgrind and keeps the logs. It carries continue-on-error, and that is a deliberate, temporary lie: the six findings in TODO.md "Memory checking" are real and all of them are libakgl's, so gating on it today would only teach everyone to ignore a red build. The flag comes off with the commit that closes the last item, and item 34 -- the missing json_decref in all four asset loaders -- is four lines. Both new jobs were run locally against a clean copy of the tree with the exact commands the workflow uses. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
212 lines
8.5 KiB
YAML
212 lines
8.5 KiB
YAML
name: libakgl CI Build
|
|
run-name: ${{ gitea.actor }} libakgl test
|
|
on: [push]
|
|
|
|
jobs:
|
|
cmake_build:
|
|
runs-on: ubuntu-latest
|
|
steps:
|
|
- name: Check out repository code
|
|
uses: actions/checkout@v4
|
|
with:
|
|
submodules: recursive
|
|
- name: Install build dependencies
|
|
run: |
|
|
sudo apt-get update -y
|
|
sudo apt-get install -y \
|
|
cmake doxygen gcc gcovr pkg-config \
|
|
libasound2-dev libfreetype-dev libharfbuzz-dev \
|
|
libpng-dev libtiff-dev libwebp-dev \
|
|
libudev-dev libx11-dev libxcursor-dev libxext-dev \
|
|
libxfixes-dev libxi-dev libxrandr-dev libxrender-dev \
|
|
libxss-dev libxtst-dev
|
|
- name: Configure and build
|
|
run: |
|
|
cmake -S . -B build \
|
|
-DCMAKE_BUILD_TYPE=Debug \
|
|
-DAKGL_COVERAGE=ON
|
|
cmake --build build --parallel
|
|
- name: Build API documentation
|
|
run: doxygen Doxyfile
|
|
# The perf suites run here too -- they are ordinary CTest tests -- but this
|
|
# is a Debug build with coverage instrumentation, where a timing is a
|
|
# measurement of gcov. AKGL_BENCH_SCALE cuts their iteration counts so
|
|
# they contribute path coverage without spending ten minutes proving
|
|
# nothing; benchutil.h already refuses to enforce budgets in an
|
|
# unoptimized build. The performance job below is where the timings are
|
|
# taken and the budgets are checked.
|
|
- name: Test (JUnit)
|
|
env:
|
|
AKGL_BENCH_SCALE: '0.02'
|
|
run: |
|
|
export LD_LIBRARY_PATH="$PWD/build:$PWD/build/deps/SDL:$PWD/build/deps/SDL_image:$PWD/build/deps/SDL_mixer:$PWD/build/deps/SDL_ttf"
|
|
ctest \
|
|
--test-dir build \
|
|
-E '^character$' \
|
|
--output-on-failure \
|
|
--output-junit "$(pwd)/ctest-junit.xml"
|
|
- name: Publish test results
|
|
if: always()
|
|
uses: mikepenz/action-junit-report@v4
|
|
with:
|
|
report_paths: 'ctest-junit.xml'
|
|
annotate_only: true
|
|
detailed_summary: true
|
|
include_passed: true
|
|
fail_on_failure: 'true'
|
|
- name: Upload coverage reports
|
|
if: always()
|
|
uses: actions/upload-artifact@v4
|
|
with:
|
|
name: code-coverage
|
|
path: build/coverage/
|
|
if-no-files-found: warn
|
|
|
|
# The benchmarks, in the only build where their numbers mean anything.
|
|
# tests/benchutil.h enforces a budget per measurement only when it is compiled
|
|
# optimized and running at full scale, which is exactly this job and no other:
|
|
# the coverage job above is Debug, and the memory job below is under valgrind.
|
|
# A budget is roughly ten times the baseline recorded in PERFORMANCE.md, which
|
|
# is what makes a shared, slower runner viable -- it catches an algorithmic
|
|
# regression and ignores a busy machine. If a budget still proves flaky here,
|
|
# raise that one budget with a measurement behind it and re-record the
|
|
# baseline; do not drop the gate.
|
|
performance:
|
|
runs-on: ubuntu-latest
|
|
steps:
|
|
- name: Check out repository code
|
|
uses: actions/checkout@v4
|
|
with:
|
|
submodules: recursive
|
|
- name: Install build dependencies
|
|
run: |
|
|
sudo apt-get update -y
|
|
sudo apt-get install -y \
|
|
cmake gcc pkg-config \
|
|
libasound2-dev libfreetype-dev libharfbuzz-dev \
|
|
libpng-dev libtiff-dev libwebp-dev \
|
|
libudev-dev libx11-dev libxcursor-dev libxext-dev \
|
|
libxfixes-dev libxi-dev libxrandr-dev libxrender-dev \
|
|
libxss-dev libxtst-dev
|
|
- name: Configure and build
|
|
run: |
|
|
cmake -S . -B build \
|
|
-DCMAKE_BUILD_TYPE=RelWithDebInfo
|
|
cmake --build build --parallel
|
|
# --verbose rather than --output-on-failure: a passing benchmark's output
|
|
# is the entire point of running it, and --output-on-failure prints
|
|
# nothing at all for a green run. pipefail because the step ends in a
|
|
# pipe, whose status would otherwise be tee's -- always 0 -- and a blown
|
|
# budget would be reported as a passing step.
|
|
- name: Benchmarks (JUnit)
|
|
run: |
|
|
set -o pipefail
|
|
export LD_LIBRARY_PATH="$PWD/build:$PWD/build/deps/SDL:$PWD/build/deps/SDL_image:$PWD/build/deps/SDL_mixer:$PWD/build/deps/SDL_ttf"
|
|
ctest \
|
|
--test-dir build \
|
|
-L perf \
|
|
--verbose \
|
|
--output-junit "$(pwd)/perf-junit.xml" \
|
|
2>&1 | tee "$(pwd)/perf-baseline.txt"
|
|
- name: Publish benchmark results
|
|
if: always()
|
|
uses: mikepenz/action-junit-report@v4
|
|
with:
|
|
report_paths: 'perf-junit.xml'
|
|
annotate_only: true
|
|
detailed_summary: true
|
|
include_passed: true
|
|
fail_on_failure: 'true'
|
|
# Kept whether the job passed or failed: the tables are the point. A run
|
|
# that went red says which measurement moved, and a run that went green is
|
|
# the next baseline if somebody re-records PERFORMANCE.md.
|
|
- name: Upload benchmark tables
|
|
if: always()
|
|
uses: actions/upload-artifact@v4
|
|
with:
|
|
name: performance-baseline
|
|
path: perf-baseline.txt
|
|
if-no-files-found: warn
|
|
|
|
# Every suite under valgrind. No memory-check programs of their own: the perf
|
|
# suites scale themselves down when they detect valgrind, which turns them
|
|
# into the broadest path coverage in the tree at a cost this job can afford.
|
|
#
|
|
# continue-on-error until TODO.md "Memory checking" items 34-39 are closed.
|
|
# The findings are real and they are all in libakgl, so this job is red today
|
|
# by telling the truth; gating on it now would only teach everyone to ignore a
|
|
# red build. Remove the flag with the commit that closes the last item -- and
|
|
# the JSON one (item 34) is four lines, so that should not be a long wait.
|
|
memory_check:
|
|
runs-on: ubuntu-latest
|
|
continue-on-error: true
|
|
steps:
|
|
- name: Check out repository code
|
|
uses: actions/checkout@v4
|
|
with:
|
|
submodules: recursive
|
|
- name: Install memory-check dependencies
|
|
run: |
|
|
sudo apt-get update -y
|
|
sudo apt-get install -y \
|
|
cmake gcc pkg-config valgrind \
|
|
libasound2-dev libfreetype-dev libharfbuzz-dev \
|
|
libpng-dev libtiff-dev libwebp-dev \
|
|
libudev-dev libx11-dev libxcursor-dev libxext-dev \
|
|
libxfixes-dev libxi-dev libxrandr-dev libxrender-dev \
|
|
libxss-dev libxtst-dev
|
|
# RelWithDebInfo rather than Debug: valgrind needs the symbols, and -O0
|
|
# would leave every inlined frame in the stacks it prints.
|
|
- name: Configure and build
|
|
run: |
|
|
cmake -S . -B build \
|
|
-DCMAKE_BUILD_TYPE=RelWithDebInfo
|
|
cmake --build build --parallel
|
|
# The character suite is excluded here for the same reason as in the
|
|
# coverage job: it fails deliberately, and a failing suite would be
|
|
# reported as "valgrind found nothing but ctest exited non-zero".
|
|
- name: Memory check
|
|
run: scripts/memcheck.sh -E '^character$'
|
|
- name: Upload valgrind logs
|
|
if: always()
|
|
uses: actions/upload-artifact@v4
|
|
with:
|
|
name: valgrind-logs
|
|
path: build/Testing/Temporary/MemoryChecker.*.log
|
|
if-no-files-found: warn
|
|
|
|
mutation_test:
|
|
runs-on: ubuntu-latest
|
|
steps:
|
|
- name: Check out repository code
|
|
uses: actions/checkout@v4
|
|
with:
|
|
submodules: recursive
|
|
- name: Install mutation-test dependencies
|
|
run: |
|
|
sudo apt-get update -y
|
|
sudo apt-get install -y \
|
|
cmake gcc pkg-config python3 \
|
|
libasound2-dev libfreetype-dev libharfbuzz-dev \
|
|
libpng-dev libtiff-dev libwebp-dev \
|
|
libudev-dev libx11-dev libxcursor-dev libxext-dev \
|
|
libxfixes-dev libxi-dev libxrandr-dev libxrender-dev \
|
|
libxss-dev libxtst-dev
|
|
# Keep CI bounded to a focused source file. Run the default target locally
|
|
# for the complete libakgl-owned src/ tree.
|
|
- name: Mutation testing
|
|
run: |
|
|
python3 scripts/mutation_test.py \
|
|
--target src/staticstring.c \
|
|
--junit mutation-junit.xml \
|
|
--threshold 50
|
|
- name: Publish mutation results
|
|
if: always()
|
|
uses: mikepenz/action-junit-report@v4
|
|
with:
|
|
report_paths: 'mutation-junit.xml'
|
|
annotate_only: true
|
|
detailed_summary: true
|
|
include_passed: true
|
|
fail_on_failure: 'false'
|