Run the benchmarks and the memory check in CI
The perf suites were already running in CI, because they are ordinary CTest tests -- but only in the coverage job, which is a Debug build with gcov instrumentation. That is the one place their numbers cannot mean anything, and it was spending full-scale iteration counts to prove it. They now run there at AKGL_BENCH_SCALE=0.02, for path coverage alone, which takes the whole coverage run to four seconds. The `performance` job is where the timings are taken: RelWithDebInfo, full scale, `ctest -L perf`, which is the only configuration in which tests/benchutil.h enforces a budget. It keeps the tables as an artifact whether it passes or fails -- a red run says which measurement moved, and a green one is the next baseline for PERFORMANCE.md. The step uses --verbose rather than --output-on-failure, because --output-on-failure prints nothing at all for a passing benchmark, and sets pipefail, without which the status reaching the runner is tee's and a blown budget reads as a pass. The `memory_check` job runs scripts/memcheck.sh over every suite under valgrind and keeps the logs. It carries continue-on-error, and that is a deliberate, temporary lie: the six findings in TODO.md "Memory checking" are real and all of them are libakgl's, so gating on it today would only teach everyone to ignore a red build. The flag comes off with the commit that closes the last item, and item 34 -- the missing json_decref in all four asset loaders -- is four lines. Both new jobs were run locally against a clean copy of the tree with the exact commands the workflow uses. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -28,7 +28,16 @@ jobs:
|
||||
cmake --build build --parallel
|
||||
- name: Build API documentation
|
||||
run: doxygen Doxyfile
|
||||
# The perf suites run here too -- they are ordinary CTest tests -- but this
|
||||
# is a Debug build with coverage instrumentation, where a timing is a
|
||||
# measurement of gcov. AKGL_BENCH_SCALE cuts their iteration counts so
|
||||
# they contribute path coverage without spending ten minutes proving
|
||||
# nothing; benchutil.h already refuses to enforce budgets in an
|
||||
# unoptimized build. The performance job below is where the timings are
|
||||
# taken and the budgets are checked.
|
||||
- name: Test (JUnit)
|
||||
env:
|
||||
AKGL_BENCH_SCALE: '0.02'
|
||||
run: |
|
||||
export LD_LIBRARY_PATH="$PWD/build:$PWD/build/deps/SDL:$PWD/build/deps/SDL_image:$PWD/build/deps/SDL_mixer:$PWD/build/deps/SDL_ttf"
|
||||
ctest \
|
||||
@@ -53,6 +62,119 @@ jobs:
|
||||
path: build/coverage/
|
||||
if-no-files-found: warn
|
||||
|
||||
# The benchmarks, in the only build where their numbers mean anything.
|
||||
# tests/benchutil.h enforces a budget per measurement only when it is compiled
|
||||
# optimized and running at full scale, which is exactly this job and no other:
|
||||
# the coverage job above is Debug, and the memory job below is under valgrind.
|
||||
# A budget is roughly ten times the baseline recorded in PERFORMANCE.md, which
|
||||
# is what makes a shared, slower runner viable -- it catches an algorithmic
|
||||
# regression and ignores a busy machine. If a budget still proves flaky here,
|
||||
# raise that one budget with a measurement behind it and re-record the
|
||||
# baseline; do not drop the gate.
|
||||
performance:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Check out repository code
|
||||
uses: actions/checkout@v4
|
||||
with:
|
||||
submodules: recursive
|
||||
- name: Install build dependencies
|
||||
run: |
|
||||
sudo apt-get update -y
|
||||
sudo apt-get install -y \
|
||||
cmake gcc pkg-config \
|
||||
libasound2-dev libfreetype-dev libharfbuzz-dev \
|
||||
libpng-dev libtiff-dev libwebp-dev \
|
||||
libudev-dev libx11-dev libxcursor-dev libxext-dev \
|
||||
libxfixes-dev libxi-dev libxrandr-dev libxrender-dev \
|
||||
libxss-dev libxtst-dev
|
||||
- name: Configure and build
|
||||
run: |
|
||||
cmake -S . -B build \
|
||||
-DCMAKE_BUILD_TYPE=RelWithDebInfo
|
||||
cmake --build build --parallel
|
||||
# --verbose rather than --output-on-failure: a passing benchmark's output
|
||||
# is the entire point of running it, and --output-on-failure prints
|
||||
# nothing at all for a green run. pipefail because the step ends in a
|
||||
# pipe, whose status would otherwise be tee's -- always 0 -- and a blown
|
||||
# budget would be reported as a passing step.
|
||||
- name: Benchmarks (JUnit)
|
||||
run: |
|
||||
set -o pipefail
|
||||
export LD_LIBRARY_PATH="$PWD/build:$PWD/build/deps/SDL:$PWD/build/deps/SDL_image:$PWD/build/deps/SDL_mixer:$PWD/build/deps/SDL_ttf"
|
||||
ctest \
|
||||
--test-dir build \
|
||||
-L perf \
|
||||
--verbose \
|
||||
--output-junit "$(pwd)/perf-junit.xml" \
|
||||
2>&1 | tee "$(pwd)/perf-baseline.txt"
|
||||
- name: Publish benchmark results
|
||||
if: always()
|
||||
uses: mikepenz/action-junit-report@v4
|
||||
with:
|
||||
report_paths: 'perf-junit.xml'
|
||||
annotate_only: true
|
||||
detailed_summary: true
|
||||
include_passed: true
|
||||
fail_on_failure: 'true'
|
||||
# Kept whether the job passed or failed: the tables are the point. A run
|
||||
# that went red says which measurement moved, and a run that went green is
|
||||
# the next baseline if somebody re-records PERFORMANCE.md.
|
||||
- name: Upload benchmark tables
|
||||
if: always()
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: performance-baseline
|
||||
path: perf-baseline.txt
|
||||
if-no-files-found: warn
|
||||
|
||||
# Every suite under valgrind. No memory-check programs of their own: the perf
|
||||
# suites scale themselves down when they detect valgrind, which turns them
|
||||
# into the broadest path coverage in the tree at a cost this job can afford.
|
||||
#
|
||||
# continue-on-error until TODO.md "Memory checking" items 34-39 are closed.
|
||||
# The findings are real and they are all in libakgl, so this job is red today
|
||||
# by telling the truth; gating on it now would only teach everyone to ignore a
|
||||
# red build. Remove the flag with the commit that closes the last item -- and
|
||||
# the JSON one (item 34) is four lines, so that should not be a long wait.
|
||||
memory_check:
|
||||
runs-on: ubuntu-latest
|
||||
continue-on-error: true
|
||||
steps:
|
||||
- name: Check out repository code
|
||||
uses: actions/checkout@v4
|
||||
with:
|
||||
submodules: recursive
|
||||
- name: Install memory-check dependencies
|
||||
run: |
|
||||
sudo apt-get update -y
|
||||
sudo apt-get install -y \
|
||||
cmake gcc pkg-config valgrind \
|
||||
libasound2-dev libfreetype-dev libharfbuzz-dev \
|
||||
libpng-dev libtiff-dev libwebp-dev \
|
||||
libudev-dev libx11-dev libxcursor-dev libxext-dev \
|
||||
libxfixes-dev libxi-dev libxrandr-dev libxrender-dev \
|
||||
libxss-dev libxtst-dev
|
||||
# RelWithDebInfo rather than Debug: valgrind needs the symbols, and -O0
|
||||
# would leave every inlined frame in the stacks it prints.
|
||||
- name: Configure and build
|
||||
run: |
|
||||
cmake -S . -B build \
|
||||
-DCMAKE_BUILD_TYPE=RelWithDebInfo
|
||||
cmake --build build --parallel
|
||||
# The character suite is excluded here for the same reason as in the
|
||||
# coverage job: it fails deliberately, and a failing suite would be
|
||||
# reported as "valgrind found nothing but ctest exited non-zero".
|
||||
- name: Memory check
|
||||
run: scripts/memcheck.sh -E '^character$'
|
||||
- name: Upload valgrind logs
|
||||
if: always()
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: valgrind-logs
|
||||
path: build/Testing/Temporary/MemoryChecker.*.log
|
||||
if-no-files-found: warn
|
||||
|
||||
mutation_test:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
|
||||
15
AGENTS.md
15
AGENTS.md
@@ -79,6 +79,21 @@ Run mutation testing with `cmake --build build --target mutation`. For a quick s
|
||||
|
||||
Generate HTML and Cobertura coverage reports with `cmake -S . -B build-coverage -DAKGL_COVERAGE=ON -DCMAKE_BUILD_TYPE=Debug`, then build and run CTest. Reports are written to `build-coverage/coverage/`; this mode requires `gcovr` and GCC or Clang.
|
||||
|
||||
## Continuous integration
|
||||
|
||||
`.gitea/workflows/ci.yaml` runs four jobs on every push, and each one exists
|
||||
because the others cannot do its work:
|
||||
|
||||
| Job | Build | What it is for |
|
||||
|---|---|---|
|
||||
| `cmake_build` | Debug + `AKGL_COVERAGE=ON` | The unit suites and the coverage report. The perf suites run here too, at `AKGL_BENCH_SCALE=0.02`, for path coverage only — a timing taken under gcov is a measurement of gcov. |
|
||||
| `performance` | RelWithDebInfo | `ctest -L perf`. The only job where budgets are enforced, because `tests/benchutil.h` enforces them only when compiled optimized at full scale. Keeps the tables as an artifact. |
|
||||
| `memory_check` | RelWithDebInfo | `scripts/memcheck.sh`, every suite under valgrind. `continue-on-error` until TODO.md "Memory checking" items 34-39 are closed; keeps the valgrind logs as an artifact. |
|
||||
| `mutation_test` | Debug | One focused source file, so CI stays bounded. |
|
||||
|
||||
The `character` suite is excluded wherever a whole run is selected: it fails
|
||||
deliberately, and pinning current-but-wrong behavior is not what it is for.
|
||||
|
||||
## Coding Style
|
||||
|
||||
The canonical style is Emacs `cc-mode` **`stroustrup`**, with tabs enabled. This
|
||||
|
||||
Reference in New Issue
Block a user