diff --git a/.gitea/workflows/ci.yaml b/.gitea/workflows/ci.yaml index 96cc00f..c382cea 100644 --- a/.gitea/workflows/ci.yaml +++ b/.gitea/workflows/ci.yaml @@ -28,7 +28,16 @@ jobs: cmake --build build --parallel - name: Build API documentation run: doxygen Doxyfile + # The perf suites run here too -- they are ordinary CTest tests -- but this + # is a Debug build with coverage instrumentation, where a timing is a + # measurement of gcov. AKGL_BENCH_SCALE cuts their iteration counts so + # they contribute path coverage without spending ten minutes proving + # nothing; benchutil.h already refuses to enforce budgets in an + # unoptimized build. The performance job below is where the timings are + # taken and the budgets are checked. - name: Test (JUnit) + env: + AKGL_BENCH_SCALE: '0.02' run: | export LD_LIBRARY_PATH="$PWD/build:$PWD/build/deps/SDL:$PWD/build/deps/SDL_image:$PWD/build/deps/SDL_mixer:$PWD/build/deps/SDL_ttf" ctest \ @@ -53,6 +62,119 @@ jobs: path: build/coverage/ if-no-files-found: warn + # The benchmarks, in the only build where their numbers mean anything. + # tests/benchutil.h enforces a budget per measurement only when it is compiled + # optimized and running at full scale, which is exactly this job and no other: + # the coverage job above is Debug, and the memory job below is under valgrind. + # A budget is roughly ten times the baseline recorded in PERFORMANCE.md, which + # is what makes a shared, slower runner viable -- it catches an algorithmic + # regression and ignores a busy machine. If a budget still proves flaky here, + # raise that one budget with a measurement behind it and re-record the + # baseline; do not drop the gate. + performance: + runs-on: ubuntu-latest + steps: + - name: Check out repository code + uses: actions/checkout@v4 + with: + submodules: recursive + - name: Install build dependencies + run: | + sudo apt-get update -y + sudo apt-get install -y \ + cmake gcc pkg-config \ + libasound2-dev libfreetype-dev libharfbuzz-dev \ + libpng-dev libtiff-dev libwebp-dev \ + libudev-dev libx11-dev libxcursor-dev libxext-dev \ + libxfixes-dev libxi-dev libxrandr-dev libxrender-dev \ + libxss-dev libxtst-dev + - name: Configure and build + run: | + cmake -S . -B build \ + -DCMAKE_BUILD_TYPE=RelWithDebInfo + cmake --build build --parallel + # --verbose rather than --output-on-failure: a passing benchmark's output + # is the entire point of running it, and --output-on-failure prints + # nothing at all for a green run. pipefail because the step ends in a + # pipe, whose status would otherwise be tee's -- always 0 -- and a blown + # budget would be reported as a passing step. + - name: Benchmarks (JUnit) + run: | + set -o pipefail + export LD_LIBRARY_PATH="$PWD/build:$PWD/build/deps/SDL:$PWD/build/deps/SDL_image:$PWD/build/deps/SDL_mixer:$PWD/build/deps/SDL_ttf" + ctest \ + --test-dir build \ + -L perf \ + --verbose \ + --output-junit "$(pwd)/perf-junit.xml" \ + 2>&1 | tee "$(pwd)/perf-baseline.txt" + - name: Publish benchmark results + if: always() + uses: mikepenz/action-junit-report@v4 + with: + report_paths: 'perf-junit.xml' + annotate_only: true + detailed_summary: true + include_passed: true + fail_on_failure: 'true' + # Kept whether the job passed or failed: the tables are the point. A run + # that went red says which measurement moved, and a run that went green is + # the next baseline if somebody re-records PERFORMANCE.md. + - name: Upload benchmark tables + if: always() + uses: actions/upload-artifact@v4 + with: + name: performance-baseline + path: perf-baseline.txt + if-no-files-found: warn + + # Every suite under valgrind. No memory-check programs of their own: the perf + # suites scale themselves down when they detect valgrind, which turns them + # into the broadest path coverage in the tree at a cost this job can afford. + # + # continue-on-error until TODO.md "Memory checking" items 34-39 are closed. + # The findings are real and they are all in libakgl, so this job is red today + # by telling the truth; gating on it now would only teach everyone to ignore a + # red build. Remove the flag with the commit that closes the last item -- and + # the JSON one (item 34) is four lines, so that should not be a long wait. + memory_check: + runs-on: ubuntu-latest + continue-on-error: true + steps: + - name: Check out repository code + uses: actions/checkout@v4 + with: + submodules: recursive + - name: Install memory-check dependencies + run: | + sudo apt-get update -y + sudo apt-get install -y \ + cmake gcc pkg-config valgrind \ + libasound2-dev libfreetype-dev libharfbuzz-dev \ + libpng-dev libtiff-dev libwebp-dev \ + libudev-dev libx11-dev libxcursor-dev libxext-dev \ + libxfixes-dev libxi-dev libxrandr-dev libxrender-dev \ + libxss-dev libxtst-dev + # RelWithDebInfo rather than Debug: valgrind needs the symbols, and -O0 + # would leave every inlined frame in the stacks it prints. + - name: Configure and build + run: | + cmake -S . -B build \ + -DCMAKE_BUILD_TYPE=RelWithDebInfo + cmake --build build --parallel + # The character suite is excluded here for the same reason as in the + # coverage job: it fails deliberately, and a failing suite would be + # reported as "valgrind found nothing but ctest exited non-zero". + - name: Memory check + run: scripts/memcheck.sh -E '^character$' + - name: Upload valgrind logs + if: always() + uses: actions/upload-artifact@v4 + with: + name: valgrind-logs + path: build/Testing/Temporary/MemoryChecker.*.log + if-no-files-found: warn + mutation_test: runs-on: ubuntu-latest steps: diff --git a/AGENTS.md b/AGENTS.md index 8e89e5d..0f28e46 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -79,6 +79,21 @@ Run mutation testing with `cmake --build build --target mutation`. For a quick s Generate HTML and Cobertura coverage reports with `cmake -S . -B build-coverage -DAKGL_COVERAGE=ON -DCMAKE_BUILD_TYPE=Debug`, then build and run CTest. Reports are written to `build-coverage/coverage/`; this mode requires `gcovr` and GCC or Clang. +## Continuous integration + +`.gitea/workflows/ci.yaml` runs four jobs on every push, and each one exists +because the others cannot do its work: + +| Job | Build | What it is for | +|---|---|---| +| `cmake_build` | Debug + `AKGL_COVERAGE=ON` | The unit suites and the coverage report. The perf suites run here too, at `AKGL_BENCH_SCALE=0.02`, for path coverage only — a timing taken under gcov is a measurement of gcov. | +| `performance` | RelWithDebInfo | `ctest -L perf`. The only job where budgets are enforced, because `tests/benchutil.h` enforces them only when compiled optimized at full scale. Keeps the tables as an artifact. | +| `memory_check` | RelWithDebInfo | `scripts/memcheck.sh`, every suite under valgrind. `continue-on-error` until TODO.md "Memory checking" items 34-39 are closed; keeps the valgrind logs as an artifact. | +| `mutation_test` | Debug | One focused source file, so CI stays bounded. | + +The `character` suite is excluded wherever a whole run is selected: it fails +deliberately, and pinning current-but-wrong behavior is not what it is for. + ## Coding Style The canonical style is Emacs `cc-mode` **`stroustrup`**, with tabs enabled. This