Type at the window with a real keyboard: the akgl_typing test
Some checks failed
akbasic CI Build / cmake_build (push) Failing after 18s
akbasic CI Build / sanitizers (push) Failing after 13s
akbasic CI Build / coverage (push) Failing after 16s
akbasic CI Build / akgl_build (push) Failing after 17s
akbasic CI Build / mutation_test (push) Failing after 17s
Some checks failed
akbasic CI Build / cmake_build (push) Failing after 18s
akbasic CI Build / sanitizers (push) Failing after 13s
akbasic CI Build / coverage (push) Failing after 16s
akbasic CI Build / akgl_build (push) Failing after 17s
akbasic CI Build / mutation_test (push) Failing after 17s
xdotool closes the gap that shipped the missing SDL_StartTextInput(). Every other keyboard test synthesises SDL events, which covers the code downstream of SDL and cannot cover the code upstream of it -- so the suite stayed green while the real keyboard was dead. akgl_typing starts the driver under a pty, waits for the window with xdotool search --sync, gives it focus, and types a program containing a string literal and a lower-case one, polling the mirrored stdout for what they print. Verified by reverting both halves of the text-input fix and watching it fail with the reported symptom: READY, and nothing after it. Skips rather than fails without a display, xdotool, script(1) or a window manager -- none of those means the answer is no. It steals keyboard focus for about fifteen seconds; AKBASIC_SKIP_INTERACTIVE=1 skips it deliberately. xdotool and script(1) are documented as optional test dependencies, alongside what gcovr and python3 already bought. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -214,6 +214,11 @@ jobs:
|
|||||||
# they are the ones that must keep passing when SDL is present, *and* they
|
# they are the ones that must keep passing when SDL is present, *and* they
|
||||||
# are what proves the SDL frontend changes no output anywhere in the
|
# are what proves the SDL frontend changes no output anywhere in the
|
||||||
# corpus.
|
# corpus.
|
||||||
|
# akgl_typing reports Skipped here and that is correct: it needs a real X
|
||||||
|
# server, a window manager and xdotool to type at a focused window, and a
|
||||||
|
# CI runner has none of the three. It is a developer-machine gate, and the
|
||||||
|
# one that covers the path upstream of SDL -- see its script for why that
|
||||||
|
# distinction cost a bug once.
|
||||||
- name: test with libakgl
|
- name: test with libakgl
|
||||||
env:
|
env:
|
||||||
SDL_VIDEODRIVER: dummy
|
SDL_VIDEODRIVER: dummy
|
||||||
|
|||||||
@@ -265,6 +265,12 @@ ctest --test-dir build --output-on-failure
|
|||||||
ctest --test-dir build -R <name> --output-on-failure # single test
|
ctest --test-dir build -R <name> --output-on-failure # single test
|
||||||
```
|
```
|
||||||
|
|
||||||
|
**Optional test tooling for this repository.** `xdotool` and `script`(1) drive the `akgl_typing`
|
||||||
|
test, which types at a real focused SDL window -- the only test that covers the path upstream of
|
||||||
|
SDL, and the one that would have caught the missing `SDL_StartTextInput()`. It needs a real X
|
||||||
|
server, steals keyboard focus for about fifteen seconds, and skips itself (CTest `Skipped`, not
|
||||||
|
a failure) when it cannot run. `AKBASIC_SKIP_INTERACTIVE=1` skips it deliberately.
|
||||||
|
|
||||||
Extra harnesses, by library (option prefixes differ per repo):
|
Extra harnesses, by library (option prefixes differ per repo):
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
|
|||||||
@@ -317,6 +317,34 @@ if(AKBASIC_WITH_AKGL)
|
|||||||
WORKING_DIRECTORY "${CMAKE_CURRENT_SOURCE_DIR}/tests"
|
WORKING_DIRECTORY "${CMAKE_CURRENT_SOURCE_DIR}/tests"
|
||||||
TIMEOUT 60
|
TIMEOUT 60
|
||||||
ENVIRONMENT "SDL_VIDEODRIVER=dummy;SDL_AUDIODRIVER=dummy;SDL_RENDER_DRIVER=software")
|
ENVIRONMENT "SDL_VIDEODRIVER=dummy;SDL_AUDIODRIVER=dummy;SDL_RENDER_DRIVER=software")
|
||||||
|
|
||||||
|
# The keyboard test, and the only one in the repository that uses a real X
|
||||||
|
# server rather than the dummy driver. Note the deliberate absence of an
|
||||||
|
# ENVIRONMENT line: forcing the dummy driver here would defeat the whole point.
|
||||||
|
#
|
||||||
|
# Everything else synthesises SDL events, which tests the code *downstream* of
|
||||||
|
# SDL and cannot test the code upstream of it. That gap shipped a bug -- the
|
||||||
|
# frontend never called SDL_StartTextInput(), so SDL emitted no text-input
|
||||||
|
# events, the line editor dropped every keystroke, and the suite stayed green
|
||||||
|
# because it was pushing those events itself. This one drives xdotool at a
|
||||||
|
# focused window, so X11, SDL's composition and the window manager are all in
|
||||||
|
# the path.
|
||||||
|
#
|
||||||
|
# Needs xdotool and script(1), and it **steals keyboard focus** for a few
|
||||||
|
# seconds. Skipped rather than failed when it cannot run -- headless, no
|
||||||
|
# xdotool, no window manager -- because none of those means the answer is no.
|
||||||
|
# AKBASIC_SKIP_INTERACTIVE=1 skips it while you are using the machine.
|
||||||
|
find_program(XDOTOOL_EXECUTABLE xdotool)
|
||||||
|
if(NOT XDOTOOL_EXECUTABLE)
|
||||||
|
message(STATUS "xdotool not found: the akgl_typing test will skip. "
|
||||||
|
"Install it to cover the real keyboard path.")
|
||||||
|
endif()
|
||||||
|
_add_test(NAME akgl_typing
|
||||||
|
COMMAND ${CMAKE_CURRENT_SOURCE_DIR}/tests/akgl_typing.sh $<TARGET_FILE:basic>)
|
||||||
|
_set_tests_properties(akgl_typing PROPERTIES
|
||||||
|
WORKING_DIRECTORY "${CMAKE_CURRENT_SOURCE_DIR}"
|
||||||
|
TIMEOUT 180
|
||||||
|
SKIP_RETURN_CODE 77)
|
||||||
endif()
|
endif()
|
||||||
|
|
||||||
if(AKBASIC_TESTS OR AKBASIC_WILL_FAIL_TESTS OR AKBASIC_KNOWN_FAILING_TESTS)
|
if(AKBASIC_TESTS OR AKBASIC_WILL_FAIL_TESTS OR AKBASIC_KNOWN_FAILING_TESTS)
|
||||||
|
|||||||
24
README.md
24
README.md
@@ -28,6 +28,30 @@ ctest --test-dir build-akgl --output-on-failure
|
|||||||
./build-akgl/basic tests/reference/language/functions.bas
|
./build-akgl/basic tests/reference/language/functions.bas
|
||||||
```
|
```
|
||||||
|
|
||||||
|
### Optional tools, and what you lose without them
|
||||||
|
|
||||||
|
Everything above works with cmake and a C compiler. Three tools are optional, and each one buys
|
||||||
|
a specific test rather than general convenience:
|
||||||
|
|
||||||
|
| Tool | Buys | Without it |
|
||||||
|
|---|---|---|
|
||||||
|
| `xdotool` + `script`(1) | `akgl_typing` — types at a real focused SDL window with a real keyboard | The test reports **Skipped**. Nothing fails |
|
||||||
|
| `gcovr` | the `coverage` target and its 90%-of-lines gate | `-DAKBASIC_COVERAGE=ON` will not configure |
|
||||||
|
| `python3` | `scripts/mutation_test.py` and the `mutation` target | The target is not created |
|
||||||
|
|
||||||
|
**`akgl_typing` is worth installing `xdotool` for.** It is the only test that exercises the path
|
||||||
|
a person actually uses — X11 delivers a key press to a focused window, SDL composes it, libakgl
|
||||||
|
rings it, the editor echoes it, the interpreter runs it. Every other keyboard test synthesises
|
||||||
|
SDL events, which tests the code *downstream* of SDL and cannot test the code upstream of it.
|
||||||
|
That gap shipped a bug: the frontend never called `SDL_StartTextInput()`, so SDL emitted no
|
||||||
|
text-input events at all, the line editor dropped every keystroke, and the whole suite stayed
|
||||||
|
green because it was pushing those events itself.
|
||||||
|
|
||||||
|
It needs a real X server and **it steals keyboard focus for about fifteen seconds**. Set
|
||||||
|
`AKBASIC_SKIP_INTERACTIVE=1` to skip it while you are using the machine. It skips itself
|
||||||
|
automatically when there is no display, no `xdotool`, or no window manager answering — CTest
|
||||||
|
reports `Skipped` rather than a failure, because none of those means the answer is no.
|
||||||
|
|
||||||
There are two workflows. **`.gitea/workflows/ci.yaml`** runs on every push: the suite,
|
There are two workflows. **`.gitea/workflows/ci.yaml`** runs on every push: the suite,
|
||||||
ASan+UBSan, coverage gated at 90% of lines, and mutation testing over two files. The coverage
|
ASan+UBSan, coverage gated at 90% of lines, and mutation testing over two files. The coverage
|
||||||
report is uploaded as a `code-coverage` artifact.
|
report is uploaded as a `code-coverage` artifact.
|
||||||
|
|||||||
23
TODO.md
23
TODO.md
@@ -529,11 +529,22 @@ box's left edge at x=100 is a solid white line (mean 255), the box interior is b
|
|||||||
outlines rather than fills — and an empty corner is black. stdout carried `HELLO WORLD` at the
|
outlines rather than fills — and an empty corner is black. stdout carried `HELLO WORLD` at the
|
||||||
same time. `QUIT` exits cleanly with status 0.
|
same time. `QUIT` exits cleanly with status 0.
|
||||||
|
|
||||||
**What that check does not cover, and could not here:** typing at a real focused window. No key
|
**The typing half is no longer manual, and it is no longer a gap.** It used to read "somebody
|
||||||
synthesis tool (`xdotool`, `xte`, python-xlib) is installed on this machine, so the keyboard
|
should type at it once", because no key-synthesis tool was installed. `xdotool` now is, and
|
||||||
path is proved only through SDL's own event queue in `tests/akgl_frontend.c` — which is the
|
`tests/akgl_typing.sh` is registered as the `akgl_typing` CTest case: it starts the driver under
|
||||||
same queue a real keyboard delivers into, but not the same as a window manager giving the
|
a pty, waits for the window with `xdotool search --sync`, gives it keyboard focus, types a
|
||||||
window focus. Somebody should type at it once.
|
program containing a **string literal** and a **lower-case** one, and polls the mirrored stdout
|
||||||
|
for what they print.
|
||||||
|
|
||||||
|
That case earns its keep twice over. It is the only test that covers the path *upstream* of SDL
|
||||||
|
— X11 → SDL composition → libakgl's ring → the editor — and everything else synthesises SDL
|
||||||
|
events, which is precisely how the missing `SDL_StartTextInput()` shipped with a green suite.
|
||||||
|
Confirmed by reverting both halves of that fix and watching it fail with the same symptom that
|
||||||
|
was reported: `READY`, and nothing after it.
|
||||||
|
|
||||||
|
It needs a real X server, a window manager and `xdotool`, and it steals keyboard focus for about
|
||||||
|
fifteen seconds. Absent any of those it reports CTest `Skipped` rather than failing, which is
|
||||||
|
what it does in CI. `AKBASIC_SKIP_INTERACTIVE=1` skips it deliberately.
|
||||||
|
|
||||||
### The five things that had to be worked around — **all gone**
|
### The five things that had to be worked around — **all gone**
|
||||||
|
|
||||||
@@ -1122,7 +1133,7 @@ requirement; exactly one case has diverged on purpose since, and
|
|||||||
| Gate | Result |
|
| Gate | Result |
|
||||||
|---|---|
|
|---|---|
|
||||||
| `ctest` | 77/77 — 41 reference golden cases, 9 local ones, 25 unit tests, 2 embedding examples. No known-failing tests: the list is empty |
|
| `ctest` | 77/77 — 41 reference golden cases, 9 local ones, 25 unit tests, 2 embedding examples. No known-failing tests: the list is empty |
|
||||||
| `ctest` with `-DAKBASIC_WITH_AKGL=ON` | 76/76 — the same, minus the three `no_device` cases the SDL driver contradicts, plus `akgl_backends` and `akgl_frontend`; the `akgl_build` CI job |
|
| `ctest` with `-DAKBASIC_WITH_AKGL=ON` | 77/77 on a machine with a display; 76 passed + 1 skipped headless. The same set minus the three `no_device` cases the SDL driver contradicts, plus `akgl_backends`, `akgl_frontend` and `akgl_typing` — the last of which is the skip, and the `akgl_build` CI job is where it skips |
|
||||||
| Golden corpus | 41/41 byte-exact from `tests/reference/` — **and 41/41 again through the SDL binary**, which is most of what proves the frontend changes no output |
|
| Golden corpus | 41/41 byte-exact from `tests/reference/` — **and 41/41 again through the SDL binary**, which is most of what proves the frontend changes no output |
|
||||||
| ASan + UBSan | 77/77 |
|
| ASan + UBSan | 77/77 |
|
||||||
| Line coverage | 94.6% (3857/4076) — above the 90% gate |
|
| Line coverage | 94.6% (3857/4076) — above the 90% gate |
|
||||||
|
|||||||
115
tests/akgl_typing.sh
Executable file
115
tests/akgl_typing.sh
Executable file
@@ -0,0 +1,115 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
#
|
||||||
|
# Type at the SDL window with a real keyboard, and check what comes out.
|
||||||
|
#
|
||||||
|
# This is the only test in the repository that exercises the path a person
|
||||||
|
# actually uses: X11 delivers a key press to a focused window, SDL composes it,
|
||||||
|
# libakgl rings it, the line editor echoes it, and the interpreter runs it. Every
|
||||||
|
# other keyboard test synthesises SDL events, which is a fine way to test the
|
||||||
|
# code downstream of SDL and no way at all to test the code upstream of it.
|
||||||
|
#
|
||||||
|
# **It exists because that gap shipped a bug.** The frontend never called
|
||||||
|
# SDL_StartTextInput(), so SDL emitted no text-input events, every keystroke
|
||||||
|
# arrived with an empty `text`, and the editor dropped the lot -- no echo, and
|
||||||
|
# nothing on stdout either, because RUN could never be typed. The suite was
|
||||||
|
# green throughout, because it pushed the text-input events itself.
|
||||||
|
#
|
||||||
|
# Skips rather than fails when it cannot run: no display, no xdotool, no
|
||||||
|
# script(1), no window, or no focus are all "this machine cannot answer the
|
||||||
|
# question", not "the answer is no". CTest is told exit 77 means skipped.
|
||||||
|
#
|
||||||
|
# **It steals keyboard focus for a few seconds.** That is inherent -- the point
|
||||||
|
# is a focused window. Set AKBASIC_SKIP_INTERACTIVE=1 to skip it while you work.
|
||||||
|
|
||||||
|
set -u
|
||||||
|
|
||||||
|
SKIP=77
|
||||||
|
BASIC="${1:?usage: akgl_typing.sh <path to basic>}"
|
||||||
|
WORK="$(mktemp -d)"
|
||||||
|
trap 'rm -rf "${WORK}"' EXIT
|
||||||
|
|
||||||
|
skip() { echo "SKIP: $*"; exit "${SKIP}"; }
|
||||||
|
fail() { echo "FAIL: $*"; exit 1; }
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------- environment
|
||||||
|
[ "${AKBASIC_SKIP_INTERACTIVE:-0}" = "1" ] && skip "AKBASIC_SKIP_INTERACTIVE=1"
|
||||||
|
[ -n "${DISPLAY:-}" ] || skip "no DISPLAY; this needs a real X server, not the dummy driver"
|
||||||
|
command -v xdotool >/dev/null 2>&1 || skip "xdotool is not installed"
|
||||||
|
command -v script >/dev/null 2>&1 || skip "script(1) is not installed"
|
||||||
|
xdotool getactivewindow >/dev/null 2>&1 || skip "no window manager is answering on ${DISPLAY}"
|
||||||
|
|
||||||
|
# script(1) gives the driver a pty for stdin, which is what makes it choose the
|
||||||
|
# *window* as its console. Redirected stdin is a deliberate signal to read the
|
||||||
|
# pipe instead -- see run_akgl() in src/main.c -- and under CTest stdin is never
|
||||||
|
# a terminal, so without this the driver would ignore the keyboard entirely and
|
||||||
|
# the test would be measuring nothing.
|
||||||
|
OUT="${WORK}/typescript"
|
||||||
|
script -qefc "${BASIC}" "${OUT}" >/dev/null 2>&1 &
|
||||||
|
DRIVER=$!
|
||||||
|
|
||||||
|
# shellcheck disable=SC2317 # reached only through the EXIT trap below
|
||||||
|
cleanup_driver() {
|
||||||
|
kill "${DRIVER}" 2>/dev/null
|
||||||
|
wait "${DRIVER}" 2>/dev/null
|
||||||
|
}
|
||||||
|
trap 'cleanup_driver; rm -rf "${WORK}"' EXIT
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------ the window
|
||||||
|
# --sync blocks until it appears rather than sleeping and hoping.
|
||||||
|
WID="$(timeout 20 xdotool search --sync --onlyvisible --name '^BASIC$' 2>/dev/null | head -1)"
|
||||||
|
[ -n "${WID}" ] || skip "the BASIC window never appeared (headless, or a WM that does not map it)"
|
||||||
|
|
||||||
|
xdotool windowactivate --sync "${WID}" >/dev/null 2>&1
|
||||||
|
xdotool windowfocus --sync "${WID}" >/dev/null 2>&1
|
||||||
|
FOCUS="$(xdotool getwindowfocus 2>/dev/null)"
|
||||||
|
[ "${FOCUS}" = "${WID}" ] || skip "could not give the window keyboard focus (got '${FOCUS}', wanted '${WID}')"
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------- the typing
|
||||||
|
# Poll for expected output instead of sleeping a guessed interval: a fixed sleep
|
||||||
|
# is either flaky or slow, and usually both.
|
||||||
|
await() {
|
||||||
|
local want="$1" limit="${2:-15}" i=0
|
||||||
|
while [ "${i}" -lt $((limit * 10)) ]; do
|
||||||
|
if tr -d '\r' < "${OUT}" 2>/dev/null | grep -qF "${want}"; then
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
sleep 0.1
|
||||||
|
i=$((i + 1))
|
||||||
|
done
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
|
||||||
|
type_line() {
|
||||||
|
xdotool type --clearmodifiers --delay 30 "$1"
|
||||||
|
xdotool key --clearmodifiers Return
|
||||||
|
}
|
||||||
|
|
||||||
|
await "READY" 20 || skip "the interpreter never reached its prompt"
|
||||||
|
|
||||||
|
# A string literal on purpose: the double quote is the character that was
|
||||||
|
# unreachable before the keystroke ring carried composed text, and a BASIC
|
||||||
|
# program that cannot contain a string is not much of a BASIC.
|
||||||
|
type_line '10 PRINT "TYPED OK"'
|
||||||
|
type_line 'RUN'
|
||||||
|
|
||||||
|
if ! await "TYPED OK" 15; then
|
||||||
|
echo "--- what the driver actually wrote ---"
|
||||||
|
tr -d '\r' < "${OUT}" 2>/dev/null | sed -n '1,20p'
|
||||||
|
fail "typing at the window produced nothing: the keyboard path is broken"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Lower case has to survive too. It could not before libakgl 0.3.0, because a
|
||||||
|
# keycode cannot express shift and the editor folded everything to upper case.
|
||||||
|
type_line '20 PRINT "lower"'
|
||||||
|
type_line 'RUN'
|
||||||
|
if ! await "lower" 15; then
|
||||||
|
echo "--- what the driver actually wrote ---"
|
||||||
|
tr -d '\r' < "${OUT}" 2>/dev/null | sed -n '1,20p'
|
||||||
|
fail "lower case did not survive the keyboard: composed text is not reaching the editor"
|
||||||
|
fi
|
||||||
|
|
||||||
|
type_line 'QUIT'
|
||||||
|
await "" 1 >/dev/null 2>&1 || true
|
||||||
|
|
||||||
|
echo "PASS: typed at a real focused window and the interpreter ran it"
|
||||||
|
exit 0
|
||||||
Reference in New Issue
Block a user