[WIP] Add experimental gameplay tests - #8926
Draft
4ian wants to merge 31 commits into
Draft
Conversation
Complete v1 plan for gameplay tests: Core model (gd::GameplayTest), GDJS harness driven over the debugger protocol, editor UI (test editor tab, project manager section, overlay preview frame, CLI runner), the run_game_tests orchestrator tool with a tester sub-agent on the backend, prompt/versioning strategy (toolsVersion v13), starter test suite and rollout phases. Folds in all lessons from the prior new-gameplay-tester experiment. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AaQQdPZ68X8zkybtsxWE1b
- Generic Core model: gd::Test / gd::TestsContainer with a serialized
type attribute ("gameplay" in v1) and JSON key "tests", leaving room
for future test types; UI label stays "Gameplay tests".
- Orchestrator tool renamed to run_tests, priced 0.5 credits.
- Last-run summary persisted in the project JSON (decided).
- Approval semantics: no approval to run an existing test; persisting a
new/changed test triggers the edit-approval row when auto-edit is off
(via a dynamic modifiesProject check).
- Test editor falls back to EditorNavigator (bottom tabs) on small
screens, like the scene/extension editors.
- Extension-scoped tests confirmed: synthetic empty scene + spawn().
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AaQQdPZ68X8zkybtsxWE1b
…sts P1) Tests (JS source + metadata + last-run summary) are stored on gd::Project and gd::EventsFunctionsExtension, serialized under a 'tests' element (only when non-empty, so existing projects are unchanged). The 'type' attribute defaults to 'gameplay' and leaves room for future test kinds. Tests are stripped from exports/previews by ProjectStripper: the test runner sends the source over the debugger channel at run time instead. Includes GDevelop.js bindings (Test, TestsContainer, GetTests on Project and EventsFunctionsExtension), regenerated types.d.ts and Flow types (generators run against the updated Bindings.idl; the wasm glue itself must be rebuilt by CI as emscripten is not available here), Core tests (GDCore_tests pass: 73 cases / 8500 assertions) and GDevelop.js tests. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AaQQdPZ68X8zkybtsxWE1b
A gameplay test is a JS script (async (harness) => {...}) run against a
real game preview. The harness pauses the game main loop and steps frames
deterministically (fixed dt, speedFactor frames per animation frame - so
tests run faster than real time), simulates keyboard/mouse/touch through
the InputManager (accepting GDevelop and Web API key names, with
location-aware key codes), inspects objects/variables/behaviors via
network-sync-data snapshots, supports scenario setup (spawn, external
layout injection, skipCreatingInstances), navigation hints, FPS aiming
with an adaptive mouse-delta controller and a pointer-lock shim,
fail-fast assertions, downscaled JPEG screenshots, per-section profiling,
and an auto-recorded event log (spawn/remove/stuck/scene changes).
The editor talks to it over the debugger connection: 'gameplayTest.run'
runs a test (progress streamed as 'gameplayTest.progress', result always
sent as 'gameplayTest.result'), 'gameplayTest.stop' interrupts it. The
runner file is only included in previews with a debugger client - test
sources never ship in exported games.
The paused main loop still calls InputManager.onFrameEnded every
animation frame, which would clear simulated inputs between manually
stepped frames: it is neutralized during a run and called by the harness
after each stepped frame instead.
Covered by GDJS Karma tests (gameplaytestharness.js, passing in headless
Chrome; the only failing suites are the pre-existing Firebase e2e tests
which require network access).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AaQQdPZ68X8zkybtsxWE1b
… CLI, AI tool (P2) Gameplay tests (stored on the project via gd::Test) are now visible and runnable from the editor: - A 'Gameplay tests' section in the Project Manager (add/rename/delete/ copy/paste/duplicate/reorder, run from the context menu or the row button), opening a new 'gameplay-test' editor tab: a Monaco code editor + a properties panel (description, last run summary, run/stop, 'Edit with AI' which copies a pre-filled prompt and opens the Ask AI). - A GameplayTestRunner that exports a fresh preview of the in-memory project into a dedicated small overlay iframe (bottom-left, with Hide/Stop controls - kept 1x1 while hidden so the game loop keeps running), waits for the game to boot, then drives tests over the debugger protocol (gameplayTest.run/progress/result) with an editor-side watchdog. Runs are globally serialized. Last-run summaries are persisted on the tests (marks the project as having unsaved changes). Dependencies (preview launcher) are registered by MainFrame so the project manager, command palette, CLI and AI function calls can all run tests. - A 'gameplay-test-frame' slot in both debugger servers (browser postMessage and Electron), next to the in-game-edition embedded frame. - Preview launchers accept isForGameplayTest (separate export directory, window-message debugger client, no preview window). The legacy S3 launcher rejects it explicitly. - Command palette: OPEN_GAMEPLAY_TEST, RUN_GAMEPLAY_TEST, RUN_ALL_GAMEPLAY_TESTS. - CLI: RUN_ALL_TESTS awaitable runner (optionally filtered by test names via --cmd-args) with a meaningful exit code for CI. - AI integration: run_gameplay_test client-side editor function (used by the backend 'tester' sub-agent; persisting a new/changed test counts as a project modification for edit approval - via the new per-call getModifiesProject hook - while just running does not), a run_tests chat rendering stub for the server-side orchestrator tool, tests listed in the SimplifiedProject sent to the AI, and AI_ORCHESTRATOR_TOOLS_VERSION bumped to v13 (requires the matching generation-api and prompts deployment). Not in this commit (follow-ups): a 'Tests' section in the extension editor UI (extension-scoped tests are fully supported by the model, the runner and the AI tool), harness autocompletions in the test code editor, and pre-filling the Ask AI chat input directly. Flow passes with 0 errors. Jest suites requiring libGD.js could not be run in this environment (libGD.js download unavailable); they run in CI against the rebuilt bindings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AaQQdPZ68X8zkybtsxWE1b
Extension-scoped gameplay tests are now visible and editable in the extension editor's list, next to Objects, Behaviors and Functions: add/rename/delete/copy/paste/duplicate/reorder, run from the row button or the context menu, and open in the same gameplay test editor tab as project tests (reusing the editor, toolbar and runner as-is). The clipboard kind is shared with the Project Manager so tests can be copied between the project and extensions. Open/rename/delete/run are routed to the MainFrame through a single gameplayTestsCallbacks prop (threaded via EditorTabsPane and the popped-out editor windows), so renames and deletions update/close the matching editor tabs, like project tests. The existing project-scoped handlers were generalized to take a scope. Flow passes with 0 errors and the full newIDE test suite is green (126 suites, 1142 tests) - now run against a locally built libGD.js (emsdk 3.1.21, dev variant), which also validates the new Test/ TestsContainer bindings end-to-end (GDevelop.js Core.js suite: 203 tests pass). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AaQQdPZ68X8zkybtsxWE1b
- Hide and tear down the gameplay test frame when a test run ends or is stopped, and exclude the gameplay test frame from the running previews count so the "update" preview button stays disabled during test runs. - Block all mouse/keyboard/touch input on the gameplay test iframe: the game is only driven by the harness. - Make the test code a central editor with no title bar, and add a toolbar toggle button to show/hide the test properties panel, like in the scene editor. - Hide the preview and share buttons in the toolbar when a gameplay test tab is active (and in popped-out editor windows). - Make "Edit with AI" pre-fill the Ask AI chat input with a prompt about the current test. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AaQQdPZ68X8zkybtsxWE1b
- Make the stop buttons actually stop a test: the stop request now interrupts the test script even when it awaits something else than the harness (a timer, a fetch...), by racing the script against a promise rejected on stop. Previously the stop flag was only checked when the script stepped frames. - Make stopping work at any point of a run: a batch-level stop controller aborts a run still exporting/booting the game and skips the remaining tests of a batch (reported as "stopped"). - Report progress at frame 0 as soon as the script starts, so the frame and properties panel show the test as running instead of staying on "Starting the game..." (and treat frame 0 as a running state). - Keep the gameplay test frame open at the end of a run, showing the frozen (paused and muted) game and the outcome; its button stops the test while running and closes the frame afterwards. The frame is always closed and reopened fresh when a new run starts. - Give newly created gameplay tests a default commented source, as a starting point for humans writing a test manually. - Focus the Ask AI input after "Edit with AI" pre-fills it. - Provide autocompletions for the `harness` API in the desktop app (the gameplay-tests runtime folder was not read for completions). - Hide the "'await' expression is only allowed within an async function" diagnostic in the test editor (the test source is actually run inside an async function), via a per-model diagnostics message filter. - Add tests for stopping a test awaiting a raw promise and for freezeWhenFinished. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AaQQdPZ68X8zkybtsxWE1b
v13 was taken on master by the hidden-instances / set-instance-variables tooling, so gameplay tests now ship as tools version v14 (the editor constant was already updated when rebasing). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AaQQdPZ68X8zkybtsxWE1b
The harness reference is prompt content and now lives with the AI prompts; Monaco autocompletions are fed directly by gameplay-test-runner.ts. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AaQQdPZ68X8zkybtsxWE1b
In development, the AI uses tools v14 and the gameplay tests sections are shown in the project manager and extension editor (plus the command palette commands). Otherwise the editor stays on tools v13 and none of the gameplay tests UI is visible, so this can be deployed without the feature appearing. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AaQQdPZ68X8zkybtsxWE1b
The runner dependencies were registered by a MainFrame effect with no dependency list: on every commit, React's passive-effects flush ran its cleanup (unregistering) before child effects — and the AI function calls processor is a child effect that synchronously starts the test run, so it always saw an empty registry. Manual runs, triggered from event handlers between commits, were unaffected. Register once on mount instead, reading the latest values through refs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AaQQdPZ68X8zkybtsxWE1b
The test source must be the body of `async (harness) => {...}`, but AI
models and users sometimes pass the whole function: evaluated as-is it was
a no-op expression reported as a false 'passed' with 0 frames. The runner
now auto-unwraps a wrapper-shaped source and calls it, and any run that
completes without stepping a frame nor asserting anything is reported as
an error instead of a pass.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AaQQdPZ68X8zkybtsxWE1b
4ian
commented
Aug 5, 2026
| // run BEFORE this component's) — and the AI function calls processor is a | ||
| // child effect that synchronously starts a test run, so it would always see | ||
| // "no editor registered". | ||
| const unsavedChangesRef = React.useRef(unsavedChanges); |
Owner
Author
There was a problem hiding this comment.
Todo: do we have an helper for this?
Lets AI edit scripts delete gameplay tests or change their name,
description and position (changed_properties approach — never their
source, which must go through the gameplay test runner so it is executed
and verified). Returns the ordered tests of the scope (capped at 50) so
changes are self-verifying. Renames and deletions keep open editor tabs
consistent via onProjectItemRenamedOutsideEditor ('gameplay-test' kind)
and the new onWillDeleteGameplayTest callback.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AaQQdPZ68X8zkybtsxWE1b
- New EditorBottomTabsSwitcher (UI): editors switched with bottom tabs for
small screens. The gameplay test editor uses it on mobile: properties by
default, a second tab for the code (instead of the side-by-side mosaic).
- GameplayTestScope is now explicit ({type: 'project'} or
{type: 'extension', extensionName}) instead of 'project' | extensionName,
removing the ambiguity with an extension literally named 'project'. Tool
arguments take {type, extension_name} accordingly.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AaQQdPZ68X8zkybtsxWE1b
…erflow EditorBottomTabsSwitcher now renders icon + label tabs with the same design and metrics as the home page mobile menu (active top border, secondary color for inactive tabs): Properties (edit icon) and Code (console icon). The editors container and the code editor column clamp their width (min-width: 0, overflow hidden) so the code editor can never leak past the screen width on small screens. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AaQQdPZ68X8zkybtsxWE1b
Snapshots now expose the conditions and expressions of each object and behavior, evaluated at snapshot time under the exact names used in the game's events: behaviors.PlatformerObject.state.IsOnFloor, .state.CurrentSpeed, configuration values (.state.MaxSpeed), and the properties of events-based behaviors/objects (.state.PropertyHealth). The inspectors are derived by the editor from the extensions' own declarations (single source of truth — built-in, TS-based and events-based extensions alike) and sent with the run payload; the harness evaluates the same public getters the game's events call. Replaces the unreadable minified network sync data, still reachable via the getRawBehaviorData last-resort escape hatch. Reading an unknown state name throws with the list of available names, so mistakes self-correct in one run. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AaQQdPZ68X8zkybtsxWE1b
…t load - The gameplay test event log now tracks the scene INSTANCE, not only its name: a replacement by a new instance of the same scene (a restart, or external interference) is recorded as a 'sceneReset' event instead of going unnoticed. - Asset loading (RuntimeGame._loadAssetsWithLoadingScreen) now restores the paused state it found instead of unconditionally unpausing: a game paused by a gameplay test (or the debugger) stays paused when a scene change loads assets mid-test. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AaQQdPZ68X8zkybtsxWE1b
…ontrol probing, progress tracking The should* booleans of getNavigationHint encoded a straight-line pursuit policy that silently deadlocks (e.g. a target directly below the platform the object stands on). The harness now provides facts and measurements, and the test script owns the navigation strategy: - getRelativePosition (replaces getNavigationHint): pure geometry - relative vector, distance, horizontalDistance, yawDiff, pitchDiff, reached. No navigation advice. - probeControls: measure what each key actually does (net + extreme displacements vs a no-key baseline, yaw change), restarting the scene between probes. Works for any control scheme, 2D or 3D. - makeProgressTracker: distance-to-target progress with stall detection (records a 'stuck' event when a stall starts), usable in manual loops. - Time-budget yielding (12ms) replaces the per-call speedFactor frame counter: every stepping pattern (stepUntil, stepFrames(1) loops...) now renders while running and stays interruptible, at near-full speed. The unused speedFactor payload option is removed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AaQQdPZ68X8zkybtsxWE1b
…ger commands during tests - probeControls -> resetSceneAndProbeControls and lookToward -> lookTowardWithMouseDelta: the names document the scene restarts and the mouse-delta mechanism. - Document 2D/3D support on every helper (hasLineOfSight is explicitly 2D-only: 2D hitboxes in the X/Y plane, Z ignored; everything else handles Z when present). - While a gameplay test runs, the debugger client ignores state-mutating commands (play/pause, hot reloads, instance edits, set/call...) and answers with a commandIgnored message: an unpause would make the main loop step in parallel, a hot-reload would reset instances mid-test. gameplayTest.stop and read-only commands stay allowed. - The editor disables the Preview/Update toolbar button while a gameplay test batch runs (useIsGameplayTestRunInProgress on the runner's batch lifecycle). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AaQQdPZ68X8zkybtsxWE1b
The menu offers 'Run as quickly as possible' (the default, also what the main button does), 'Run at 4x speed' and 'Run at normal speed', so a paced run can be watched to understand what the test does. The pacing is done by the harness (payload speedFactor: game seconds simulated per real second, omitted = full speed): the stepping loop waits for the wall clock to catch up with the simulated game time, re-anchoring when a run falls behind (heavy frame, scene load) instead of rushing to catch up. Paced runs get a 5-minute default timeout instead of 30s (they are slow by design and can be stopped anytime). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AaQQdPZ68X8zkybtsxWE1b
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AaQQdPZ68X8zkybtsxWE1b
…ve test mouse deltas Mouse-look extensions (like MousePointerLock's FirstPersonPointerMapper) do not read the input manager: they listen to the canvas 'pointermove' DOM events and gate everything on document.pointerLockElement. The gameplay test pointer lock shim now: - patches canvas.requestPointerLock/document.exitPointerLock and fakes document.pointerLockElement (dispatching 'pointerlockchange'), so the pointer is reported as locked as soon as the game requests it - without ever locking the real mouse; - is fully restored when the test ends. setMouseDelta now also dispatches a real 'pointermove' event on the canvas carrying movementX/movementY, in addition to feeding the input manager - so both channels receive the simulated deltas and lookTowardWithMouseDelta can rotate FPS cameras driven by such extensions. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AaQQdPZ68X8zkybtsxWE1b
…k, pointer lock hint
- Scene change events (sceneChanged/sceneReset) now carry a 'cause':
'harness' (a goToScene), 'game' (a change requested during a stepped
frame), 'networkSync', or 'external' (interference - with a call stack
snippet in causeDetail). The scene stack is instrumented for the
duration of the run and restored afterwards.
- stopProfiling now returns a flat, JSON-safe summary ({avgStepTimeMs,
sections: [{name, avgTimeMs}]} sorted by time descending, nested
sections flattened as 'parent > child') instead of the raw profiler
tree (circular, map-shaped).
- lookTowardWithMouseDelta returns {aimed, yawDiff, pitchDiff,
sawYawResponse, sawPitchResponse} (null if object/target missing), with
a per-axis gain adaptation. When the measured pitch never responds (a
game mapping the vertical aim to another axis), the vertical input is
undone and the aim falls back to yaw-only instead of winding the view
into the ground.
- setMouseDelta records a one-time hint in the test logs when the game
never requested the pointer lock (mouse-look usually engages after a
click).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AaQQdPZ68X8zkybtsxWE1b
…ion semantics Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AaQQdPZ68X8zkybtsxWE1b
…e API
- SceneStack now records the cause of its own changes first-class:
runWithSceneChangeCause(cause, action) attributes changes to a label,
the stack labels its own call sites ('game' for changes requested by
the game logic during step, 'networkSync' for applied sync data), a
deferred push (assets loading) keeps the cause it was requested under,
and anything unlabeled is recorded as 'external' with its call stack.
consumeLastSceneChangeCause() exposes it. The gameplay test harness
now just consumes this instead of monkey-patching the scene stack
(tracker install/uninstall and classification flags removed).
- Profiler.getFramesAverageMeasures() returns a new FrameMeasureOutput
type: a plain tree without the parent back-references (which made the
output circular and impossible to JSON.stringify). The in-game overlay
percentages now thread the parent time explicitly, and the debugger
payload no longer relies on circular-reference replacement for it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AaQQdPZ68X8zkybtsxWE1b
…text - Profiler.getFramesMaxMeasures(): per-section maximum over the captured frames (the spikes averages hide) and Profiler.getFrameTimes(): the chronological per-frame totals from the ring buffer. - Gameplay test stopProfiling now returns maxStepTimeMs, per-section maxTimeMs, the frame-by-frame timeline (to correlate spikes with eventLog frames), the live object counts, the 3D renderer counters (draw calls, triangles, geometries, textures) when the game uses 3D, and the JS heap in use (Chromium). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AaQQdPZ68X8zkybtsxWE1b
…meline stopProfiling() results are now part of the test result (result.profiles, last 5 kept) so they reach reports without going through console logs (where they were truncated). Each profile reports its window in harness frame numbers (startFrame/endFrame), the most expensive frames (worstFrames, correlatable with the eventLog), and a timeline downsampled past 120 frames by bucket-max (spikes preserved, frameTimesBucketSize). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AaQQdPZ68X8zkybtsxWE1b
…, UI cleanups - Harness: getRuntimeGame() and getCurrentRuntimeScene() low-level escape hatches (documented as last resorts). - Debugger guard during tests is now a whitelist (refresh, getStatus, profiler and gameplayTest commands): fail closed, a command added later cannot accidentally mutate the game state mid-test. - Editor: useForceUpdate instead of a manual counter; the test editor toolbar is a function component; the run-speed menu is shared and the 'Run the test' button of the properties panel is a split button too; getGameplayTestProjectItemName is imported from the runner everywhere (re-export removed); unsavedChangesRef uses useStableUpToDateRef. - Preview/hot-reload guarded everywhere while a gameplay test runs: the _launchPreview funnel early-returns (covers menu bar and palette), the preview commands are disabled, and the split-menu entries too. - Non-persisting gameplay test runs (persist: false, temporary probes) no longer count as modifying the project, so they don't require an approval when auto-edit is off. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AaQQdPZ68X8zkybtsxWE1b
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.