Skip to content

Make the EditMode suite pass headlessly - #41

Merged
vbucc merged 2 commits into
mainfrom
fix/editmode-test-failures
Aug 31, 2026
Merged

vbucc merged 2 commits into
mainfrom
fix/editmode-test-failures

Conversation

@vbucc

@vbucc vbucc commented Aug 31, 2026

Copy link
Copy Markdown
Collaborator

Three EditMode tests failed on every headless run and were waived by hand at merge time (most recently #38). They are two unrelated problems, and neither is a production bug — both are tests asserting on ambient editor state that headless Unity does not provide.

Correction to #38: its body says Unity's native runner passes all three. It does not. It passes only EditorStateCache_IsChanging. The two ReadConsoleTests failures reproduce under the plain -batchmode -nographics -runTests runner with no MCP involvement at all — before this change, the native full suite was 1039 tests, 972 passed, 2 failed, exactly those two.

EditorStateCache_IsChanging_TracksTransitionWindowNotWholeSession

The test drove _isPlayModeChanging by reflection but left the other input to ComputeActivityPhase at whatever the ambient run had set. "running_tests" outranks "playmode_transition", and TestRunStatus.IsRunning is true whenever the suite was started through the MCP run_tests tool — which is exactly what the local harness does. So it passed natively and failed under the harness.

Now drives TestRunStatus._isRunning in the same try/finally instead of assuming a quiescent editor. Safe to poke: a [Test] body runs synchronously inside one editor frame and TestRunStatus is only read from main-thread callbacks.

ReadConsoleTests

Root cause, established against a live headless editor:

  • read_console reports 0 entries at rest.
  • Provoking a McpLog.Warning from an ordinary command handler → 1 entry, and it persists. So LogEntries works headlessly.
  • Running HandleCommand_Get_Works alone and then reading the console filtered on "Test Log Message" → 0 matches, while that exact string appears both in Editor.log and in UTF's captured per-test output.

Unity does not record a log emitted inside a Test Framework test into the console buffer under -batchmode -nographics. These tests only ever passed in a GUI editor.

Split in two:

  • SplitMessageAndStackTrace becomes internal and gets five direct tests (blank line inside the body preserved, CRLF normalization, stack frames split off, no-stack cases). That is what PreservesMultilineMessageBody was really about, and it runs identically in the GUI and headless. One visibility change is the whole production diff.
  • The live-console tests seed an entry and Assert.Ignore when it is not observable — full strength in the Test Runner window, an explicit documented skip headlessly. They now filter server-side rather than scanning a count = 1000 window: GetConsoleEntries walks oldest-first and stops at count, so a busy console can push a freshly logged message out of an unfiltered window entirely.
  • HandleCommand_Clear_Works drops its "console should have logs" precondition, which was satisfied only by unrelated tests' logs — the class failed the moment it was run alone. It now asserts what it owns: clear succeeds, console reads empty.

Harness

_start_utf now sends discardUntitledScenes. The smoke leg dirties the untitled scene (it creates and deletes GameObjects) and run_tests fail-fasts on that, because an interactive editor would wedge on a native Save dialog. A headless gate has none. This did not fire during #38's gate but fired immediately when probed, and would have surfaced as "run_tests start failed".

Gate

  • Python: 1502 passed, 3 skipped.
  • Harness unit tests: 140 passed.
  • Native full EditMode suite: 1044 total, 977 passed, 0 failed, 49 skipped (was 1039 / 972 / 2 / 47).
  • Harness --legs editmode: exit 0, 977/1026 passed, 0 failures.
  • Harness --legs smoke,editmode: exit 0, smoke 7/7.

The gate is now green with no documented exception.

Not verified: the GUI Test Runner path. The two live-console tests should run there rather than skip; if they skip in the GUI too, the observability check needs revisiting.

🤖 Generated with Claude Code

vbucc and others added 2 commits August 31, 2026 19:05
Three EditMode tests failed on every headless run and were waived by hand at
merge time. They are two unrelated problems, neither a production bug.

EditorStateCache_IsChanging asserted activity.phase == "playmode_transition"
while leaving TestRunStatus.IsRunning at whatever the ambient run set it to.
"running_tests" outranks "playmode_transition", and IsRunning is true whenever
the suite was started through the MCP run_tests tool — which is exactly what
the local harness does. Drive that input by reflection alongside
_isPlayModeChanging instead of assuming a quiescent editor.

The two ReadConsoleTests failures were never MCP-specific: under
-batchmode -nographics Unity does not record a log emitted inside a test into
the console's LogEntries buffer, so they fail under the plain native runner
too. Logs from ordinary editor code do land, so the buffer itself works.

Split those tests in two. SplitMessageAndStackTrace becomes internal and gets
five direct tests — that is what PreservesMultilineMessageBody was really
about, and it runs identically everywhere. The live-console tests seed an
entry and Assert.Ignore when it is not observable, keeping full strength in
the Test Runner window without failing the headless gate. They now filter
server-side rather than scanning a count=1000 window: GetConsoleEntries walks
oldest-first and stops at count, so a busy console can push a fresh message
out of an unfiltered window.

Clear_Works dropped its "console should have logs" precondition, which was
satisfied only by unrelated tests' logs — the class failed when run alone.

Also opt the harness into discardUntitledScenes: the smoke leg dirties the
untitled scene and run_tests fail-fasts on that, because an interactive editor
would wedge on a native Save dialog. A headless gate has none.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@vbucc
vbucc merged commit b3ece1b into main Aug 31, 2026
@vbucc
vbucc deleted the fix/editmode-test-failures branch August 31, 2026 23:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant