Trim JS-port screenshot suite skip list to mirror port.js#5124
Merged
Conversation
The JavaScript-port screenshot suite is gated by two parallel skip
lists. The Java gate (Cn1ssDeviceRunner.shouldForceTimeoutInHtml5)
runs first and prevents the test from being invoked at all; the
port.js bridge (cn1ssForcedTimeoutTest{Classes,Names}) catches what
slips through. port.js has been carefully maintained as runtime fixes
landed and now skips ~19 tests; the Java list had drifted to ~73 and
was silently gating off the graphics grid, every chart test except
the three known to hang, KotlinUi, MainScreen, transitions, animation
grid, sticky headers, sheets and pickers -- all of which port.js
considers fixed.
The CI workflow already expects this larger set to run (the budget
was bumped to 1740s in scripts-javascript.yml specifically "with the
previously-silent graphics / chart / kotlin / mainscreen / transition
tests now actually rendering and emitting full PNG streams"); they
weren't running because the Java gate fired first.
This change mirrors the Java skip list to port.js's authoritative
set. Two methods:
* isJsSkippedNativeTest -- APIs/bridges the JS port does not
implement (media, vpn, crypto,
accessibility, background-thread,
browser-component-load-event, etc.)
* isJsSkippedKnownRuntimeBug -- tests with active runtime bugs,
each tagged with the same working
name port.js uses (canvasContextWipe,
chatInputEmitHijack,
chartCombinedXyCapture,
sheetTearDownLeak,
simdLargeAllocaCorrupt)
Net: ~54 previously-silenced tests are now exercised against their
existing JS-port goldens. The remaining 19 isolated tests are
clearly categorized and cross-linked to port.js so the two lists
stay in sync going forward.
Safety: the per-test 10s HTML5 cap (TEST_TIMEOUT_MS_HTML5) protects
the suite budget if any newly-enabled test regresses; failures
surface in the comparison report so the offender can be re-parked
with a documented reason.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Collaborator
Author
|
Compared 122 screenshots: 122 matched. Native Android coverage
✅ Native Android screenshot tests passed. Native Android coverage
Benchmark ResultsDetailed Performance Metrics
|
Collaborator
Author
|
Compared 121 screenshots: 121 matched. Benchmark Results
Build and Run Timing
Detailed Performance Metrics
|
Collaborator
Author
|
Compared 58 screenshots: 58 matched. Benchmark Results
Build and Run Timing
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Cn1ssDeviceRunner.shouldForceTimeoutInHtml5skip list down to mirror the authoritative set already maintained inPorts/JavaScriptPort/src/main/webapp/port.js(cn1ssForcedTimeoutTestClasses+cn1ssForcedTimeoutTestNames).scripts/javascript/screenshots/.isJsSkippedNativeTest(APIs the JS port doesn't implement) andisJsSkippedKnownRuntimeBug(each entry cross-linked to the port.js working name:canvasContextWipe,chatInputEmitHijack,chartCombinedXyCapture,sheetTearDownLeak,simdLargeAllocaCorrupt).Why
The JS port screenshot suite has two parallel skip gates: Java fires first (no JS exec) and port.js catches anything that slips through. port.js is the carefully-maintained source of truth — entries are un-parked there as runtime fixes land. The Java list had drifted: it was still gating off ~73 tests, including the whole graphics grid, every chart except
Linefamily, KotlinUi, MainScreen, animations, transitions, sticky headers — all of which port.js considers fixed.The CI workflow already accounts for these tests running (
CN1_JS_TIMEOUT_SECONDS=1800,CN1_JS_BROWSER_LIFETIME_SECONDS=1740, with the comment explicitly mentioning "the previously-silent graphics / chart / kotlin / mainscreen / transition tests now actually rendering and emitting full PNG streams"). They weren't running because the Java gate fired first.Safety
TEST_TIMEOUT_MS_HTML5) bounds the worst case: even if every newly-enabled test hangs, that's 54 × 10s ≈ 9 min within the 1740s suite budget.ProcessScreenshotsonly countsdifferent/erroras failures withCN1SS_FAIL_ON_MISMATCH=1. Tests without a JS-port golden will appear in the PR comment as preview-only and won't break CI.Test plan
scripts-javascriptworkflow runs to completion (suite reachesCN1SS:SUITE:FINISHEDwithin the 1740s budget).equalagainst their existing goldens (ordifferentfor ones whose pixels have drifted — those are real signal to investigate).canvasContextWipe/chartCombinedXyCapture/ etc.) continue to be force-finalised cleanly with no suite-level hang.Display.getInstance().getPlatformName()is"HTML5").🤖 Generated with Claude Code