Run tests client-side on a wasm kernel (behind an experimental flag) - #9462
Open
iHiD wants to merge 5 commits into
Open
Run tests client-side on a wasm kernel (behind an experimental flag)#9462iHiD wants to merge 5 commits into
iHiD wants to merge 5 commits into
Conversation
SharedArrayBuffer is only available to a cross-origin isolated document, which means the editor needs to send COOP and COEP. We use `Cross-Origin-Embedder-Policy: credentialless` rather than `require-corp`. Both grant isolation, but under `require-corp` every cross-origin subresource without Cross-Origin-Resource-Policy is blocked outright - which would include the stylesheets we pull from the assets host and any image an exercise author puts in their markdown. `credentialless` fetches those without credentials instead. Cross-origin iframes are strict under both values though, so the Vimeo embed (the one iframe reachable from the editor - the hello-world tutorial video) gets the `credentialless` attribute. Nothing else can get an iframe in there: Markdown::ParseDoc enables `tagfilter`, which escapes raw <iframe> tags. The subtle part is Turbo. Isolation is a property of the document, and every in-app navigation on the site renders into the persistent tf-main frame, so the editor would inherit whatever isolation the *previous* page was loaded with - i.e. none, silently. So the editor now renders the full layout even for frame requests, and carries `<meta name="turbo-visit-control" content="reload">`, which makes Turbo hand over to a real browser navigation. The direct links into the editor also get `data-turbo="false"` so the common paths skip the wasted round trip. This needs Cross-Origin-Resource-Policy on assets.exercism.org to land first (separate terraform change) so we keep the option of tightening to `require-corp` later. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VGsPyM4G17uXGDLmmF8caF
Generalises the client-side test runner beyond JavaScript. The dispatch in generalTestRunner.ts becomes `runTests(language, ...)`, and which languages can run in the browser - and what to load for them - is answered by a manifest fetched at runtime rather than by a switch here. For everything but JavaScript this runs the track's own `bin/run.sh` inside a wasm Linux kernel, so a browser test run executes the same script, against the same tools, as a run on the tooling fleet. The manifest points at immutable, content-addressed artifacts, so publishing or rolling back a language's runner is an edit to one JSON file with no deploy here. A student's solution is untrusted code that can loop forever, so each run gets its own kernel and that kernel is destroyed afterwards - terminating the worker rather than asking a possibly-wedged kernel to tidy up after itself, and leaving nothing in /solution for the next run to see. Booting is expensive enough that doing it on the critical path would defeat the point, so one spare is kept warm: started when the editor mounts, and replaced as soon as one is taken. Everything degrades to null, which the caller already treats as "run this on the server". Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015v94EHpgxqrdVPdkt7KaWk
Cloudflare routes these to the assets distribution in production, so they never reach the app there. Nothing serves them in dev or test though, and the editor asks for a manifest on every page load, so Rails raised a routing error - which Capybara turns into a failed system test on every test that opens the editor. 404 is the honest answer, and the one the runner already reads as "this track has no client-side runner" before falling back to the server. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015v94EHpgxqrdVPdkt7KaWk
Rails now serves /test-runners/* out of S3. It has to be our own origin: the editor boots the wasm kernel in a worker, `new Worker()` refuses a cross-origin script URL, and exercism.org resolves to the ALB, so the path cannot be rerouted at the edge. Cloudflare caches what we serve, so S3 is read about once per edge location per release. Three things that are easy to get wrong and fail obscurely, so they are all explicit: content types (instantiateStreaming rejects anything that is not application/wasm), the forgery exemption (Rails 422s any non-XHR GET that returns JavaScript, whatever the origin), and COEP on the response (a worker script must declare a policy compatible with its owner's, even same-origin). The editor gains an `experimental` flag, false everywhere but a new maintainers' page that renders the editor for jq. Students' editors do not fetch a manifest, do not prefetch a kernel, and behave exactly as before, so this can ship without touching what they use. The exercise's own files - the test file and .meta/config.json, which run.sh reads to find it - are sent only when that flag is set. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015v94EHpgxqrdVPdkt7KaWk
credentialless is not supported in Safari at all, so an isolated page using it would simply not work there. require-corp works everywhere, at the cost of being strict: a cross-origin subresource without Cross-Origin-Resource-Policy is blocked rather than fetched without credentials. That strictness is why isolation now applies only to the maintainers' experimental editor and not to the editor students use. The assets host sends CORP, but a third-party script or an image in someone's markdown will not, and would fail on a page students actually use. The experimental page can wear it. The Turbo machinery on the students' editor stays as it is, ready for when isolation moves back there. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015v94EHpgxqrdVPdkt7KaWk
iHiD
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Runs an exercise's tests in the browser instead of on the tooling fleet, for tracks that have a client-side runner published. Nothing students use changes: it is all behind an
experimentalflag that is false everywhere except a new maintainers-only page.How it works
For everything but JavaScript, this boots a wasm Linux kernel in a Web Worker and runs the track's own
bin/run.shinside it — the same script, the samebats, the samejqthat a server-side run uses. So a browser run and a fleet run execute identical code, and the result is the same v3results.json, which goes to the server exactly as a server-produced one would.Which languages can do this, and what to load for them, is answered at runtime by a manifest rather than by anything in this repo. Publishing, updating or withdrawing a language is an S3 operation, not a deploy.
Serving the artifacts
The artifacts have to come from our own origin:
new Worker()refuses a cross-origin script URL and no header lifts that.exercism.orgresolves to the ALB rather than CloudFront, and rerouting a path at the edge needs Cloudflare Origin Rules, whose Host Header and Origin Host overrides are both Enterprise-only. So Rails serves them, reading from a newexercism-test-runnersbucket (terraform). Cloudflare caches the result — including for logged-in users — so S3 is read roughly once per edge location per release.Three things there fail obscurely if you get them wrong, so they are explicit and commented:
WebAssembly.instantiateStreamingrejects anything that is notapplication/wasm; a module worker needs a JavaScript type. S3's own metadata is not trusted.import()is not an XHR. Same exemption the JS worker endpoint already needed."error".Isolation:
require-corp, experimental page onlySharedArrayBufferneeds cross-origin isolation. This usesrequire-corp, notcredentialless— Safari does not supportcredentiallessat all.require-corpis strict: a cross-origin subresource without CORP is blocked outright rather than fetched without credentials. Our assets host sends CORP, but a third-party script or an image in a user's markdown will not. So isolation applies only to the experimental page, not to the editor students use. The Turbo machinery on the students' editor is left in place, ready for when isolation moves back there.One kernel per run
A student's solution is untrusted code. It can loop forever, and whatever it leaves in
/solutionmust not be visible to the next run. So each run gets its own kernel, destroyed afterwards —worker.terminate()rather than asking a kernel that may be wedged to tidy up after itself.Booting is expensive enough (a multi-megabyte sysroot to unpack, then the runner tarball) that doing it on the critical path would defeat the point, so one spare is kept warm: prepared when the editor mounts, replaced as soon as one is taken.
The experimental page
/maintaining/experimental_editorslists jq exercises; each renders the normal editor with the flag on. Underensure_maintainer!, and it joins the track for you, sinceSolution::Createrefuses to work on a track you have not joined.Everything degrades to
null, which the caller already treats as "run this on the server" — a missing manifest, a failed boot, a broken artifact, an unisolated page. There is no path where this can stop a student running their tests.Depends on
exercism-test-runnersbucket, IAM, and Cloudflare cache/transform rulesexercism-config0.138.0 foraws_test_runners_bucket🤖 Generated with Claude Code
https://claude.ai/code/session_015v94EHpgxqrdVPdkt7KaWk