Conversation
Add direct keyboard, pointer and touch controls, display presets and favorites, toolbar menus, sharp captures, native desktop hover, and stable frame buffering. Preserve page state during mode changes and reduce redundant frame transfers. Include regression tests and fork reconciliation notes.
… review Apply pinned formatting, type the viewport fixture, align the smoke with default capture quality, and document isolated validation and the supported SDK.
…contracts Add encoded packet limits, independent density and encoder targets, and versioned display preferences. Keep additions compatible with existing image clients and separate continuous human admission from strict agent frame targeting. Validate the contract and settings regressions and the complete TypeScript build.
Capture the exact workspace tab through a private immutable extension and isolate encoder cohorts. Bound encoded buffers, retire idle media, and recover source failures without changing geometry or resetting healthy peer encoders. Cover packet backlog, byte conversion, failure isolation, cleanup, codec allocation and source timestamps with native helper regressions.
… control Fence media and input across native attachment, document, bridge and viewport transitions. Admit continuous human geometry while retaining strict agent receipts, reserve control and heartbeat capacity, and reconcile navigation without blocking steady video reads. Add redacted operation diagnostics, demand-driven JPEG capture, bounded recovery and transition regressions. Register shared contract tests in the standard unit gate. TypeScript and all 475 checkpoint tests pass.
Decode bounded native packets into a retained canvas, keep input continuous during paints, and hand off to qualified image fallback without flashing stale frames. Add independent encoder and capture-density controls. Fence delayed mutations across viewer and controller replacement and split panel styles, chrome, native keyboard and presence into focused modules. Include decoder, mounted settlement and real native capture-to-canvas regressions.
…servation Do not invalidate a pending acquisition when media first observes the same viewer as controller. Competing ownership and viewer replacement still revoke delayed replies. Reproduce the race in the full React panel and verify the returned token reaches the next navigation.
…n input Bind in-flight agent work to the original ticket and native page. Validate agent requests with the shared schemas and preserve the original failure while attempting paired cleanup once.
Format the embedded extension program and document frame units, peer ownership, codec recovery and teardown contracts without changing its capture protocol.
…d SDK Require Paseo 0.11 with the tested beta allowance. Ship design and research references, document isolated validation and quality trade-offs, and add real SDK compilation and Linux video decoding to CI.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add genuine encoded video playback for desktop/web and Android Shared Browser clients, building on the separate interaction and display PR. Android requires a Paseo build exposing the host encoded-video canvas. Other native clients and clients without usable video decoding retain JPEG fallback.
Changes
Transport and trade-offs
Encoded packets use authenticated Paseo RPC. This adds no public media listener or TURN requirement; it is not WebRTC and still shares Paseo's reliable connection and relay.
A human click can reach the page ahead of its displayed image. Exact source/decoded geometry and ownership barriers remain enforced. Video is lossy, and higher density/bitrate increase CPU and bandwidth use. There is no lossless idle overlay.
Validation
The reviewed combined tree passes 545 regression tests, real Chromium capture-to-production-decoder smoke, existing image smoke, mounted client checks, TypeScript, pinned Biome, SDK compilation and mobile Hermes compilation.
Linux native video is exercised locally and added to CI. Windows/macOS video and latency across arbitrary LAN/relay configurations are not qualified by these checks. After rebasing onto current upstream main and the prepared interaction branch, the 545-test suite, native video smoke, TypeScript, pinned Biome and SDK compilation were rerun successfully.
Latest timing follow-up: 40 focused buffer/policy regressions and 2 authenticated socket regressions passed, along with TypeScript and Biome. An isolated real Chromium touch-navigation probe recovered fresh video after navigation. On a physical Pixel 9 Pro using a locally built APK, the Example Domain link opened IANA while playback remained on Video; leaving the browser tab retired its recorder, returning resumed Video, and background/foreground recovery passed. These checks used a separate test workspace rather than PrintStream. No emulator was used.
The live failure was confirmed before the change: a packet with roughly 34 ms native age was rejected as roughly 74 seconds in the future after Node wall time diverged from its stored calibration. Private monotonic receipt timing now protects both playback freshness and post-input authority without changing the public packet contract.
Latest input follow-up: all 37 focused input-queue regressions passed, along with TypeScript and Biome. The tests cover release before admission acknowledgement, release during stale-frame retry, actual control replacement, and unknown admission outcomes. Two short physical-phone link taps opened IANA while retaining Video after the queue fix was loaded. These observed passes do not establish that every earlier ignored tap had the same cause.
Dependency
Depends on #274. This PR is intentionally a draft against
mainuntil the interaction PR merges. GitHub currently includes the prerequisite changes in its diff; review the additional commits after9915531for the streaming work. Rebase after the dependency merges so only this PR's changes remain.AI disclosure: this comment and the related code were written with the assistance of AI.