Native frame generation (optical-flow interpolation) - #537
Open
maxjivi05 wants to merge 76 commits into
Open
Conversation
Compositor frame generation on open GLSL shaders: block-matching motion estimation and fragment-shader interpolation wired into the Vulkan present path. A headroom-driven scheduler posts at the target rate under a non-blocking present mode (so an adaptive panel ramps up) and passes through under FIFO; the real frame always presents, so it never drops below native. Frame Gen controls live in the FX tab as an expanding toggle (like SGSR): 2x/3x/4x multiplier, quality preset and smoothness; Other-settings toggle. The HUD FPS reports the output rate (real + generated) while FG is active.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 83c2550805
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…s idle Under FG only real game presents set fgNewScene, so cursor and window changes (paused, idle menus, static scenes) never reached a HOLD and the compositor stayed frozen on the last frame. Mark a scene-dirty flag on non-game render requests; the pump recomposites and presents it without interpolating.
Replace the full per-submit GPU drain (wait_inflight_frames) in fg_submit with a targeted wait on only the history slot a HOLD is about to overwrite. Grow the history ring 2->3 slots so the overwritten slot is never the pair an in-flight INTERP is sampling, and track a fence per slot. Add a fragment->compute WAR barrier so a new pair's motion recompute waits for the prior pair's interp reads on the single graphics queue. This lets CPU record/submit overlap GPU execution, cutting frametime variance under the Balanced/Quality presets.
…droom Refine the block-matching result with a per-axis parabolic SSD fit so motion vectors are sub-pixel instead of quantized to 2 full-res px (removes slow-pan wobble); guarded to keep the +/-1 taps inside the search tile. Apply a separable 3x3 component-wise median to the flow field in interpolate.frag to reject outlier vectors without blurring motion edges. Raise the non-FIFO swapchain image floor 3->4 so interpolated frames stop being dropped at acquire, and count those drops in the FG cadence log.
…acer Hold the adaptive (LTPO) panel high by over-posting under MAILBOX, and pace presents evenly with a clock_nanosleep(CLOCK_MONOTONIC, TIMER_ABSTIME) deadline loop anchored to the Choreographer vsync grid -- the smoothness comes from the absolute-time sleep, not the present mode or VK_GOOGLE_display_timing (Android ignores desiredPresentTime under MAILBOX). The display-timing extension is kept only for read-back telemetry (FG timing: avgInterval log). The FPS limiter pins the engine rate; the surface frame-rate vote targets engine*multiplier with a never-below-native floor; the FIFO cadence branch gains an epsilon so an exact ratio doesn't drop a multiplier.
Wake ~120us before the deadline so the present latches the current vblank; compute the interp phase from real-frame arrival times vs the present deadline instead of a fixed k/(n+1) fraction; replace the 4-sample avgInterval log with windowed CoV/min/max present-interval stats.
…rder-independent mode resolve
…ings - FIFO presents with deterministic slot phases; real frames present sharp - Bidirectional warp at every multiplier; median occlusion fallback; static HUD mask - Extrapolation present path (no added latency) - Runtime frames-in-flight (Buffering 1-3) - Per-game FG settings; Smoothest / Low Latency presets in the drawer
Interpolated frame generation on a dedicated worker thread with a deferred-promote history ring and a changed-pixel content-dedup, so distinct frames are kept and the source rate is measured cleanly. Slot-grid cadence with even-hold pacing for high-refresh panels, native-max refresh pinning, and deadline-paced presents. Occlusion-gated interpolation with motion-faded detail. Fixes: enable/disable crash (stale worker fence left in the shared slot-fence array), drawer-overlay pause/resume, and low-motion content-rate collapse. Adds compute shaders for the flow-generation path.
Brings in the 7 upstream commits (Fx tab effects, frontend support, controller auto-hide touchscreen controls, wrapper update, DXVK/D7VK ddraw, settings strings) on top of the frame-generation work. Conflicts in vk_renderer.c, vk_state.h, and VulkanRenderer.java resolved by keeping both sides.
Brings in the 5 upstream commits (Z drive boot & shortcut fix, touch controls refinement, Boot-to-Desktop/Graphics-test hero buttons, Box64 env vars, README) on top of the frame-generation work. Conflicts in XServerDisplayActivity.java, XServerDrawerMenu.kt, and 14 locale strings.xml resolved by keeping both sides.
Pace each presented frame at its true temporal position instead of quantizing to vblanks: - fg_compute_deadline no longer snaps the present target to the panel vsync grid; the target is the free, evenly-spaced instant on the measured content-rate grid (vsync-snapping was the 3:2-pulldown-style judder source for non-integer/variable content:panel ratios). - The worker requests that instant from the display via VkPresentTimeGOOGLE.desiredPresentTime (was hardcoded 0), so a panel that honours display-timing latches each frame on the correct vblank. - FG presents under a non-blocking mode on the toggle path (MAILBOX, IMMEDIATE fallback) to match the attach path, so the deadline nanosleep drives the present instant instead of FIFO vsync-blocking. - One present per pump tick (fgComputePerTick=1): the pump fires once per vblank, so emitting more posted multiple frames into a single vblank, which showed as an uneven ">panel" present rate.
Replace the open-loop slot-grid/tick-counter cadence with a time-based rule and lock pacing to the source clock: - fgEmitOne derives the output sub-frame from elapsed time since the last real frame (frac*M), not a tick counter, so irregular promote arrival can't misalign placement. One present per vblank; sub 0 = real frame, sub 1..M-1 = tweens at phase sub/M. - Divisor-snap the cadence multiplier to the largest divisor of the panel:content ratio so output divides the panel evenly (e.g. 3x of 30Hz = 90Hz holds frames 1,1,2 vblanks on a 120Hz panel = judder; snaps to 2x). fgTargetHz + the adaptive ratchet use the snapped multiplier; diag reports it as cad=. - Stabilise the rate lock with a light EMA toward the measured content rate instead of the drift-relock threshold that left the lock stale. - Native fg_compute_deadline anchors each present to curr_arrival + phase*(curr-prev) (the source clock) instead of a per-enqueue period accumulator that drifted when holds emit nothing; nativePresentLast now carries phase + arrivals.
The floor(frac*M) sub-frame rule skipped tweens when the source rate jittered against the EMA period, so the output ran far below target (interp ~16/s vs 30 expected at 2x, ~80 missing at 4x) and the adaptive ratchet misread that as a GPU limit and dropped the multiplier (4x ran as 2x) even though generation is cheap (over-budget ~7/240, interp ~0.07ms). Replace it with a deterministic steady gate: present a new frame every hold=slots/M vblanks and sample the tween phase continuously from the content clock, so a fresh frame is produced on every gate vblank. 4x now holds (eff=4x cad=4x) and the output tracks the target (~110fps from a 30fps source) instead of collapsing to ~48fps. Real frame shown sharp on promote; gate restarts there.
- fg_sig_delta noise floor lowered 4->2 so subtly-moving frames (3-4/channel change) count as distinct instead of being held as duplicates. Exact re-presents still score 0 and drop, so the rate measurement is unaffected. - Wire the Max preset (index 5) to enable deep (bidirectional) flow in both the startup load and the in-game preset handler; it was hardcoded off for all presets. Other presets stay single-flow.
The cadence anchored to the native content-promote time, which is quantized to the pump vblank. Anchor it instead to the precise onFramePresented arrival timestamp (fgLastGameNs) - the game's own buffer-swap/present clock. Content-dedup still runs upstream to drop redundant re-presents (the compositor sees some), but the cadence timing now comes from the real arrival instants. Measured on a 30fps source at 4x: present-interval cov ~37% -> ~22%, worst-case gap 24.9ms -> 16.6ms (no more 3-vblank holds).
Brings in the upstream commits synced on the remote (Fx effects per shortcut, glass control opacity, drive-creation fix) on top of the frame-generation pacing work. Clean auto-merge.
Built GT validation harness (fgtest cnn_flow_run + wnfg_53): the real GT chain produces NON-DEGENERATE logits at all 7 levels (out ch ranges 1.5-5.2, maxsep 1.9-4.1) with b32=hD8(D)/b33=hD7(C) pair + b34=seed. PROVES the wnfg_53 kernel + wiring + chain structure are correct. Renderer wired to match. Renderer logits still degenerate ONLY on the sparse D3D11 test scene (mostly-black low-variance features); delta5-8 wiring is identical to the validated harness, so this is a feature-sparsity artifact of the synthetic test content, not a port bug — needs a real game (dense features, what wnfg_53 was trained on) to confirm 1:1.
MHS (and games presenting CPU-accessible linear swapchains, usage&0xFF!=0, allocationSize==w*h*4) showed vertical blue/cyan stripe corruption on the new Adreno 840 phone (CPH2749) but not Adreno 830. Root cause: vkr_texture_import_ahb hardcoded VK_IMAGE_TILING_OPTIMAL; on Adreno 840 OPTIMAL is a real tile swizzle, so linear buffer data was read through a tiling pattern. Fix: import CPU-accessible (linear) AHBs as VK_IMAGE_TILING_LINEAR, keep OPTIMAL for GPU-only buffers. NOT yet on-device-verified (phone disconnected mid-test). FG kept on two-flow (wnfg_53 logits preserved in history at 37518545).
RE (workflow): MHS vertical-stripe corruption = the game writes its swapchain AHB via the guest WN-Turnip in Adreno-840 macro-tile order, but winnative's compositor read it via the system Qualcomm driver -> tile-order mismatch (coincided on Adreno 830). The compositor driver came from graphicsDriverConfig 'version' and fell back to 'System' when unset, ignoring the game's actual driver. Fix: when no explicit compositor version is set, match the GAME's driver (graphicsDriver) so writer==reader, per the existing 'match guest libvulkan' intent. Also revert the inert CPU-access LINEAR import toggle (dedicated AHB layout comes from gralloc metadata, not ic.tiling) and add an OPTIMAL->LINEAR vkCreateImage fallback so no driver can black-screen. Diagnostic: XServerDisplayActivity logs Compositor graphics driver='...'. UNVERIFIED on-device (phone disconnected).
Reverts the unconditional guest-driver-matching for the compositor (a6acd95c) for FG-off games; on Adreno 840 reading the AHB via guest Turnip OR System both stripe, so this isn't the full fix but restores the pre-FG default. Regression hunt continues.
User-confirmed: the libvulkan_wrapper BCn emulation (auto/full) patches the Adreno driver and corrupts the whole frame as vertical stripes on Adreno 840; setting bcnEmulation=none fixes it. Mesa Turnip decodes BC natively, so force none on Adreno automatically. Also revert the wrong-diagnosis COLOR_ATTACHMENT import change.
…hen static) On a real game (MHS) camera spin, interp dropped 30->0/s: X11 damage events fire requestRenderCoalesced() -> fgSceneDirty every vblank during motion, and fgEmitOne's 'dirty && !newGame' branch presented sharp instead of running the interp cadence. Fix: defer the UI-only recomposite to the END and only take it when the content interval is fully spanned (static); mid-interval, run the interpolation cadence so motion is actually smoothed. Isolated via dumps/20260620_2047_mhs_spin_BEFORE_interp0.
…+ frame-sequence dump
… harness Warp: signed mvScale so the midpoint gather uses -0.5*flow. The old +0.25 was wrong-sign and caused ghosting/shake at intermediate phases (endpoints were unaffected, which is why raising m0 made it worse). Debug-viz moved to a flags bit; default m0 -0.25. Pacing: snap the interp present deadline to the even vblank grid; complete the motion to curr on a late-frame hold. Harness (debug, default-off): debug.winnative.fgsynth rigid-shift / fgpat noise field + full-res crop + flow-field dump for ground-truth warp verification.
conv36 was dispatched cinT=4 but the trained wnfg_36 is cinT=2 (8ch = prev4 + curr4), so it read past its weight file -> bias-only, content-independent features that froze the whole flow chain. Fixed: cnn_concat2 feeds prev.L0 ++ curr.L0 at cinT=2, plus the missing trained wnfg_45 (8->16) expansion before conv42. Verified on-device the gamma now produces content-varying features. The flow magnitude is still a fixed ~19 regardless of motion - the producer never tracked motion. Root-caused to the cost volume (wnfg_14) self-correlating combined prev+curr features plus a dominant flow-regression bias, with the coarsest pyramid level sub-pixel for the test shifts. Feeding the cost stage separated per-frame features is correct but insufficient alone; the full fix is a multi-stage RE, documented separately. Flow images get TRANSFER_SRC so the controlled-motion harness can dump them.
Resolves the sync (was 57 ahead / 13 behind upstream). Conflicts: - vk_renderer.c: kept the FG-integrated renderer (manage_scene_targets helper + record_and_submit_frame which already carries the scene-snapshot-under-mutex + graveyard improvement); merged upstream's no-surface present-mode guard with the FG worker stop/restart. - Steam DB converters (AppConverter, UserFileInfoListConverter): took upstream's schema-drift tolerance fixes (no FG changes there). - strings.xml: kept both the FG strings and upstream's file-manager strings.
Collaborator
|
Going to need to ensure that any change in the xserver menu has the new PANE_NAV |
Resolve conflicts across the drawer split (main moved the panes into per-pane files), the settings screen's LazyColumn->Column change, VulkanRenderer's attachSurface locking, the Vulkan dispatch table, vk_state.h and vk_renderer.c.
The scheduler presented the newest real frame the moment content arrived and then presented the interpolations of that same interval behind it, so content time ran 1.0, 1/M, 2/M ... 1.0 and jumped backward by (M-1)/M of a frame every content interval. Interpolation now holds the real frame back to the last slot of its interval and emits the generated frames ahead of it, which is the only ordering that keeps content time monotonic; extrapolation keeps the real frame first because its generated frames lie past it. Phase came from the raw vblank index while the emit gate fired on the emit index, so whenever the vblank count was not a multiple of the emit count the generated frames were spaced unevenly in content time against their on-screen durations. Phase is now anchored to the vblank a frame actually occupies, which holds velocity constant for every multiplier/refresh pair, including the 3x cases where 3 emits never divide evenly into 4 vblanks. Slot count is also derived from the measured vblanks per content interval instead of falling back to the multiplier when content matches the panel rate. Present deadlines were derived from the phase, which no longer corresponds to a slot; the scheduler now passes the vblank it targets and native paces to it. cnn_generate gathered with a hand-tuned scalar that conflated the flow-field resolution ratio, the warp sign and the blend split. The backward pair gathered with an inverted sign and both pairs at roughly a quarter of the true displacement, which collapses the two gathers onto the output pixel and reads as ghosting and doubling; the error also scaled with the flow-scale preset. It now derives the flow-to-pixel scale from the texture sizes and applies the sign each field actually carries, and the forward slot is fed the forward field instead of a second copy of the backward one. interpolate.frag bound a forward motion field in deep mode and never sampled it, so Quality cost an extra motion pass and changed nothing. Deep mode now warps the prev side along the forward field and gates the blend on true forward/backward consistency. Preset selection is the single source of truth for quality/model/flow-scale/deep instead of three copies of the tables that could disagree with the stored quality.
…le, present before recording interpolate.frag took the Smoothness value as a push constant and never read it, so the setting did nothing; it now scales the occlusion tolerance window, with the shipped default mapping to the current behaviour. The pane had no control for it either, so the slider is added next to the multiplier and preset. The shader derived its unsharp-mask texel size from the flow-field resolution, so the kernel widened to 2.5 full-res texels on the low flow-scale presets and the apparent sharpening changed with the quality preset. Full-res and flow-field sizes are now read separately from the samplers. The generation worker recorded the next frame before presenting the pending one, so every present was pushed out by a varying record time and could straddle a vblank boundary. It now presents first; the frame being presented has already had a full vblank of GPU time, and the next record still overlaps the following interval.
fgEmitOne ran once per onDrawFrame and treated every call as one vblank, but the surface view also draws on its transient path, which cursor motion arms at up to 120Hz on top of the FG pump. Each extra draw advanced the slot counter, so the cadence walked ahead of the vblank grid and the interpolation phases ran ahead of wall time for as long as the pointer moved. The counter now advances only when the pump reports a new vsync, and an extra draw within the same vsync can still stage content and refresh a static frame but cannot consume a slot. With FG on, cursor motion no longer arms a transient render at all: the pump is already drawing every vblank.
…vblank, keep the pointer out of the interval PRESENT_LAST resolved its image from the live history instead of the frame the job captured. The scheduler enqueues the real frame at the last slot of an interval and the worker presents it one iteration later, by which time the next content frame has usually promoted, so the real slot showed frame N+1 and the following slot then interpolated N->N+1 from its first phase. Content time ran forward a frame and a half then back most of a frame, every interval: at 4x a quarter of all presented frames moved backward by three quarters of a content frame, at 3x a third of them by two thirds. A job now presents what it captured unless that pair has been recycled. The staged frame was only resolved by the next HOLD, so a frame could not be promoted until the frame after it arrived - a full content frame of latency before anything could be shown or interpolated. Resolution is now a non-blocking fence poll run every vblank, which brings it down to roughly one vblank. Pointer motion was entangled with content. A cursor move staged a frame that the 64x36 dedup signature often could not tell apart, so it was dropped and the pointer sat still until the 100ms backstop; when it was not dropped it counted as a promote, which reset the slot counter mid-interval and collapsed prev/curr onto two cursor variants of one frame. UI recomposites are now force-promoted so they never get dropped, are only allowed once content has genuinely stopped, and no longer re-anchor the cadence - while content is live the pointer rides along with each content composite. vkGetFenceStatus was missing from the dispatch table.
…phase-dependent shading Default to the classical block-matching flow. The CNN flow and generate path ran by default, and its warp geometry rests on a hand-tuned scalar that folds together the flow-to-pixel ratio, the warp sign and the blend split; nothing about its output units or sign is derivable from the code, and the gather it produces cannot be checked. The motion.comp / interpolate.frag pair has a convention that can be derived end to end and checked against ground truth, so that is what runs now. debug.winnative.fgcnn=1 brings the CNN back for whoever continues that work. Every generated frame in an interval recomputed the flow into the same image while an earlier submit could still be sampling it, with nothing ordering the two. At 2x the parity rotates each interval so it rarely bit; at 3x and 4x every frame after the first in an interval raced the one before it, which is why it got worse with the multiplier. The flow is now computed once per interval and barriered for the rest, as the CNN branch already did. Interpolated frames were dropped whenever a swapchain image was not free that instant, because the acquire used a zero timeout. A dropped frame leaves a hole in the cadence. It now waits a couple of vblanks. The worker presented the pending frame only after being woken by the next job, so the present instant tracked whenever the scheduler happened to enqueue - and on a content vblank that sits behind a whole scene composite. Presents were late by a varying amount once per content frame. It now paces to the deadline before waiting, so the present instant depends on nothing but the deadline. Slot count came from a rounded EMA with no hysteresis, so a content rate near a half integer multiple of the vblank rate flipped it constantly - 48fps on 120Hz sits at 2.5 and reshuffled the cadence 96 times in 600 frames under 3% jitter. Now 0. The slot index counted draws, but the surface view coalesces render requests, so a missed draw left the cadence a slot behind for the rest of the interval. It is derived from the vsync clock now. In the shader: the detail gain was scaled by the curr-side blend weight, so generated frames sharpened across the interval and snapped back at the real frame, once per content frame; the detail term now follows the same blend as the colour and the gain is constant. The low-confidence fallback hard-switched from prev to curr at t=0.5, stepping part of the image a whole frame mid-interval; it crossfades. And near-black pixels were treated as invalid samples, collapsing the blend weights in dark content - the bounds check alone decides validity now.
… made quality worse The flow field was sized as a fraction of the screen and the presets pushed that fraction up: Balanced ran it at 0.6 of screen and Max at 0.8, so on a 1440p panel Max searched a 2048x1152 field. The block search loads a 42x42 descriptor tile per 8x8 output block, and each descriptor ran a per-channel pow for gamma. At 60fps content that is 158 billion transcendental ops and 294 billion shared-memory reads per second for Max, against 4.9 and 4.8 billion for Eco - which is why Eco was tolerable and everything above it collapsed, and why the multiplier made no difference: the flow is computed once per content frame, not per generated frame. The presets were also backwards for quality. Search radius is fixed in flow-field texels, so a higher-resolution field covers less actual motion: Max could only track +-38 px of movement per frame at 1440p while Eco tracked +-150. Anything faster saturated the search and produced wrong vectors, which is the artifacting. The expensive presets were the ones that tracked motion worst. Flow resolution is now capped at 480 px on the long edge and the presets scale underneath it (0.08 to 0.20). Max lands at 480x270: 1.45 G transcendental ops and 16 G shared reads per second, ~109x and ~18x lower, and cheaper than the old Eco while tracking +-160 px instead of +-38. Every preset now tracks more motion than the old Eco did. luma() ran exp2(log2(c) * 1/2.2) per channel - six transcendentals per texel fetch, on a descriptor tile that is fetched with 27x redundancy. It dots to luma first and takes a sqrt: one transcendental, and gamma on the scalar is the same thing for a matching descriptor. The interpolation shader is back to one 4-tap blur for the detail term rather than one per side; the phase-dependent gain that caused the sharpness pulse stays removed.
…flow field Ported motion.comp and interpolate.frag to a CPU harness and ran them on image pairs with known motion. The interpolation shader is fine: handed the exact analytic flow it reconstructs the midpoint at 44-47 dB. The motion search is what was broken - it returned a flow field with a std of 8-12 texels on a scene whose true motion is uniform, and the generated frame scored consistently *worse* than a naive 50/50 blend of the two real frames. An exhaustive argmin over the same cost volume finds the true offset on 82-99% of pixels, so the cost function and descriptor were never the problem. Three defects in how the search walks that cost surface: The descent started at step 8 and took at most one move per step size before halving. With real motion of 2-5 texels the first probe lands 4x too far out, a spurious minimum there captures the search, and nothing afterwards can walk back. Replaced with a step-2 descent that iterates until it stops improving, then a step-1 pass. On the same scenes the flow std drops from 8-12 texels to 0.8-3.6, matching or beating exhaustive - iterating from a small step follows the local basin instead of jumping to a spurious global minimum. Cost is 54 blockCost calls against 46. The fine pass could not represent odd offsets. It began at step 2 and halved down to minStep, so Balanced/Flow/Clear (minStep 2) reached only -2, 0, +2 and Eco (minStep 4) never entered the loop at all. The flow was quantised to two flow texels, 8-16 full-res pixels at the shipped scale. The sub-texel refinement that should have covered this weighted the 3x3 by exp(-(cost - best) * 4), which against these cost magnitudes is nearly uniform and just returns the centroid. Replaced with a parabola fit on the three cost samples per axis - true sub-texel precision from 5 blockCost calls instead of 9. The confidence probe read outside the loaded tile. The fine pass only fills [RMAX-FR, RMAX-FR+span) of shared memory, but the probe at bestD +- 5 indexes past the end and was bounds-checked against RMAX rather than the loaded region, so it read uninitialised shared memory and produced garbage confidence - which is what gates the fallback to a crossfade in the interpolation shader. All probes are now bounded by what the tile actually holds, and the fine tile is sized to cover them. Result on the harness: 25.0 -> 41.7 dB on smooth content, 14.4 -> 19.5 dB on hard-edged content, and the generated frame now beats a 50/50 blend in every case tested rather than losing to it. The frame dump only ever ran inside the CNN generate path, so with the CNN defaulted off it captured nothing. It now runs on the active path and captures both real frames plus the flow field, which together determine the generated frame exactly. tools/fgdump_view.py turns a pulled dump into the real frames, a flow and confidence visualisation, and the generated frame reconstructed at each phase.
tools/frame-gen-tester builds a Win64 tester: a character walks a textured ground plane, spins, walks back, spins and loops, with wind in the trees behind and the camera following - so the walk phases give large global motion and the spin phases give a static background with independent local motion. Every element is a pure function of time, so rendering at t = n + 0.5 gives the exact frame a perfect interpolator would produce. The same source builds natively and writes real frames alongside their true 0.25/0.5/0.75 phases, which turns generated frames into something measurable instead of something to squint at. A ticker advancing one cell per frame makes a duplicated, dropped or reordered frame visible on its own. Running the shipped algorithm against that ground truth showed two things. The fine level's search was doing nothing. It is seeded by the coarse level and bounded to +-5 texels, so the 7 step-2 iterations it inherited could never be used. Cutting it to 2 gives bit-identical output for 22-36% less work - the fine level is where most of the cost is, since it has four times the pixels of the coarse one. The flow resolution cap was set too aggressively in the previous pass. On real content at 5 px per flow texel the generated frame only just beats a 50/50 blend, and at the Eco and Balanced settings it was actually *worse* than not interpolating - the flow could not resolve tree trunks or canopy edges, so it warped them wrongly. Quality climbs steeply to about 3.3 px per texel and plateaus after. Presets now run 0.20 to 0.30 of screen with the cap raised to 640, spending the saving above on resolution. Block radius was checked too: dropping it to 1 would pay for a lot of resolution but costs 1.3-2.4 dB, so it stays. Measured over the walk, spin, walk-back and spin phases at the 0.25/0.5/0.75 slots, generated frames now average +0.7 dB over a 50/50 blend and win 9 of 12 samples, with the clear wins on the panning phases (+1.0 to +2.6 dB) where interpolation is worth doing. The three losses are static-background frames where there is nothing to interpolate and any resampling can only lose. Static pixels also snapped to curr regardless of phase; they now snap to the phase-correct blend, which is what a static pixel should show at t.
…ering it exposed The tester could not hold its own frame rate. It cost 32.8 ms per frame natively - at the 30 FPS budget before emulation - because the ground and sky evaluated six transcendentals per pixel: two sin, a cos, and three hash1 calls that each contain a sin, 5.5 million per frame. Both users of that noise are periodic, so they are baked into tables at startup: the sky samples a precomputed fBm cloud strip, the ground indexes a tile colour table by integer tile coordinates, and per-scanline terms are hoisted out of the inner loop. 8.1 ms per frame now, four times faster, and the bar in bar() only takes a square root inside the antialiased band. Because holding 30 FPS is the point - that is the content rate the compositor is being tested against - it now benchmarks itself at startup and picks the largest of five sizes that renders inside budget, shows the measured rate and late-frame count in the title, and puts a green/red block on screen so the run is self-validating. The loop is 30 s out, turn, 30 s back, turn: 64 s total. The forest repeats every 140 m so the walk never leaves it, and the wrap happens well outside the view frustum. Running the search against the new scene showed it wandering. More iterations made the result *worse* - it2=7 lost 0.4 dB against it2=3 - because a search that moves on any improvement at all walks off into spurious minima, worst on static content where the right answer is not to move. It now requires a 6% improvement before stepping, which makes the result nearly independent of the iteration count, and the coarse budget drops from 4-7 iterations to 3-5. Both are cheaper and better. Measured against exact ground truth across the loop, every walking phase now beats a 50/50 blend at every preset and phase, by +0.4 to +1.7 dB. The spin phases still lose about half a dB: the background is static there so blending is already near-exact, and a rotating character cannot be represented by a translational flow at all.
…o the tester The tester gets three buttons (and keys 1/2/3) for 30, 10 and 2 FPS. The animation clock stays real-time, so a lower rate skips the frames in between rather than slowing the scene: the character jumps to where it would have been. The frame counter runs in 30 FPS units and only the step changes, so switching rates mid-run keeps the timing invariant and needs no re-anchoring. The tick marker jumping 3 or 15 cells makes the skip obvious, and it gives frame generation a much larger gap to bridge - near-ground motion goes from 18 px per frame at 30 FPS to 52 at 10 and 260 at 2. Running that found the cause of the blotches. Rendering the tester's frames through the shader and cropping into a low-confidence region shows the artifact is not blur from the crossfade fallback, as the shape suggested: it is blocky tearing. Single flow texels disagree with their neighbours, and each wrong one drags a patch of the frame in from the wrong place. 43% of the frame sits below half confidence at 10 FPS, in regions as large as 232k pixels, so there is plenty of room for isolated outliers. interpolate.frag now samples the flow through a 3x3 component-wise median rather than a single tap. A median drops isolated outliers while leaving motion boundaries intact, which a blur would round off. Measured against exact ground truth it gains +0.24 dB at 30 FPS, +0.13 at 10, and is neutral at 2, with one -0.06 dB case out of seven - and it removes the blocky tearing visibly, which PSNR understates because the artifact is localised. Against a 50/50 blend the mean is now +0.63 dB. A 5x5 median is worth another 0.17 dB but needs a 25-element selection per texel; the 19-compare 3x3 network is most of the benefit for a fraction of the cost. Dropping the first pyramid level to 1/8 was also tested: better on the walking phases but -1.95 dB on a static background, so it stays at 1/2. The fine pass now derives its upscale factor from the actual level dimensions instead of assuming 2.
The character was being shredded: limbs about four pixels wide broken into fragments with background cutting through them. Measured three candidate fixes against the exact frame a perfect interpolator would produce. Making the flow finer is not the fix. Going from 3.3 to 1.3 pixels per flow texel - 6.25 times the cost - buys only +0.88 dB on the character, and whole-frame quality peaks partway and then falls. The limiting factor is not texel density: the block window is 5x5 texels, so at any resolution where the flow is affordable that window spans both the limb and the background behind it, and the match is ill-posed no matter how finely the result is sampled. Per-pixel vector selection - having each pixel keep whichever neighbouring vector best explains it in both real frames - is also not the fix. It gains 0.4 dB on the character but costs 1.2 to 1.7 dB across the frame, because in flat regions many vectors explain a pixel equally well and noise decides. An acceptance threshold made no difference. What does work is refusing to warp where the warp is about to tear. When neighbouring flow texels disagree by more than a few texels, the two sides of the blend are pulling from unrelated places and the result fragments; those pixels are handed back to the temporal blend instead. Soft beats shredded. The nine samples were already being fetched for the median, so the spread costs sixteen min/max ops and no extra taps. Whole frame +0.65 dB averaged over the rate and animation cases, +1.26 dB against not interpolating at all, with one case out of seven regressing by 0.01 dB. On the character specifically, +0.49 dB at 10 FPS and +0.25 at 2 FPS - the rates where the shredding was visible. At 30 FPS the character was already intact and is unchanged.
…tput Measured where the softness comes from. Two findings. First, the output was not mostly warped. Wherever the flow is not trusted the shader fell back to an unwarped dissolve between the two real frames, and that path covered 41% of a 30 FPS frame, 48% at 10 FPS and 71% at 2 FPS - the tear gate added in the previous commit had pushed those up. A dissolve between two frames that differ is a double image, which is the blur. Second, quantifying it: measuring gradient energy where the ground truth actually has edges, generated frames retained 0.92 of the reference edge contrast at 30 FPS, 0.83 at 10 and 0.52 at 2. Catmull-Rom sampling instead of bilinear recovers almost none of that (0.907 to 0.931 on one frame) so resampling is not the cause; the averaging is. The fallback now takes the nearer warped side where a dissolve would visibly double, and keeps the dissolve where the two frames are close enough that it cannot be seen. Taking a single warped sample on an incoherent vector puts a sharp wrong patch on screen - visibly worse than the soft one it replaces - so it is gated on the same flow-coherence term the tear gate uses, and incoherent pixels keep the dissolve. Edge detail improves in 7 of 7 cases and SSIM in 6 of 7. PSNR falls slightly in 6 of 7, which is expected and is the point: PSNR is minimised by averaging under uncertainty, so it has been rewarding exactly the blur being removed. That bias is why earlier passes kept selecting the dissolve. Committing to one side whenever the two warped samples disagree was also tested. It adds a little detail but costs both SSIM and PSNR, so it is not included.
…uessing it The confidence driving the warp-versus-fallback decision was (second - best) / second from the block search. In a low-texture region both costs are near zero, so it collapsed to zero and handed the region to the fallback regardless of whether the match was fine - the warp was being distrusted for having nothing to match against rather than for being wrong. Fitted a replacement rather than guessing one. 5.2M pixels from 90 generated frames at 30 and 10 FPS across five points of the loop, each with the exact frame a perfect interpolator would produce, and for each pixel the optimal blend weight that minimises squared error against it. Ranked candidate signals by the error a fitted weight curve actually achieves. Two things came out of that which contradicted what the code assumed. Ranking signals by classification accuracy says the block score is the best predictor and photometric agreement is worse than chance; ranking by achievable error says the opposite. The classification view is dominated by the 87% of pixels where warp and fallback are already within 0.01 of each other and the decision is irrelevant. On the pixels where the choice matters, how well the warp's own two samples agree is the stronger signal - and unlike the block score it does not care how much texture the region has, which is exactly the defect being fixed. Confidence is now floor + (1-floor)*a/(a+|cA-cB|) with a = 0.09 and floor = 0.15, scaled by a block-score term bounded to [0.55, 1] so it can nudge but never blank a region, and by the existing flow-coherence term. Swept a and the floor; the result is flat to within 1% across a wide range, and this point dissolves less than the marginally better one. Error on the pixels where the choice matters drops 19% against the old form, and the share of the frame that is dissolved rather than warped falls from 46% to 31%. Whole-frame PSNR, SSIM and edge detail are close to unchanged, because the fallback was already replaced in the previous commit and is no longer a plain dissolve - so using it less often no longer buys much. The remaining gap to a per-pixel oracle (0.0235 against 0.0146) is a limit of the predictor, not of its tuning.
…o lose The test scene was flat-shaded shapes on a flat-coloured checkerboard. It has almost no high-frequency content, so it could not show texture loss at all - every measurement so far has been blind to the artifact being reported. The scene now carries real surface detail: a precomputed multi-scale noise table sampled in world space on the ground, so it moves correctly with the surface and fades with distance the way a mip chain would, plus grain on the trunks. 2.2x the high-frequency content of before. With that in place the artifact is measurable and visible. Blending two warped samples that are misaligned by even a fraction of a pixel cancels high-frequency detail, so a generated frame arrives with the surface texture washed out - 0.83 of the reference detail at 30 FPS and 0.76 at 10, worst at t = 0.50 where the blend is closest to even. The shader already put detail back after blending, but with two faults. It was scaled by (1 - smoothstep(9, 64, |mv|^2)), which fades it out as motion grows - so it switched off exactly where the cancellation is worst; removing that gate alone recovers most of the loss at 10 FPS. And it was set to 0.30, calibrated for a much smaller loss than actually occurs. The detail is now taken from the same weighted mix the colour came from rather than the curr side only, is always on, and is scaled 0.70. Texture retention goes 0.809 to 0.883 averaged over both rates and all three phases, 7 of 7 cases improved. PSNR falls 0.20 dB and SSIM 0.002, both within the noise of the earlier passes, and the result stays below 1.0 so nothing is being sharpened past the detail the true frame has.
…cts actually are Detail restoration raised from 0.70 to 0.95. The +-0.25 clamp turns out never to bind on this content (0.25 and 0.50 give identical results), so the gain was the only lever, and raising it costs 0.17 dB PSNR and 0.0016 SSIM while taking texture retention from 0.879 to 0.926 averaged over both rates and all three phases. It stays below 1.0, so nothing is being sharpened past the detail the true frame has. Also ran a decomposition to find where the remaining error lives, and it corrected two things I had assumed. Reconstructing with a per-pixel flow fitted directly to the true frame scores 47 dB and retains 99.7% of texture, which looks like proof that the flow is the whole problem. It is not: that field varies 11.89 px between neighbours against 0.70 px for the real one - it is noise fitted to the target, not a motion field. Forcing it back to a coherent field collapses it to 26.4 dB against the shipped 25.5, and on smooth-motion ground the shipped flow beats it outright. There is no clean headroom figure to quote. Flow resolution was re-swept on the textured scene, since the earlier sweep used the flat-shaded one and could not see texture loss. Going from 3.3 to 1.25 px per flow texel costs 7.1x and moves texture retention by 0.005 at 30 FPS and 0.016 at 10, with SSIM falling monotonically. Confirmed not worth it. Per-pixel flow refinement against photometric consistency - choosing, for each pixel, the nearby vector that best explains both real frames - gains 0.66 dB and 0.076 texture on the character crop at 30 FPS and almost nothing at 10, for roughly 50 extra taps per pixel per generated frame. Not included. What remains is structural. A 5x5 block match on a coherent field cannot separate a four-pixel limb from the background behind it, and no amount of tuning downstream of that recovers it.
…f warping it anyway The fallback for untrusted pixels still warped the nearer real frame - applying a vector we had already decided not to believe. That is what left thin objects doubled and tree trunks smeared into vertical streaks. Those pixels now simply hold the nearer real frame, unwarped. The difference is obvious in the frames and invisible in the metrics, which is why it took this long to find. Holding scores 0.33 dB WORSE on PSNR and 0.001 worse on SSIM, yet the character comes out sharp instead of doubled and the trunks solid instead of streaked. Texture retention, which does track it, goes 0.862 to 0.882. PSNR is minimised by averaging under uncertainty, so it has been quietly rewarding the exact artifact being reported, and every earlier pass that consulted it picked the ghost. At 30 FPS the change is a straight win on all three measures, because the flow is trusted almost everywhere and the fallback rarely runs. At 10 FPS it trades PSNR for a visibly cleaner image: regions we cannot track now judder between real frames rather than showing a persistent ghost. Also removed the flow-coherence term from the fallback gate. It was there to avoid putting a sharp wrong sample on screen, which is no longer a risk once the fallback stopped warping.
…ck to a real one The generated frames were largely copies of the real frames. At 10 FPS content, 49% of every moving pixel was taken straight from a real frame with no motion applied, and the output sat 32.9 dB from real frame n while being only 25.9 dB from the frame it was supposed to be. That is frame repetition wearing the costume of frame generation, and it is why the last build looked clean: the parts that looked clean were not generated. Root cause was the fallback. Every version of it - crossfade, warped near side, unwarped hold - answered 'the vector here is untrustworthy' by choosing a different image source. None of them answered it by choosing a better vector, so wherever the block matcher failed the frame stopped moving. New flowfix.comp runs once per generated frame over the flow grid and picks, per texel, the vector that best explains the two real frames at that specific phase: the estimate itself, its neighbours at 2 and 5 texels in 8 directions, and zero, scored by a 5-tap cross of |prev(p + t*mv) - curr(p - (1-t)*mv)|. The interpolator's existing 3x3 flow median makes the choice spatially coherent. Selecting per pixel instead of per flow texel was tried and shreds thin limbs. interpolate.frag drops the fallback path, the flow-coherence gate and the flow-consistency occlusion terms. With a phase-fitted field they cost quality rather than buying it. Measured against exact ground truth at both rates and all three phases: 30 FPS SSIM 0.9024 -> 0.9212 texture 0.949 -> 1.024 PSNR 32.78 -> 32.13 10 FPS SSIM 0.8120 -> 0.8341 texture 0.925 -> 0.948 PSNR 25.40 -> 26.30 Texture retention above 1.0 means the generated frame carries as much high-frequency detail as the real one. PSNR falls at 30 FPS because copying a real frame is what PSNR rewards when motion is small; it has favoured the artifact all along. At 30 FPS the character and the ground now track ground truth with no visible ghosting. At 10 FPS motion is genuine but edges on the fast-moving arm are still ragged: a 5x5 block match cannot separate a limb from its background across a 15 px jump.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Compositor frame generation on open GLSL shaders: block-matching motion estimation and fragment-shader interpolation wired into the Vulkan present path. A headroom-driven scheduler posts at the target rate under a non-blocking present mode (so an adaptive panel ramps up) and passes through under FIFO; the real frame always presents, so it never drops below native. Frame Gen controls live in the FX tab as an expanding toggle (like SGSR): 2x/3x/4x multiplier, quality preset and smoothness; Other-settings toggle. The HUD FPS reports the output rate (real + generated) while FG is active.