Visualization epic: out-of-core streaming viewer (sphereql-vis-server) + WASM studio + docs#35
Draft
bkahan wants to merge 53 commits into
Draft
Visualization epic: out-of-core streaming viewer (sphereql-vis-server) + WASM studio + docs#35bkahan wants to merge 53 commits into
bkahan wants to merge 53 commits into
Conversation
Introduce `sphereql-vis`, a pure (`sphereql-core`-only) emitter that owns a serializable `Scene` (point cloud + `SceneStats` + a `#[non_exhaustive]` `Overlay` set) and one hardened Three.js renderer. `to_html()` inlines the Three.js + OrbitControls runtime so files open offline (the old Python viewer claimed "self-contained" but loaded from a CDN); `to_html_cdn()` emits a smaller CDN-backed file. Fixes at the source: - projection-aware stats label (no more hardcoded "PCA variance" on a UMAP/Laplacian pipeline) - reduce-based min/max (fixes a latent large-N RangeError crash) - single-place non-finite filtering, deterministic per-category decimation, centralized </script> escaping, empty-scene handling Draw the structure the engine computes: 9 overlay kinds (centroids, classified bridges, slerp geodesic paths, Voronoi caps, antipodes, coverage maps, domain-group spokes, globs, manifold slices) with legend/overlay toggles, OrbitControls, and click-to-inspect. Consumers consolidated onto the shared crate: - sphereql-python `visualize`/`visualize_pipeline`: API byte-identical, now offline + projection-aware; old viz_template.html removed - new CI-safe `visualize_corpus` example (the true e2e viz demo) - `e2e_transformer` migrated off its duplicated inline template - umbrella `sphereql --features vis` (does not leak into --no-default-features) - shared example helpers extracted into a testable sphereql-examples lib Tests: sphereql-vis (11 + doctest), a CI-run overlay-join smoke test, XSS, NaN, offline-URL-absence, CDN-mode, radius-alignment, and a geodesic no-duplicate-vertex regression. CI/docs wired: crate tables, dependency edge list, publish order, feature matrix, CHANGELOG. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Reworks sphereql-vis/src/template.html into a celestial-observatory console (fully offline; no web fonts/assets): - Atmosphere: nebula gradient + film grain + vignette; a (θ,φ)-aligned lat/long graticule globe with a backdrop shell; crisp solid-disc point shader with a bright core. - Instrument chrome: hairline panels with HUD corner brackets, monospace typography, a wordmark + projection-kind pill badge, a hover reticle, and a staggered load reveal (prefers-reduced-motion aware). - Tabbed control deck — Domains / Overlays / Settings — with select all/none. - Settings: Scale (default 5×, up to 50×; the model spreads apart on screen so overlapping blobs separate), Radial spread (thickens the r shell), Domain spread (angular de-clumping), point size (down to 0.5), a Reference globe toggle, auto-rotate, and Reset to defaults. - 2D sky-chart minimap (equirectangular θ→φ) with a live camera reticle and click-to-aim navigation. - Reliable click-to-select via pointerdown/up + movement threshold (the `click` event is unreliable under OrbitControls); auto-rotate defaults off. All data/placeholder/escaping contracts preserved; 11 tests + doctest green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- UI scale slider (0.70×–1.60×) — zooms the HUD panels independently of the 3D scene (CSS zoom). - Color schemes — Aurora / Spectral / Viridis / Sunset / Ice, switchable live (recolors points, legend, minimap, selected point). - Nearest-chord neighbor rows tinted with the neighbor's domain color. - Pick threshold now scales with the model (maxR · 0.07 · scale) so points stay clickable at any Scale (incl. 50×). - Clickable "© bkahan/sphereQL" credit linking to the public GitHub repo. - All covered by Reset to defaults. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…s nav - Default Scale 12× (max 100×); camera frames the 12× model on load. - Zoom-speed control (slider; default 0.5) — less sensitive scroll zoom. - Save/Load view settings as a TOML file (scale, zoom, radial, domain spread, point size, UI scale, color scheme, globe, auto-rotate). - Anchored, domain-colored label pinned on the selected sphere; the hover reticle now snaps onto the actual point so the click target is unambiguous. - Click a point to auto-center the orbit on it; Esc / empty-click returns to the main sphere (smooth target tween). - Spread-aware bridges: endpoints follow the spread/radial transform. Adversarial review fixes: robust behind-camera reject for screen labels (getWorldDirection in-front test), clear hover on select, re-apply selection highlight in place on palette change (no panel flicker), and surfaced config load errors (console + button feedback) instead of swallowing them. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…mbed Bridges: - Tie each bridge to its exact source-point index (full-precision coord match) so its endpoint follows that sphere precisely under spread/scale, instead of a nearest-centroid approximation. - Highlighting a sphere now draws its bridges from the sphere to the connected domains (classification-colored) regardless of the global toggle, with a `bridges: N` count in the info panel. Navigation / input: - Zoom-to-cursor: intercept the wheel and dolly while keeping the world point under the pointer fixed (OrbitControls r128 has no zoomToCursor); touch pinch still uses the built-in zoom. Touch / mobile / embed: - canvas touch-action:none; panels are position:fixed and the page is locked to the viewport (overflow:hidden, overscroll-behavior:none). - Responsive layout ≤720px: header across the top, Domains/Info share the upper row, stats above a full-width filter bar pinned to the bottom; minimap + hints drop out; a ⛶ hide-UI toggle collapses all panels. - visualize_corpus prints a ready-to-paste <iframe> embed snippet. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…xed)
The radial coordinate r was a faithful pass-through of the raw embedding
magnitude, which on sparse/uniform corpora clusters in a narrow near-maximum
band (e.g. HandCrafted/UMAP r ∈ [0.95, 2.13]). Add a way to remap it.
- `PipelineConfig.radial` (`RadialConfig` { mode, lo, hi, percentile }) with
`RadialMode::{Magnitude (default), Fixed, Stretch}`.
- `RadialStrategy::MagnitudeStretch { m_lo, m_hi, r_lo, r_hi }` — affine map of
the embedding magnitude from a corpus source band onto [r_lo, r_hi], clamped.
- The fit path resolves the mode once (two-pass percentile scan of corpus
magnitudes) and bakes the bounds into the strategy, so corpus points AND
later queries get consistent r. UmapGraph now carries per-point magnitudes
so the graph-reuse (tuner) fit path can resolve bounds too.
- PCA's volumetric mode (which overrides any RadialStrategy) is auto-disabled
when a non-default radial mode is chosen, so Stretch/Fixed take effect.
- `visualize_corpus --radial lo:hi`; the example prints the resulting r range.
Default is unchanged (Magnitude pass-through, identical to before). Verified:
`--radial 0.2:1.9` on HandCrafted/UMAP gives r ∈ [0.200, 1.900] with the same
winning projection + score. Tests for the stretch math, the percentile bounds,
mode resolution, and a pipeline integration check.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- DOM labels anchored above centroid / antipode / domain-group markers, with a connector tick. Projected each frame, scaled by camera distance (0.7–1.7×), hidden behind the globe (front-hemisphere occlusion) and when the matching overlay layer is off. - Clickable: click any label to tween-focus the camera on it; clicking a domain (centroid) label solos that domain (others dimmed; click to restore). - Settings → "Labels": one checkbox per label kind present in the scene (Centroids / Antipodes / Domain groups), each toggling independently. Covered by Reset. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
refactor(vis): extract viewer.js + add ScenePoint.id and Scene::to_json Cross-cutting foundation for the visualization roadmap (decouple viewer from generator so it can rebuild from any Scene at runtime). - ScenePoint gains optional `id` (stable identity for cross-scene k-NN highlight and morph; viewer will key on id, not array position) plus a `with_id` builder. Serialized only when present. - Scene::to_json / from_json: canonical JSON wire form (doctested) for the drag-drop loader and the future WASM bridge. - Viewer runtime moved out of template.html into src/viewer.js, inlined at a new /*__SPHEREQL_VIEWER__*/ placeholder, so the baked HTML and the upcoming WASM studio share one implementation and cannot drift. Still a single self-contained offline page. - Re-anchor tests/emit.rs::embedded_payload: the data line is no longer adjacent to `const pts=D.points` (now in viewer.js); read the single `const D=` line directly. All 11 emit tests + 2 doctests green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> @
Split the monolithic viewer init into persistent setup + a rebuild(sc) / teardown() core so the viewer can swap to any Scene at runtime (a dropped file, or a live WASM pipeline) without reloading the page — the foundation for the generalize/studio/compare roadmap. - Persistent (created once): renderer/scene/camera/controls/lights/ raycaster and every static DOM event listener. Per-scene state moved to module-level lets that rebuild() reassigns and listeners read live (never a stale snapshot). - teardown() disposes the prior scene's GPU geometries/materials (points, globe, lines, every overlay group), removes them from the THREE scene, and clears generated DOM; only ever called synchronously at the top of rebuild() so animate() never sees half-torn state. - rebuild(sc) reads the Scene shape (points/overlays/stats/title/ surface_radius/show_axes) and resets all view settings to defaults, so a swapped scene starts clean. Added null guards (getHovered/search/ drawMinimapBase) for the transient null window. - Baked page just calls rebuild(D) once at boot. - Re-anchor tests/emit.rs::embedded_payload on the payload script's opening (the const pts=D.points line it keyed on moved into rebuild()). Verified: node --check passes; end-to-end emission unchanged (1097 KB, self-contained, 0 external scripts); adversarial diff-review vs the pre-refactor extraction found no behavioral regressions, leaks, TDZ, or stale-state carryover. All 11 emit tests + 2 doctests green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The viewer is now a general 3D sphere/embedding explorer: drop a Scene JSON anywhere on the page (or use Settings ▸ Open Scene JSON…) and it teardown()s the current scene and rebuild()s around the new one — no reload, no regenerate. - parseScene(obj): normalizes arbitrary input into the Scene shape. Accepts the full Scene::to_json shape OR a bare points array, and tolerates minimal points — each needs only finite x/y/z OR finite r/theta/phi; the missing pair is derived with the sphereql-core convention (x=r·sinφ·cosθ …; θ=atan2(y,x)∈[0,2π), φ=acos(z/r)). surface_radius falls back to the median ‖xyz‖, stats/overlays/title get sensible defaults. Non-placeable points are dropped; empty or shapeless input throws a human-readable message. - Drag-drop overlay (#dropzone) bound on window with a file-drag guard and an enter/leave depth counter; hidden #scene-file input + an Open Scene button reuse the soft-fail (✓/✗) feedback pattern. - Documented the public Scene JSON runtime contract in the crate docs. Verified with a headless THREE+DOM harness booting the real viewer.js: 20 checks incl. full-scene/bare-array/spherical-only/mixed/error inputs, a clean second-scene rebuild() (the drag-drop swap path, recomputing categories), and a real 775-point emitted scene round-tripping through parseScene. Rust: all sphereql-vis tests + doctests green; doc-snippets gate green; node --check clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Three pure-client explorer tools, added as header buttons: - Ruler (⟀): click two points; reports the great-circle angle (deg + rad) and chord, and draws the connecting geodesic on the shell via slerp (same arc construction as overlay.rs geodesic_path). Esc clears. - PNG (⤓): preserveDrawingBuffer renderer → canvas.toDataURL → download. - Share (🔗): copies a link whose #v= hash is base64(JSON) of camera pose + view settings + tool toggles (NEVER scene data); applyViewHash() restores it once after the boot rebuild, reading only validated numbers/known keys. Folds in fixes from an adversarial review workflow (8 confirmed of 21): - MAJOR: ruler slerp blew up for (near-)antipodal picks (1/sin(om)→∞, arc collapsed through the globe). Now special-cased: coincident → short segment; antipodal → clean great semicircle about a ⟂ axis; else slerp. Arc verified on-shell for all cases. - MAJOR: stored XSS via stats.sampled_from / dropped_nonfinite on the drag-drop path — now coerced to numbers in parseScene AND escHtml-ed at the render chokepoint. - MAJOR: rulerOn wasn't reset on scene swap — teardown now fully disarms the tool (flag + button + picks). - MINOR: dropped the deprecated escape/unescape from the hash codec (encodeURIComponent/decodeURIComponent round-trip verified). - MINOR: malformed overlays in a dropped scene no longer abort rebuild (per-overlay try/catch + kind filter in parseScene). - Documented the median-shell parity with Scene::surface_radius_for and the preserveDrawingBuffer tradeoff (kept for cross-browser PNG). Verified via the headless THREE+DOM harness (24 checks: ruler 0/90/180° + on-shell antipodal arc, PNG download, hash round-trip, ruler-reset, stats coercion, overlay filtering) plus all sphereql-vis Rust tests + doctests; page stays fully offline (1.1 MB, 0 external scripts). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Decouples the viewer's data source: a fitted pipeline (and free text) can now produce a sphereql-vis Scene entirely in the browser, feeding the same viewer as the baked HTML and drag-drop. - New optional deps behind features `scene` (sphereql-vis) and `lingua` (sphereql-lingua, implies `scene`), both added to `default` so `cargo --all-features` and the wasm jobs build them. - Pipeline.buildSceneJson(title) → JSON: maps exported_points → ScenePoint (stable id + certainty/intensity), category centroids, classified bridges, Voronoi territory caps, and domain-group spokes, with projection-kind/evr stats. Colors use the viewer's aurora palette + sorted-category assignment so overlays match the point colors. - LinguaStudio.process(text) → JSON: runs LinguaPipeline in-browser; each placed concept becomes a point (domain → category, surface form → label, normalized → id), each typed relation a geodesic path, with mean salience as the headline stat. - Both return JSON strings (not tsify): a Scene is deep and the viewer parses JSON anyway, keeping the generated .d.ts small. Sizes (measured): raw .wasm 1.5M → 2.3M after wasm-bindgen (no wasm-opt; wasm-pack's -Oz trims ~30-40%, gzip far more) — separate-file packaging is the right default; single-file base64 stays opt-in (Phase 5). Verified: native --all-features build + clippy -D warnings clean; new native tests (build_scene_json shape + ids/quality/overlays/stats; LinguaStudio smoke); wasm32 release build + wasm-bindgen confirm buildSceneJson / LinguaStudio.process are exported. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A live, in-browser projection studio under sphereql-wasm/studio/: paste prose and sphereql-lingua places each concept on the sphere as you type, or switch to Corpus JSON mode to run the full embedding pipeline and pick a projection — all client-side via WASM, re-projecting live. - Shared viewer, zero drift: the studio shell is emitted from Scene::to_html() (three.js + the SAME viewer.js inlined, empty initial scene) by examples/build_studio.rs, with the studio chrome injected after <body> and studio.js before </body>. studio.js drives the viewer's global rebuild()/parseScene(). - Off-thread compute: a classic Web Worker (worker.js) importScripts the --target no-modules wasm glue and runs LinguaStudio.process / Pipeline.buildSceneJson, posting Scene JSON back. Requests are debounced (~320ms) and id-tagged so the freshest paste wins; pre-init requests are queued (newest only) and replayed on ready. - build.sh assembles dist/ (self-contained index.html + worker/driver + pkg/*.wasm); prefers wasm-pack, falls back to wasm-bindgen + wasm-opt. Separate-file packaging is the default; single-file (base64 wasm) is a documented future opt-in. dist/ is gitignored. Headless-verified: studio shell emits with all chrome IDs + inlined viewer globals + empty scene + offline three; studio.js/worker.js pass node --check; a worker-protocol harness (stubbed wasm) covers queue-before-ready, ready, lingua/corpus dispatch, newWithConfig, and the unknown-kind error path; build.sh assembles dist end-to-end and the no-modules glue exposes wasm_bindgen/LinguaStudio/Pipeline/buildSceneJson. Remaining: a manual browser smoke-test (paste → worker → render). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Corpus mode gains a nearest-neighbor query: enter a query vector, the
studio runs pipeline.nearest off-thread and the viewer highlights the
hits by stable id.
viewer.js:
- rebuild() builds an idToIndex map (stable id → point index).
- highlightByIds(ids) emphasizes the matches (closest largest), dims the
rest, and fans geodesics from the nearest match to the others on the
shell. Accepts raw ids or {id,…} objects (NearestOut). Returns the
resolved count; [] clears. Persistent queryGroup wired into scalables
and cleared on teardown / re-query.
- Factored the robust great-circle builder out of the ruler into a shared
shellArc(a,b) (coincident → segment, antipodal → ⟂-axis semicircle,
else slerp), now reused by the query fans.
studio (worker.js / studio.js / chrome.html):
- The worker keeps the last corpus Pipeline alive and adds a `query`
handler → pipeline.nearest(query, k) → posts neighbors; a query before
any corpus fails gracefully.
- A query row (vector input + k + Find, Enter to submit) shows in corpus
mode; onmessage now branches scene-rebuild vs neighbor-highlight.
Headless-verified: a viewer harness covers id resolution (raw + object
ids), the geodesic-fan count, unknown-id skipping, clear, and
idToIndex/queryGroup reset on rebuild; the worker harness covers
corpus-keeps-pipeline → query → neighbors (honoring k) and the
query-before-corpus error. sphereql-vis Rust tests + doctests green; the
studio rebuilds with the query chrome. No Rust source changed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two ways to compare projections of the same corpus, both id-aligned. viewer.js: - Morph: setMorphTarget(sceneB) keys B's points by stable id; applyMorph(t) slerps each A-point's direction toward its id-matched B-direction and lerps the radius by t∈[0,1] (t=0 = A, t=1 = B). Unmatched points stay; t=0 falls back to the normal spread/radial view. Cleared on teardown. - Compare embedding (opt-in via #embed): the viewer accepts an injected scene + a synced camera over postMessage and broadcasts its own camera moves to the parent. The broadcast is epsilon-gated (not a bare flag), so the OrbitControls damping that follows an applied update can't start a feedback storm. Fully inert without #embed (baked viewer unaffected). studio: - A morph control (corpus mode): pick a second projection → the worker builds it as a morph target → a slider drives applyMorph. - compare.html / compare.js: two embed.html#embed iframes fed two projections by one worker; the parent relays each pane's camera to the other for synced orbiting. build_studio.rs now also emits embed.html (a plain, chrome-less viewer for the panes); build.sh copies the compare assets. Headless-verified: morph endpoints exact (t=0→A, t=1→B, t=0.5 on the geodesic with lerped radius), unmatched points untouched, clear/rebuild reset; embed sync covers scene injection, camera broadcast, the epsilon gate, the echo guard (no feedback storm), and inertness without #embed. All 6 JS harness suites + sphereql-vis Rust tests green; clippy clean. Remaining: a manual browser smoke-test of the live studio panes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two isolated viewer additions, both persisted in the TOML config. - Density shading: rebuild() bins each point's (original) direction into a θ×φ grid (the minimap's scheme) and stores its bin's normalized count as a per-point `density` attribute. A "Density shading" toggle flips a shader uniform that recolors the cloud as a cold→hot heatmap (and fades sparse points), so dense regions of the sphere read at a glance. - Pins: a 📍 pin-mode tool — click the globe shell to drop an annotated (θ,φ) marker with a floating, depth-culled label (clickable to remove). Pins survive scale/zoom (angular coords, re-placed on the current shell). Esc exits pin mode; rebuild clears pins; reset clears them. - Persistence: both ride the existing flat TOML — `density` as a bool and `pins` as a single base64-JSON value (not array-of-tables), encoded with the same encodeURIComponent codec as the share hash. Loaded pins are validated (finite θ/φ) and rendered via textContent (no innerHTML), so a hostile config can't inject markup. Headless-verified: density attribute carries correct normalized bin counts (clustered=1.0, lone=0.25), the toggle flips the uniform; pins add markers, keep explicit/auto(ASCII) labels, round-trip through the TOML base64 key, and reset on rebuild. All 7 JS harness suites + sphereql-vis Rust tests green; the studio rebuilds with the new controls. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The studio opened on an empty sphere (then auto-ran a tiny lingua example); it now opens on the rich 775-point HandCrafted corpus — the same auto-tuned, fully-overlaid scene visualize_corpus renders — because it simply looks better as a landing view. - New sphereql_examples::demo_scene(CorpusId) factors the load → embed → auto-tune (deterministic seed) → build_corpus_scene flow so it can be reused without duplicating the pipeline setup. - build_studio bakes demo_scene(HandCrafted).to_html() as the studio's (and the compare panes') opening scene; sphereql-examples + sphereql- corpus are dev-deps (native example only — not compiled into the wasm cdylib, so the shipped bundle is unchanged). - studio.js no longer auto-runs the lingua example on boot, so the baked scene stays until the user acts (type / example / corpus / morph). Verified: build_studio emits index.html + embed.html with 775 points / 1020 overlays (umap_sphere); sphereql-wasm bridge tests pass; clippy + fmt clean on the changed crates. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…default
Two studio fixes:
- The studio now OPENS in Corpus JSON mode (lingua is the alternate),
matching the baked 775-point demo scene that's already on screen.
- Switching to lingua and back to corpus no longer runs the tiny sparse
6-point example (the bug that replaced the 775). Corpus mode now
restores the baked demo scene via rebuild(D) — instant, no recompute —
so the 775 is always the corpus default.
Mechanics: per-mode input buffers (corpus starts empty for your own
{categories,embeddings}; lingua seeds the prose example); switching modes
saves/restores each buffer; entering corpus mode calls restoreDemoScene()
(rebuild of the baked global D); the debounced live re-projection now only
runs in lingua mode (corpus JSON runs explicitly via Run/example). The
demo corpus itself (2.1 MB of dense 128-dim vectors) is intentionally NOT
dumped into the textarea — the box is for pasting your own corpus, and the
'example' button drops a small editable template.
node --check clean; build.sh assembles dist with the 775 baked + Corpus
mode active.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…se one
Addresses three studio issues:
1. Remove the sparse corpus entirely. The 6-point CORPUS_EXAMPLE is gone;
the "example" button is now lingua-only (hidden in corpus mode), so
there's no path back to the sparse scene.
2. Return from the side-by-side: compare.html gains a "← studio" link.
3. Compare did nothing because it had no corpus to project. The demo
corpus is now shipped as a fetched dist/demo-corpus.json (emitted by
build_studio via demo_corpus_json) — NOT dumped in any textarea. The
worker can now re-project / compare / morph it:
- compare.js: empty box → compare the demo corpus (projA vs projB),
so it works out of the box; robust injection waits for each pane's
new `sphereql-embed-ready` signal (no lost scene if a pane is still
loading); also relays cameras as before.
- studio.js: run / morph fall back to the demo corpus when the box is
empty, so they act on the 775-point demo.
- viewer.js: the #embed block posts `sphereql-embed-ready` to the
compare host once its message listener is live.
The textarea stays empty for your own paste (per the earlier choice);
the 2.1 MB corpus lives only in the fetched file + the worker.
Verified: all 7 JS harness suites pass (incl. embed sync with the new
ready ping); build.sh assembles dist with demo-corpus.json (2.1 MB),
both drivers fetching it, the back link present, and no CORPUS_EXAMPLE;
clippy + fmt clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Query needs a worker pipeline, so it previously failed on the default
demo until you ran a corpus. The studio now primes the demo pipeline in
the background on boot — without touching the textarea or the displayed
scene:
- worker.js: new `load` kind builds the pipeline (kept for queries) but
returns no scene, so the baked demo scene on screen is untouched.
- studio.js: primeDemo() sends `load` with the fetched demo corpus once
wasm + the corpus are ready (config:null → fast PCA build). Because
nearest() works in embedding space, the projection used for priming
doesn't matter — query highlights the baked scene correctly by id.
Guarded by demoPrimed + corpusRan so it never overwrites a corpus the
user has run/morphed themselves.
Net: open the studio → query immediately finds neighbors in the 775-point
demo; paste your own corpus + Run and query switches to it. The textarea
stays empty (no 2.1 MB dump).
Verified: worker harness covers the new load kind (load → {ok,loaded},
then query works); all query/morph/embed harness suites still pass;
build.sh assembles dist with the priming wired in.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Bridge classification was gated on the outer projection's EVR, but for UmapSphere `explained_variance_ratio()` returns a kNN-recall score (neighbourhood preservation) orthogonal to relation validity, and ~0.09 on real corpora -- so it suppressed EVERY bridge to Weak (0 Genuine wormholes). Remove the gate; classify from projection-independent full-dim signals, implementing the documented balanced_affinity_quantile intent: Genuine when min(affinity_to_source, affinity_to_target) clears the home-affinity quantile AND the two category centroids are distinct (new overlap_separation_quantile floor). territorial_factor stays a soft discount; min_evr_for_classification deprecated (kept for serde). Replaces the misleading low-EVR test with one asserting classification is independent of the deprecated gate. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…pane locks
Four UI fixes:
1. Compare bar no longer shifts on "compare": #status had a variable
width (margin-left:auto) so its text changes reflowed the flex bar,
pushing the controls right. It's now a fixed-width, right-aligned,
ellipsised cell. Also reflowed the compare page to a flex column (bar
auto-height + panes fill the rest) so a wrapping bar can't overlap the
panes.
2. SphereQL header fits the screen: #hdr gains max-width:calc(100vw-32px)
+ box-sizing, and the subtitle flexes/shrinks (min-width:0), so the
centered header never overflows the viewport.
3. Resizable minimap: #mini gets CSS `resize:both` (min/max bounded); a
ResizeObserver keeps the drawing buffer matched to the element size
(MW/MH are now mutable) so the sky chart stays crisp at any size.
4. Independent orbit / zoom locks in side-by-side compare: two toggles
("⟲ orbit", "⊙ zoom") broadcast a `sphereql-lock` message to both
panes; the viewer's #embed block maps them to controls.enableRotate /
enableZoom + a wheel-zoom `zoomLocked` flag. Locks are re-sent to a
pane when it (re)loads.
Verified: all 7 JS harness suites pass (incl. a new lock-message test:
lockRotate→enableRotate=false independently of lockZoom→pinch off +
wheel zoomLocked); sphereql-vis Rust tests green; build.sh assembles the
dist with all four wired in.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…dio failure modes)
From the adversarial production-readiness audit:
- BLOCKER (correctness): parseScene computed the median surface_radius
over ALL norms, but Rust Scene::surface_radius_for filters finite &&
>0. A dropped scene containing origin points landed on a different
shell than baked. JS now applies the same filter — exact parity.
- BLOCKER (robustness): a missing/failed demo-corpus.json fetch was
swallowed silently while the status still claimed "demo corpus",
leaving query broken with a misleading message. Now tracked
(demoFailed) in studio.js + compare.js: honest status, and run/find/
morph give a clear "demo unavailable — paste/run your own" instead of
a confusing failure.
- MAJOR (robustness): a wasm load failure ({type:'fatal'}) left the UI
interactive but inert (pendingRun stuck, no recovery signal). Now a
wasmFailed flag short-circuits run/find/morph/build with "wasm
unavailable — reload the page" (studio.js + compare.js).
- MINOR (security): the #embed viewer accepted scene/cam/lock postMessage
from any window. It now requires e.source === parent (the compare
host); scene data was already parseScene-sanitized, so this closes
camera/lock spoofing.
- MINOR (correctness): morph's antipodal branch used lerp+normalize,
which collapses to the origin at the midpoint. Replaced with the
perpendicular-axis great-circle sweep (same as shellArc) so the path
stays on the sphere. Also corrected the "upper-median" comment.
- MINOR (robustness): compare shows "✓ compared" only once BOTH panes
receive a scene (was set on the first); build.sh errors clearly if
neither wasm-pack nor wasm-bindgen is installed.
Verified: all 7 JS harness suites pass (added a surface_radius-parity
test + a non-parent-source rejection test); sphereql-vis Rust tests +
doctests green; node --check + bash -n clean. The earlier deterministic
sweep (fmt/clippy -D/4 CI gates/wasm32 build) was already green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The viewer runtime (sphereql-vis/src/viewer.js) and the studio worker are browser JS with no Rust around their logic, so the Rust suite never covered them — the harness that caught real bugs during development lived only in a scratch dir. This commits it as js-tests/ and runs it in CI. - js-tests/harness.cjs boots the real viewer.js in a Node vm against thin THREE + DOM stubs and exposes its internals for assertions. - 7 suites (01-parse-scene … 07-worker) cover parseScene + surface_radius Rust-parity, the ruler/PNG/share tools, semantic query, morph, the compare embed-sync (incl. the echo guard + non-parent rejection), density+pins, and the worker message protocol. - run-all.cjs runs each suite in its own process and exits non-zero on any failure; a new `js-tests` CI job runs `node js-tests/run-all.cjs`. - Paths are repo-relative; the one integration check that needs a real emitted page is env-gated (SPHEREQL_EMIT_HTML) and skipped otherwise, so CI needs no Rust artifacts. Verified: `node js-tests/run-all.cjs` → all 7 suites pass; check-docs OK (js-tests is not a crate, no table/floor impact). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ates The rust-api audit flagged Scene::to_json's `.expect` as a potential panic on NaN/Inf. It isn't: serde_json renders non-finite floats as JSON `null` rather than erroring, so a built — or hand-constructed — Scene with non-finite coordinates serializes without panicking (the WASM bridge's buildSceneJson relies on this to never trap the worker). This adds a regression test that constructs a Scene with NaN/±Inf via the public fields, asserts to_json doesn't panic and yields valid JSON with nulls, and documents that Rust from_json won't read those nulls back (serde null↛f64) — fine, since the runtime consumer is the viewer's JS parseScene, which coerces null→0. No code change — the audit finding was a false positive; this verifies and pins the behavior. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Introduce the wire types the server-backed, millions-of-points viewer streams over, kept in the pure `sphereql-vis` crate so both the axum server (Phase B) and the JS DataSource (Phases A/C) share one definition. - `tile.rs`: SQT1 binary point tile (16B header + 20B records: x/y/z f32, cat u16, row u32, little-endian). encode/decode with magic, version, and length validation; TileError impls Display+Error. This is the on-the-wire streaming unit — positions exact f32 in v1, quantization reserved behind flags/version. - `manifest.rs`: point-free scene descriptor (bounds, palette, LOD scheme, total_points) reusing SceneStats/Overlay so one set of panels drives both the inline Scene and the streamed manifest. Bounds::enclosing falls back to the unit cube on empty input. Bounded regardless of N: the manifest is fetched once, points arrive as tiles by viewport/LOD. 9 unit tests (round-trip, empty, and every decode rejection path). Offline single-file path and tests/emit.rs unchanged; also fmt-splits a long line in the existing panic-free emit test. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…iewer
New `sphereql-vis-server` crate — the backend half of the out-of-core
viewer. It holds a corpus, its projection, and the indexes in memory and
streams the visible working set to the browser, so the renderer never has
to hold all N points (the offline single-file path stays the demo artifact;
this is the millions/streaming artifact behind the same data contract).
Build (`AppState::from_corpus`): load a corpus (any CorpusId — demo, stress,
or a Parquet path), embed it with the demo_scene seed convention, project it
(gating the O(n²) families — Laplacian eigenmap, kernel PCA — to PCA above
~10k points, and UMAP above ~100k), then build:
- a SpatialIndex over the projected positions, for cone/viewport tiling;
- an AnnIndex over the raw embeddings, for semantic neighbor (trace) queries;
- row-indexed per-point metadata (label, quality, raw 128-d vector) for
lazy inspection;
- a bounded Manifest (stats, category centroids + domain-group overlays,
palette, bounds, LOD) — size independent of N.
Endpoints (axum 0.8 + tokio, permissive CORS, body limit, graceful Ctrl-C):
GET /manifest bounded scene descriptor, fetched once
GET /tiles binary SQT1 tile of the points in a viewport cone,
stratified down to an LOD budget (per-category
proportional + even stride; deterministic)
POST /points lazy per-point metadata by row (the inspector)
POST /nearest ANN neighbors of a row or query vector (trace)
GET /category_stats the palette (name -> color -> count)
GET /health
Reuses the pure sphereql-vis Manifest/TilePoint contract and mirrors the
examples' corpus->scene join (palette, EVR labels, centroid/domain-group
overlays) so server-streamed colors match the offline viewer.
15 tests: 7 unit (projection gate, state build, budget/cone/decimation
determinism) + 8 integration driving the real router via tower oneshot
(manifest round-trip, decodable tiles, budget enforcement, inspect vectors,
self-excluding sorted neighbors, palette). Verified end-to-end over a real
socket. Registered in the workspace + README/architecture crate tables;
full-workspace clippy/test/fmt, the 4 gates, and strict rustdoc all green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…nics
Adversarial review found a remotely-triggerable panic: `AnnIndex::query`
opens with `assert_eq!(query.len(), self.dim)`, and the embedder always
produces 128-d vectors, so a `POST /nearest {"vector":[1,2,3]}` (any length
!= the index dim) panicked the handler — on an endpoint documented to ignore
bad input, and axum does not catch handler panics by default.
- Record the embedding dim on `AppState` and length-check the query vector in
`collect_nearest` before calling `ann.query`; a mismatch now returns an
empty result instead of panicking.
- Add a `CatchPanicLayer` to the router as defense-in-depth, so any future
handler fault returns 500 rather than dropping the connection.
- Tests: a wrong-length vector returns 200 + empty (regression), and a
correctly-sized vector returns neighbors.
Review also confirmed-safe (verified against upstream source): row alignment
across exported_points/labels/vectors/ANN/tiles (exported_points is strict
input order), `state.points[r]` indexing (cone queries only yield inserted
rows), cone/degenerate-angle paths, the LOD budget shift/truncation, and the
f64→f32 position downcast.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Phase A of the out-of-core viewer: introduce the seam that lets the viewer be fed two ways behind one async interface — manifest()/tiles()/pointMeta()/ nearest() — so callers (the studio, a later streaming renderer) don't care whether data is one inline blob or streamed from the server. The offline path is unchanged: boot now routes `rebuild(D)` through `InlineSource(D)` (whose .scene === D), so the baked file renders byte-identically. New in viewer.js (all inlined; the file stays self-contained — emit.rs's no-external-`src` + size contract still holds): - decodeTile(): reads the binary SQT1 tile (sphereql-vis tile.rs) via DataView, validating magic/version/length and honouring a non-zero byteOffset. - InlineSource: serves the interface from the in-memory scene (manifest with derived bounds/palette, all-points or budget-decimated tiles, lazy meta with derived spherical coords, positional-cosine nearest as a local stand-in for the server's ANN). - ServerSource: streams from sphereql-vis-server (/manifest, /tiles → decode, /points, /nearest, /category_stats) with an injectable fetch. - TileCache: bounded in-memory LRU + optional IndexedDB persistence (guarded). - makeWorkerDecoder: off-thread decode from an inlined Blob-URL Worker, with an inline-decode fallback when Workers are unavailable. - stratify()/catOrder()/tileQuery() shared helpers mirroring the server. Cross-language wire-format lock: tile.rs gains `golden_bytes_match` asserting encode_tile() of a fixed input (cat=260, row=70000 pin u16/u32 LE), and js-tests/08 decodes that exact byte string — so the JS reader and Rust writer can't drift without breaking a suite. Adversarially reviewed (4 lenses × independent verification). The one confirmed bug is fixed + locked by a test: a worker-backed decode transfers (detaches) its buffer, which corrupted the retained TileCache blob and broke every subsequent cache hit; tiles() now decodes a throwaway copy when caching so the cache keeps the pristine blob. Tests: 3 new headless suites (08 tile-decode, 09 datasource, 10 server-source) + harness gains opts.globals injection (fetch/indexedDB stubs) and exposes the new symbols; all 10 js suites + sphereql-vis Rust (incl. offline emit contract) green. GPU vertex-shader transform/picking — the other half of Phase A's perf work — is deferred to a browser-validated follow-up; the CPU path is untouched. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Phase D's server-side "trace" depth: category-graph paths, glob clustering,
and within-category drill-down — the queries the streaming viewer's trace
panel will issue. They run against the projection pipeline's category-
enrichment layer, so AppState now retains the SphereQLPipeline (previously
dropped after the manifest was built); it's Send+Sync, shared read-only behind
the existing Arc, no locking.
- POST /path {source,target} → the category-graph shortest path (ordered steps
with cumulative distance + per-hop confidence, total distance, path
confidence), or null when the categories aren't connected. Unknown category
→ 400.
- GET /globs?k=&max_k= → concept-cluster detection over the whole cloud
(centroid, member_count, radius, top categories); k omitted = silhouette
auto-select.
- POST /drill_down {category,k,vector} → k-NN within one category relative to
the query embedding, using the category's inner-sphere projection when
available; results enriched with row/label/category. Wrong-length vector →
400 (the projection would otherwise reject it).
All three route through pipeline.query() with serializable DTOs decoupled from
the internal embed types; the category/glob queries ignore the query embedding
so they're handed a zero vector. 5 new integration tests (glob detection, path
between real categories + unknown-category 400, within-category drill-down +
wrong-dim 400). Server crate: 7 unit + 15 integration green, clippy/fmt/strict
rustdoc clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The viewer's per-point work is now O(1)-per-interaction instead of O(N): - Transform/morph in the vertex shader. spread/radial/morph move out of the per-frame CPU loops (applyTransform/applyMorph) into `sphTransform` (shared GLSL const VERTEX_TRANSFORM), driven by uSpread/uRadial/uSR/uMorphT/uHasMorph uniforms + per-point aCatDir/aMorphDir/aMorphR/aMorphHas attributes. A slider tick is now a uniform write, not an N-point buffer rewrite. The `position` attribute stays at origPos; `curPos(i)` becomes the CPU mirror of the GLSL (the single source of truth for selection/minimap/ruler/geodesics), and drawMinimapBase + bridge sync read through it. pointsMesh.frustumCulled=false (the transform can push points past origPos's bounds). At defaults the transform is the identity, so the initial render is unchanged. - GPU id-buffer picking. getHovered renders only the points (pickMat bakes each id into color, sharing the transform uniforms so picks match what's drawn) to a 1px target at the cursor and reads it back — O(1) at any N vs the raycaster's O(N) intersect. Falls back to a transform-correct CPU screen-space pick when render targets are unavailable (the old raycaster path would now be wrong: it read untransformed origPos). Picks are coalesced to one per frame (animate→updateHover) so a fast mouse can't fire multiple synchronous readbacks per frame. pickGPU saves/restores all render state (target, visibility, material, viewOffset, and clear color/alpha). Verified: new js-test suite 11 locks curPos↔formula parity (spread/radial + morph), that applyTransform pushes the uniforms the GLSL consumes (CPU mirror and GPU get identical inputs), and the pick-id codec round-trip; all 11 js suites + sphereql-vis Rust (offline emit contract) green. Adversarially reviewed (4 lenses × verification): GLSL↔curPos lockstep and "no missed position-buffer reader" both confirmed safe; two minor review findings fixed (clear-state restore, readback coalescing). The actual GLSL render is browser-validated — needs a one-time smoke-test (open an emitted scene / the studio, drag the spread/radial/morph sliders, hover + select points). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…r (Phase C) The viewer can now render a corpus too large to hold in memory by streaming the visible working set from sphereql-vis-server, instead of inlining the whole scene. - TileStreamer (orchestration): turns camera motion into tile requests against a DataSource and keeps a bounded working set of per-tile meshes via an injected sink. A persistent coarse BASE tile (whole sphere, LOD 0) gives context; a DETAIL tile tracks where the camera looks (finer LOD as you zoom in). Recently seen detail tiles stay cached up to a mesh budget (LRU); identical viewports dedup; a load that resolves after its entry was evicted/cleared is dropped. - tileMeshSink: builds a THREE.Points per tile (palette colours by cat id, point sizes, and a pick id baked from the GLOBAL row so id-buffer picking resolves a row across tiles). removeTile disposes only the per-tile geometry — the material is shared (disposing it would thrash GPU programs on every eviction). - connectToServer(url): fetch the manifest, build the scene chrome (globe + overlays + stats + a palette legend) with zero inline points, then stream by viewport. Camera 'change' drives throttled, cancellable tile updates; the ServerSource carries an IndexedDB+LRU tile cache + worker decoder. disconnectServer tears it all down (listener, pending timer, group, streamer). - Boot: `#server=<url>` switches to streaming mode; offline-by-default otherwise (no such hash → the viewer never touches the network, contract intact). Verified: js-test suite 12 locks the streamer orchestration (camera→request, base+detail, dedup, LRU, cancellation), tileMeshSink (geometry + global-row pick ids + clean re-add/remove), and the safeColor palette-injection guard. All 12 js suites + sphereql-vis Rust (offline emit contract) green. Reviewed (focused adversarial pass): three findings fixed — a dangling throttle timer dereferencing a nulled streamer on disconnect/reconnect, removeTile disposing the shared material, and CSS-attribute injection via a server-supplied palette colour. The actual streamed render needs a one-time browser smoke-test (`cargo run -p sphereql-vis-server -- --corpus stress`, then open a viewer at `#server=http://127.0.0.1:8080`). Streaming-mode hover/select/inspect over tiles is deferred to Phase D. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ver side)
The debugger's diagnostics + query/filter depth, server side (fully testable):
- GET /diagnostics → projection-health dashboard data: projection_kind + EVR,
the pipeline's projection_warnings (message/severity/evr), 16-bin certainty
and intensity histograms over all points, and the lowest-certainty outliers
(where the projection is least faithful), each with row/label/category.
- GET /tiles gains `cats=<id,id,…>` and `min_certainty=<0..1>` filters, applied
to candidate rows before LOD decimation — isolate a category, hide
low-fidelity points. Read-only, no state change.
Client plumbing so the streaming viewer can use them:
- ServerSource.diagnostics() (GET /diagnostics).
- TileStreamer.setFilter({cats:[ids], minCertainty}): merges the filter into
every tile request (base + detail) and reloads so the whole streamed view
reflects it.
Tests: server gains 2 unit (category + certainty filtering) + 2 integration
(/diagnostics shape + histogram coverage + ascending outliers; /tiles cats +
min_certainty) → 9 unit + 17 integration green. js suites 10/12 gain
diagnostics() + setFilter coverage → all 12 green. clippy/fmt clean; the
sphereql-vis offline emit contract holds. The diagnostics-dashboard + filter
UI panels (browser) are the next, smoke-tested layer.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…able state The "tune" pillar: re-project the in-memory corpus with a different projection kind at runtime and see the new layout, without restarting. - AppState gains assemble() (the project→index→manifest core, factored out of from_corpus) and reproject(kind), which reconstructs the pipeline inputs from the live state (f32 inspector vectors → f64 embeddings, palette → category names, stored labels) and rebuilds with the new kind (size-gated as usual). - Server state is now hot-swappable: Shared = Arc<RwLock<Arc<AppState>>>. Read handlers snapshot the inner Arc under a brief read lock (lock-free in practice); POST /reproject builds the new state off the async runtime via spawn_blocking and atomically swaps it under the write lock — so the heavy rebuild never blocks concurrent tile/query reads. Returns the new manifest. - parse_projection lifted into the lib (shared by the CLI + the endpoint); ServerSource.reproject(kind) on the client. Verified: state unit test (reproject PCA→UMAP preserves the 300-point corpus + categories, swaps the kind), 2 integration tests (live swap visible to a subsequent /manifest; unknown kind → 400) → 10 unit + 19 integration green; js suite 10 covers ServerSource.reproject. Socket-validated end-to-end: PCA → /reproject umap_sphere → /manifest reflects umap, tiles re-serve at the new positions. clippy/fmt/strict-rustdoc clean. This completes the cargo-verifiable server side of the debugger (inspect via /points, trace via /nearest+/path+/globs+/drill_down, diagnostics, filter, and now tune). The viewer-side debugger panels (inspector/dashboard/filter/tune UI, streaming-mode pick) are the remaining browser-validated layer. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… (Phase D) The browser half of Phase D: turn the connected (streaming) viewer into a projection debugger. All of it is gated on streaming mode, so the offline inline path — and its 12 js suites + the emit contract — is untouched. - Inspect: click a streamed point → pick its global row (CPU pick over loaded tile points, coalesced to one/frame) → fetch /points + /nearest → fill the info panel (label, category, θ/φ/r + certainty, a diverging raw-vector sparkline, clickable cosine-neighbors that drill in). Hover shows a minimal tooltip + reticle without a per-point fetch. - Diagnostics dashboard (new "Diag" tab): EVR bar, projection warnings (severity-styled), certainty + intensity histograms, and low-certainty outliers (clickable → inspect), from /diagnostics. - Tune: a projection dropdown → ServerSource.reproject(kind) → re-stream at the new positions + reload diagnostics, live. - Filter: click a legend category to toggle it out of the stream, + a min-certainty slider; both push TileStreamer.setFilter (server-side tile filtering). safeColor guards the palette-driven legend. template.html gains the Diag tab (tune/filter controls + diag-content), an inspector vector-sparkline slot, and histogram/warning CSS. Verified: new js suite 13 drives the whole flow against a mock fetch (connectToServer fetches manifest + base tile + diagnostics and installs a live streamer; selectStreamRow issues /points + /nearest; renderDiagnostics runs clean) — all 13 js suites + the sphereql-vis offline emit contract green. The inline path is fully gated off (no regression). The actual rendering/interaction is browser-validated (open a viewer at #server=… and click/tune/filter). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…, filter reset) Adversarial review of the streaming debugger UI found three real issues: - Inspector showed θ/φ/r as "—": the /points PointMeta DTO carried no coordinates (only the suite-13 mock wrongly added them). Add x/y/z to PointMeta (from the stored display position) and derive r/θ/φ client-side in selectStreamRow — so coords show for ANY inspected row (clicked, neighbor, or outlier), not just the picked one. - Dropping an inline scene while connected to a server leaked the streamer + camera listener AND left getHovered/pointerup routing to the (dead) streaming path, breaking interaction on the new inline scene. teardown() now calls the idempotent disconnectServer() first (no-op in the pure-inline path — the 13 js suites stay green). - The min-certainty slider wasn't reset on connect, so a reconnect showed a stale value out of sync with the (reset) category filter. connectToServer now resets it. Review-confirmed safe (no change): mode-branching (streaming returns a global row, never indexed into pts[]), selectStreamRow error handling, the tune re-stream flow, the palette→cat-id filter mapping, and diagnostics escaping/ empty-field handling. Tests + mock aligned to the real DTO; 13 js suites + 10 unit / 19 integration server tests + the offline emit contract all green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ViewHash restores server+filter+cam shareLink() now serialises the connected server URL, the category filter off-set, the mincert value, and the selected row into the #v= hash alongside the existing camera/settings payload. applyViewHash() detects a `state.server` key and reconnects asynchronously, then applies camera + settings + filter on top of connectToServer's rebuild (which would otherwise reset the view to the default frame). _streamSelectedRow is added as a module-level variable to track the inspected row across shareLink/selectStreamRow calls. Returns the connectToServer Promise from applyViewHash so callers can await full restoration. Harness gains _streamFilterOff and _streamSelectedRow getters. Suite 14 (23 assertions) exercises offline round-trip, streaming encode, and streaming restore including camera, filter, and selRow. All 14 JS suites + Rust tests + 4 gates pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
rebuild({points:[]}) sets #empty to display:flex (z-index 5, above the
canvas at z-index 0) which never gets cleared in streaming mode, so the
"no points — empty corpus" text permanently overlaid the tile canvas.
One-liner after rebuild() in connectToServer() hides it immediately —
tiles will render on the canvas beneath, and reproject clears/reloads
tiles without touching #empty so it stays hidden.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Three bugs killed the UMAP switcher and server-side filtering: 1. tileQuery silently dropped `cats` and `min_certainty` params, so filters never reached the server. 2. ServerSource.tiles() used the same cache key before and after /reproject, so UMAP got PCA tiles from cache. Fix: add _gen counter incremented on reproject(); append @n to the cache key only (fetch URL stays clean). 3. Reproject errors were swallowed — now surfaced via flashButton. Also adds a server connect UI to the Settings tab (URL input + Connect/Disconnect button, id=server-url / id=server-connect), wired in viewer.js so the user no longer has to hand-edit the hash. The #server= boot path still works. 14 JS suites + 19 server tests green. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…aked dist labelRefDist was computed once at rebuild() as DEF.scale*maxR*2.6 and never updated. When the scale slider changed curScale, label positions moved correctly (projectToScreen * curScale) but the font-size formula still compared against the stale constant — labels appeared frozen across a 6–24x globe size range. Fix: compute the reference distance live in updateLabels() as curScale*SR*2.6 so font scales proportionally with both scale changes and zoom. Also widens the font clamp from [0.7,1.7] to [0.5,2.2] for more range. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds `--emit-html <path>` and `--open` to sphereql-vis-server so that starting the server and opening the viewer is a single command: cargo run -p sphereql-vis-server -- --corpus stress --emit-html target/viewer.html --open `emit_viewer_html` builds a minimal empty Scene, calls `scene.to_html()` (three.js inlined, fully offline once written), and injects a small `<script>` before `</body>` that sets `location.hash = 'server=<url>'` on load — triggering the viewer's existing auto-connect path without requiring any manual hash editing. If the bind addr is `0.0.0.0:PORT` the auto-connect URL is rewritten to `127.0.0.1:PORT` so the browser can actually reach it. `--open` / `-o` opens the file in the OS default browser (cmd /c start on Windows, open on macOS, xdg-open elsewhere). The `--emit-html` / `-e` flag creates parent directories as needed. All 19 Rust server tests + 14 JS suites pass; clippy + fmt clean. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…tml) Bare `--emit-html` (no path argument) now defaults to sphere_viz.html so the minimal one-liner works as intended: cargo run -p sphereql-vis-server -- --corpus stress --emit-html --open Refactored parse_args to collect argv into a Vec<String> with index-based iteration so the next token can be peeked without consuming it; if the token after --emit-html is absent or starts with '-', the default is used. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The previous approach (location.hash = 'server=...') was a no-op: the #server= IIFE in viewer.js runs when the script is first evaluated, so by the time the injected <script> fires the IIFE has already exited with an empty hash. Setting the hash afterwards triggers no re-check. Fix: since the injected script runs after viewer.js, connectToServer() is already defined as a top-level function — call it directly. The guard (!location.hash || location.hash === '#') still skips the auto-connect when a #v= session hash or explicit #server= hash is present. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…studio When `sphereql-wasm/studio/dist/` exists (built by `cargo run --example build_studio`), the server now auto-detects it and serves the full WASM studio as its front-end: GET / → studio index.html (auto-connect injected in memory) GET /studio.js → WASM driver GET /compare.html, /embed.html, /demo-corpus.json, /pkg/* → ServeDir This combines the two previously separate workflows into one: cargo run --example build_studio # once cargo run -p sphereql-vis-server -- --corpus stress --open The one-liner opens http://127.0.0.1:8080/ with the full studio: corpus textarea + lingua mode + side-by-side compare + WASM pipeline, all on the same page as the live streaming debugger (streaming tiles, inspect, tune/reproject, diagnostics). --open now fires after TcpListener::bind so the browser doesn't race the server. build_router gains a studio: Option<StudioAssets> param (tests pass None); tower-http gains the "fs" feature for ServeDir; lib.rs re-exports StudioAssets. --emit-html still writes a self-contained offline viewer. All 19 Rust server tests + 14 JS suites pass; clippy + fmt clean. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…onerror When the WASM studio is open alongside the streaming server, changing #studio-proj to UMAP posted to the WASM worker, which ran UMAP synchronously on the demo corpus (O(n²), 30+ s or crash). With no worker.onerror handler the worker crash/hang left the status stuck at "computing…" indefinitely; subsequent lingua requests queued behind it. - studio.js: add worker.onerror so a crash surfaces an error message - studio.js: in run()'s corpus branch, check window.__sqServerReproject first; if set (server connected) delegate projection to the server and return without touching the WASM worker - viewer.js/connectToServer: set window.__sqServerReproject (closes over doReproject with PascalCase→snake_case normalisation for studio values) and sync both #studio-proj and #tune-proj from the server's initial projection_kind; doReproject also syncs both dropdowns after each swap - viewer.js/disconnectServer: clear window.__sqServerReproject Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…dates, reasoning-chain rendering
Rework viewer.js into a createViewer(rootEl, opts) factory so multiple
viewers can run side by side, each with its own GPU/DOM/scene state.
- createViewer returns { rebuild, updateScene, drawChain, highlightByIds,
setMorphTarget, applyMorph, clearMorph, dispose, camera }; DOM nodes
resolve via data-* attributes with #id fallbacks so template.html works
unchanged. Event listeners use an AbortController + ResizeObserver, and
dispose() also releases OrbitControls listeners.
- updateScene(sc): view-preserving update. Same N/cats rewrites position,
color, strength and catDir buffers (and idToIndex, since slots recycle)
in place, keeping camera/spread/radial/scale and any active selection;
structural changes fall back to a camera+transform-preserving rebuild.
- strength channel: deriveStrength (explicit > certainty > intensity > 1.0)
drives point size and fragment opacity, robust to raw emitter JSON that
lacks the field. parseScene shares the same helper.
- drawChain(chain): reasoning chain as an animated draw-on line with hop
markers and billboarded -[rel]-> labels; accepts the emitter nodes
[{id,pos,rel}] shape. geodesic_path overlays render through the same path.
- Drop the server/streaming client, tile decoder, minimap, pins, ruler,
PNG export and config chrome; keep the #embed compare protocol.
- onSelect(id) callback; selection re-applies in place across ticks without
recentering the camera or re-firing the callback.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…late The viewer.js refactor removed the minimap and the server/debugger raw-vector sparkline, leaving their markup and CSS in template.html as dead, never-queried elements (canvases #mini and #info-vector). Remove both panels, their styles, and the mobile media-query rule. Studio rebuilt and verified headless: 12/12 checks pass, only the #c canvas remains. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adversarial review of the ef5d558 createViewer rewrite turned up nine real bugs; fixes are confined to viewer.js. The new live-update paths (updateScene/drawChain) have no consumer yet, so several were latent — fixed before the streaming-debugger caller lands. - drawChain: drop the redundant grp.scale (chainGroup is already in `scalables`, so it was scaling the chain by curScale²). - updateScene fast path: refresh the size buffer from catVisible when nothing is selected — slots recycle across categories per tick. - selectPoint: gate the dimmed-others size on catVisible so selecting a point no longer reveals hidden categories (HIGH). - dispose: release the #embed `message` + OrbitControls `change` listeners (cleanup hook + ,sig + aniRunning guard) so a torn-down viewer can't be rebuilt/mutated by late postMessages (HIGH). - disposeObject: dispose material.map — Material.dispose() doesn't cascade, so makeTextSprite CanvasTextures leaked on the GPU. - highlightByIds: reset a prior selection (no camera revert) so its lines/label don't linger and the next tick can't clobber the query. - hover/select tooltips: derive strength via deriveStrength(p) so raw emitter points (no `strength` field) don't throw in the rAF loop. - bridges: keep the `to` end raw during morph to match the shader's no-spread/radial frame instead of detaching to a stale position. - soloCat: clear it on direct legend toggle / setAll so the solo state and its CSS class can't go stale. - spread slerp: add the antipodal guard (rotate by (1-spread)*om toward the morph branch's helper axis) so a point antipodal to its category pivot no longer collapses/NaNs and GPU pick stays in sync with draw. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add docs/visualization.md — the full runbook + architecture + design + API for the 3D viewer (offline emitter, WASM studio, streaming server), framed for extracting viewer.js as a generic embedding-visualization tool — and a sphereql-vis-server README (the crate had none). Refresh the stale vis docs to the current feat/vis-server reality: - sphereql-vis/README.md: rewrite for the manifest/tile (SQT1) streaming contract and viewer.js as the shared createViewer runtime - architecture.md + root README crate rows: globs/manifold slices, the SQT1 contract, vis-server as a consumer - docs index, project-status, CHANGELOG, quickstarts, examples, use-cases: visualization pointers + honest streaming-server status Docs honestly flag that the streaming browser client was removed from viewer.js by the instantiable refactor (server complete + tested; client mid-migration). Doc-only; check-docs / check-versions / check-doc-snippets all green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
…rded auto-connect)
Prompt 20 of the streaming-viewer re-port epic (docs/.prompts/20). Closes
S5/S6/N1 and adds the B1 server-side guard:
- serve a minimal static landing page at GET / when no WASM studio is built,
so `--open` never lands on a 404; add root_serves_landing_page_when_no_studio
- replace find_studio_dir with a 3-state StudioProbe (Ready/Partial/Absent); a
partial build (index.html without studio.js) is refused with an actionable
"run sphereql-wasm/studio/build.sh" warning rather than silently
- collapse open_url to a single unconditional `{server_url}/`
- guard inject_auto_connect with `typeof connectToServer==="function"` +
try/catch, so the injected script is an inert no-op until the client re-port
defines connectToServer (no uncaught ReferenceError, no blanked page)
- fix the now-stale USAGE --open help and the README GET / row
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…wer factory The streaming/debugger client and the offline Settings/tools chrome that ef5d558 dropped are re-ported into the createViewer(rootEl,opts) factory as instance-scoped closures (q() lookups, {signal}/cleanups[] teardown), not the old module-globals. All changes share viewer.js so they land together; prompts 21-25 (docs/.prompts) + the offline full-parity re-port: - 21: module-level DataSource seam — decodeTile/catOrder/stratify/tileQuery/ safeColor/InlineSource/TileCache/ServerSource/makeWorkerDecoder (SQT1 decoder, LRU+IndexedDB cache, worker decode). - 22: streaming renderer (TileStreamer/tileMeshSink/streamColorMaterial) + connectToServer/disconnectServer; fixes B1 (connectToServer exists again) and S2 (window.__sqServerReproject set on connect / nulled on disconnect). - 23: streaming debugger UI — tab switcher, inspect (selectStreamRow), the /diagnostics dashboard, live re-projection (tune), category + min-certainty filtering, re-added #info-vector sparkline; fixes S4. - 24: #v= streaming session restore + shareLink; fixes S3. - 25: gated opts.expose hook + js-tests harness rewrite for the factory (boot- capture shim, live-getter merge via Object.defineProperties); fixes S1. 14 suites pass (02-tools/06-density-pins restored for the re-ported tools). - offline parity: Settings sliders (+uiScale/DEF.ui), scheme/applyPalette, globe/auto-rotate/density toggles, reset, Open-Scene + file drag-drop, search, hud-toggle, AND the ruler, PNG export, pins, and TOML config save/load. Hard invariants preserved: inline sphTransform<->curPos/transformPos lockstep untouched; streamed tiles carry no client transform; the shared tile material is disposed once; escHtml/safeColor on every foreign string; offline HTML stays self-contained (no external src=). Review fixes (code-review high): teardown() disconnects any active stream so an offline scene swap can't leave a zombie stream (getHovered/updateHover stranded on the streaming branch); "hide all" categories hides the tile group instead of showing everything; updateScene re-syncs sliders after a structure change; applySettings isFinite-guards numeric fields (malformed .toml can't zero-scale the scene). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…s for wasm32 The CI `wasm-test` job (`wasm-pack test --node sphereql-wasm`) compiles dev-dependencies for wasm32, which dragged the native-only `sphereql-examples` and `sphereql-corpus` (parquet/arrow → `getrandom` without the "js" feature) into the wasm target and failed: "the wasm*-unknown-unknown targets are not supported by default". Pre-existing (failing before the streaming re-port; the build-only `wasm` job passes because it skips dev-deps). Move those two dev-deps under `[target.'cfg(not(target_arch = "wasm32"))'.dev-dependencies]` so they are excluded from wasm builds. The `build_studio` example runs on the host, where they remain available; the wasm-bindgen-test suite (tests/web.rs) doesn't use them. Verified: `cargo build -p sphereql-wasm --target wasm32-unknown-unknown --tests` compiles cleanly, and the native `build_studio` example still builds. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…compiles Follow-up to the dev-dep target-gating: that fixed the getrandom error but surfaced the next layer — `wasm-pack test --node` also compiles examples for wasm32, and build_studio uses the (now wasm-excluded) sphereql-examples / sphereql-corpus. Gate the example's real main on `not(target_arch = "wasm32")` with a trivial wasm stub; it's a host-only tool (bakes the demo scene/corpus from parquet) that never runs on wasm. Verified: `cargo build -p sphereql-wasm --target wasm32-unknown-unknown --all-targets` compiles (including the example), and the native all-targets build still compiles the real example. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This branch delivers the full sphereQL 3D visualization epic:
sphereql-vis) — aScenemodel + hardened, self-contained Three.js emitter (Phases 0–7: drag-drop reload, ruler/PNG/share, semantic query, compare + morph, density + globe pins).sphereql-wasm/studio) — a live in-browser projection studio (paste prose / corpus JSON → pipeline → scene), sharing oneviewer.jsruntime with the offline file.sphereql-vis-server, axum) — holds a corpus + projection + ANN/spatial indexes in memory and streams binary SQT1 tiles by viewport for million-point corpora, with a boundedManifest, lazy per-point metadata, and trace/tune endpoints (Phases A–D).viewer.jsis now acreateViewer(rootEl, opts)factory.49 commits; ~84 files; +11.5k/−1.3k.
viewer.jsby the instantiable refactor (ef5d558) and is mid-migration — no in-tree front-end currently consumes/tilesA correctness review of the recent changes found one blocking issue and several should-fix items, all stemming from the streaming client being dropped while its scaffolding was left in place:
--emit-html/ studio auto-connect injectconnectToServer(url), which no longer exists → uncaughtReferenceError; nothing streams.js-tests/headless harness is fully broken (all 14 suites fail at boot); the CIjs-testsjob is red on this branch.window.__sqServerReprojectno longer set).#v=share links restore camera only (streaming session lost).template.html(connect/tune/filter, null-guarded but unwired).find_studio_dir'sstudio.jssentinel isn't produced by thebuild_studioexample alone.--openwith no built studio opens a 404.The render path itself is verified sound (the 9-bug correctness pass, GPU↔CPU transform parity, leak-free
dispose(), server routing) — this is decoupling fallout from a mid-flight extraction, not rot.Validated
cargo test -p sphereql-vis -p sphereql-vis-server— green (53 tests).--corpus hand_crafted(every endpoint returns real data;/reproject→laplacian gave EVR 0.92).check-docs,check-versions,check-doc-snippets.node js-tests/run-all.cjs— failing (S1), to be fixed before merge.Remediation plan
A sequenced, production-grade implementation spec is staged locally at
docs/.prompts/19–25(gitignored, like the existing prompts): re-port the streaming client onto thecreateViewerAPI and resolve B1/S1–S6 as six PR-sized prompts.How to verify
Full runbook, architecture, wire contracts (Scene JSON / Manifest / SQT1), HTTP API, and the generic-vs-coupled extraction map:
docs/visualization.md.🤖 Generated with Claude Code