Skip to content

ARCHITECTURE.md

Latest commit

 

History

History
123 lines (93 loc) · 20.4 KB

ARCHITECTURE.md

File metadata and controls

123 lines (93 loc) · 20.4 KB
Raw

Web Architecture

The web/ app is the Patternflow site at patternflow.work: landing page, the interactive guide, Pattern Lab, the pattern community, browser flasher, edition shelf, journal and roadmap in a single Next.js project. One codebase, two deployments: Vercel serves the public site, and a Raspberry Pi at community.patternflow.work runs the same build with COMMUNITY_ENABLED=1, which switches on the SQLite-backed community, the build worker and the hosted edition images. Everything else is static or client-side.

Stack: Next.js 16 (App Router) · React 19 · TypeScript · Tailwind CSS v4 · Three.js via react-three-fiber · Zustand · dockview · MDX · Drizzle + better-sqlite3 · Better Auth.

Route map (src/app/)

Route What it is
/ Landing page (HomeView opened on the hero tab)
/build · /pattern · /inside The same HomeView, deep-linked to a tab — real URLs for SEO/sharing, not separate pages
/build/breadboard Breadboard electronics build path
/guide · /guide/build · /guide/play · /guide/make · /guide/audio (each + /ko) The interactive guide: a hub and four guides on one continuous 3D stage (components/guide/), with the firmware's input logic and screens ported to lib/guide/ and the real console and audio editor in windows (public/guide/console-demo/, audio-editor/, generated by firmware/toolchain/console_demo.py and editor_demo.py). Words in components/guide/copy.ts and copy/; old /guide#flash-style anchors land on /guide/play (legacy.ts)
/inside/[build] One community build's page
/pattern-lab The pattern workspace — see below
/community/** The community: feed, pattern pages (/p/[id]), profiles (/u/[username]), decks (/d/[id], /decks), the Workshop (/workshop/[code]), territories and the atlas, notifications, featured, reports. Renders a pointer to the community host unless COMMUNITY_ENABLED=1
/editions The edition shelf: every firmware you can put on a panel, official and community, with one-click install. /variants (the URL until 2026-09, baked into shipped console pages) redirects here permanently
/features The catalogue: every feature on its own — what it needs, which firmware carries it, and a reel of it running (Instagram embeds by permalink, lazy). The other axis of the shelf; hand-curated in features-data.ts. A feature that is not on the shelf can carry a frozen try-out image (tryOut), installed the way the shelf installs
/update The device's firmware-update handoff: the browser downloads an image and POSTs it to the panel over the LAN, because the panel cannot fetch over TLS
/flash/* (static files; /flash itself redirects to /guide/play#flash, next.config.ts) public/flash/manifest.json + the images in public/flash/bin/ — what the esp-web-tools Flash Patternflow button installs (on /pattern, PatternPanel.tsx, and in the Play guide, components/guide/Extras.tsx). Only the currently served images are committed
/journal · /journal/[slug] (+ /en) Bilingual (ko/en) MDX journal with per-article OG image generation
/roadmap Roadmap rendered from roadmap-data.ts plus GitHub issue progress via /api/roadmap
/compliance Printed on the box — never move, redirect or noindex it
/terms · /business · /contact Static pages
/api/community/** The community's JSON API (30-odd routes: patterns, decks, posts, comments, builds, workshop, moderation…). Every route guards on communityEnabled(), and src/proxy.ts closes the whole prefix with a 404 on a deployment without the community, so a route that forgets is still closed
/api/auth/[...all] Better Auth
/api/variant-bin/[...path] Serves hosted edition images from VARIANT_BIN_DIR with the CORS headers /update needs
/api/roadmap Open issues + sub-issue progress (10 min revalidate; optional GITHUB_TOKEN)
feed.xml · sitemap.ts · robots.ts Feeds and SEO plumbing
/r/* Campaign short links, rewritten to /; PostHog tags the visit with the route's UTM properties (src/lib/campaignRoutes.ts, next.config.ts, src/lib/posthogEvents.ts)

src/proxy.ts (Next.js proxy/middleware) routes journal visitors between ko/en using the pf-journal-lang cookie, and answers 404 for /api/community/* wherever COMMUNITY_ENABLED is not set.

Landing page composition

HomeView (src/components/HomeView.tsx)
├── ViewerPanel (components/3d/)         ← sticky 3D panel
│   ├── HeroScene.tsx                    ← Canvas, the picked case's GLB (public/cases/<id>/model.glb), LED ShaderMaterial
│   ├── caseModels.ts + cases/           ← one model per case on the Build panel's switch: URL, node names, exploded view, finishes. The GLBs are built by tools/case-models
│   ├── LedMatrixTexture.ts              ← renders JS patterns into a texture for the 3D LED mesh
│   └── patterns/                        ← GLSL pattern ports (common.ts = shared vertex shader + registry)
└── RightPanel (components/sections/)    ← tabbed content panel
    ├── Hero.tsx / Deck.tsx              ← hero copy + panel open/close orchestration
    ├── BuildPanel.tsx                   ← build paths, the case switch, flasher entry
    ├── PatternPanel.tsx                 ← preset browser + code editor; every CTA leads to /pattern-lab
    ├── InsidePanel.tsx + InsideGlobe/   ← concept content + community builds globe
    └── Sponsor.tsx

Cross-component landing state (active tab, virtual knob values, bloom toggle, the case picked on the Build panel and its finish) lives in the Zustand store src/store/useAppStore.ts.

Pattern system

Mirrors the firmware; the same JS runs in the browser, in the community's sandbox iframe and in the lab's worker.

  • src/lib/presets/ — JS pattern sources, one file per pattern. Source of truth for the firmware preset .h files (firmware/patternflow/presets/; the web has more presets than the firmware — the cut list is in firmware/patternflow/README.md). _TEMPLATE.ts is the skeleton; index.ts the registry.
  • src/lib/pattern/harness.ts — runs pattern JS on a 128×64 virtual matrix with 4 virtual encoders (24 detents/turn), matching device semantics (knobDeltas, btnPressed/btnHeld); also the OKLab/OKLCH colour ramp math and the LUT builders. public/pattern-sandbox.html is built from it — scripts/build-sandbox.ts bundles src/sandbox/sandbox.ts (the canvas, the live loop, the postMessage protocol) with the runtime it imports, and check:sandbox-sync fails when the committed page is not that build. It was a hand-kept ES5 port until 3.9.4, and it drifted (next.config.ts records one such bug; another was 20 detents a turn against the lab's 24). The 24 now comes from src/lib/shared/constants.ts, and so do the panel size and the knob count. npm run gen:shared writes that file from shared/constants.json at the repository root, the same source as the firmware's core_shared.h.
  • Annotations — a pattern's frame, ramp, knob ranges and layer stack travel inside its code as one-line comments so they survive every copy: @matrix (src/lib/pattern/matrix.ts), @ramp (src/lib/pattern/ramp.ts), @knobs (src/lib/pattern/knobs.ts — the lab's knob state in src/lib/lab/annotations.ts and the community's card reader in src/lib/community/knobs.ts both build on it), @stack (src/lib/lab/stackShare.ts).
  • src/lib/pattern/controls.ts — knob-scale constants shared with the firmware.
  • src/lib/pattern/share.ts — licence header + attribution footer, injected at export time only.
  • src/lib/pattern/packs.ts — the Basics pack that ships with the firmware (public/packs/, built by firmware/toolchain/make_pack.py).
  • src/lib/ai/gemini.ts — bring-your-own-key Gemini generation. The key lives in localStorage and calls go straight from the browser to Google; no server proxy, no bundled key.

Pattern Lab (src/app/pattern-lab/ + src/lib/lab/)

A layered, dockable workspace: nine panels over a layer-stack project. A panel is one entry in panels/registry.tsx (id, title, component, where it docks by default) plus its component file — the shell derives the dockview component map, the Panels menu and the default layout from that list, and the four heavy editors load on demand. The project is one Zustand store, src/lib/lab/store.ts, which the render engine (engine.ts) reads imperatively each frame and the panels subscribe to; its actions are composed from one slice per concern under src/lib/lab/store/ (project, layers, knobs, ramp, director, gallery), with the helpers they share in store/shared.ts. Everything the lab keeps in localStorage is listed in src/lib/lab/persist.ts, with the read/write helpers every persistence site uses. Pixel undo stacks are src/lib/lab/pixelHistory.ts.

Panel Where What
Preview, Layers, Code, Knobs, Color Ramp panels/ The editor proper. Layers are code (JS) or pixel (RGBA buffer, edited in place and versioned by rev); ramps are per layer, knobs are one shared set
Pixel panels/PixelPanel.tsx + panels/pixel/ + lib/lab/pixelTools.ts Pixel-art editor. The panel composes one hook per concern (usePixelToolState, usePixelSelection, usePixelViewport, usePixelCanvases, usePixelPointer, usePixelKeyboard) and two toolbar components; pixelToCode.ts turns a sprite into standalone pattern code
Gallery panels/GalleryPanel.tsx Gemini variant generation queue
Graphic Export panels/CapturePanel.tsx + lib/lab/capture/ Stills and clips at print sizes, rendered by a second engine in a Web Worker (WebCodecs + mediabunny, all client-side), plus a GLSL shader twin for posters. probe.ts decides whether a pattern can be re-rendered at another size or must be upscaled
Director panels/DirectorPanel.tsx + panels/director/ + lib/lab/director/ Knob automation over time — keyframes, bezier eases — baked to .pfs show files (src/lib/pattern/pfst.ts is the codec) and exported as MIDI. The panel composes useTransportView, useTimelineZoom, useShowEditing, the file helpers and two bars; the lanes are DirectorLanes.tsx

Each panel owns its stylesheet (panels/<Panel>.module.css); PatternLab.module.css keeps only what two or more files draw with, and LabPanels.module.css the dock chrome. The lab reaches the community through exactly one file, community.ts (the publish and send modals, the configured switches, the handoff, the .pfs reader) — ESLint refuses any other import of @/components/community or @/lib/community from under the lab.

Exports flatten the visible stack to one standalone pattern (flatten.ts); hExport.ts writes the C++ header; HardwareModal.tsx sends it to the build service or the panel. Persistence is localStorage: the project (serialize.ts, patternflow_lab_project_v2), the dock layout, a ring of parked sessions (sessions.ts), capture settings and shaders under their own keys, the generated gallery (patternflow_lab_gallery_v1, still the live key, read and written by src/lib/lab/legacyDraft.ts), and the draft and ramp keys the lab stopped writing at Pattern Lab v2 on 27 July 2026, which migrateLegacyDraft in serialize.ts lifts into a project when none restores.

Community (src/lib/community/ + src/components/community/)

SQLite via Drizzle (server/schema.ts; migrations in web/drizzle/, applied on first open by server/db.ts), Better Auth (server/auth.ts), reads in server/queries/ — one file per domain (patterns, decks, workshop, moderation, users, notifications, plus the shared sub-selects), re-exported by server/queries.ts — attachments and thumbnails on disk under communityDataDir() (default web/data/, gitignored — set COMMUNITY_DB_PATH on a real deployment). lib/community/server/ is the server side — everything that touches SQLite, the filesystem or a session — and ESLint refuses a value import of it from components/ or a *Client.tsx (type imports are fine; a value crosses through an API route). The rest of lib/community/ is either "use client" (auth-client, deviceHost, download, handoff, knobs, thumbs) or pure and usable anywhere (validate, license, deck, workshop, cors, …).

Patterns render in a sandboxed iframe (public/pattern-sandbox.html, driven by components/community/SandboxPreview.tsx). Pattern Lab hands off to the community through handoff.ts (fork vs. edit-in-place), and publishes through components/community/PublishModal.tsx.

The firmware build service (/api/community/builds + scripts/build-worker.ts + src/lib/firmware/moduleRunner.ts) compiles a submitted header into a .pfm module with the firmware toolchain. It is gated behind BUILD_ENABLED and off by default: compiling submitted C++ is arbitrary code execution at compile time. The community host runs it inside a systemd sandbox (no network, read-only system, three writable paths, resource limits) — the drop-in, what it does not cover, and the two steps left are in docs/SERVICES.md.

A published header is compiled once and kept beside the code in module_cache (server/moduleCache.ts): one row per (sha256 of the normalised header, toolchain revision), holding the .pfm bytes and its sidecar. The worker is the only thing that compiles and the only writer of a module's content (the web only enqueues bakes and bumps used_at, at most once a day, for retention); scripts/build-worker.ts is just its loop, and the work itself is server/buildJobs.ts with the compiler injected, so check:modcache drives it with a fake one. Jobs come in two kinds: send (somebody's arbitrary code via POST /builds → a zip artifact, per-user limits, taken first) and bake (one stored header → a cache row, charged to the header's owner, outside every per-user limit). Header writes queue a bake, a download miss queues one, and an idle worker warms whatever published header is missing. The toolchain revision (src/lib/firmware/builderRev.ts — the ABI and shared headers, the loader's symbol table, module.ld, both toolchain scripts, the compiler version) is computed by the worker only and published to build_meta, because a request must never read firmware/. The web process assembles every download from cache rows with no queue: /api/community/patterns/[id]/zip (one module, no catalog.txt) and /api/community/decks/[id]/zip (server/deckZip.ts — every includable slot, catalog.txt in running order, the .pfs; a slot that cannot go in is skipped and reported, never fatal). Both answer 202 while a first bake runs, ?status=1 for polling, and write the sidecar's author, licence and URL from the database (server/modulePack.ts). Only failures with positive evidence against the code (an error located in the pattern's own source, a symbol the device loader cannot resolve) are cached, and only after the worker's canary — a curated preset — still compiles; a timeout, a broken sandbox or a missing toolchain is retried with backoff, and a compiler that cannot be found or will not answer --version publishes no revision at all. Bakes are capped at four in flight per owner, a bake whose header text has left the site is skipped, and only one worker may run per firmware checkout (build_module.py shares firmware/modules/.build/<slug>/ between compiles).

Content pipeline

  • Section copy — content/build.md, content/pattern.md, content/inside.md, parsed with gray-matter via src/lib/content.ts.
  • Journal — content/journal/<slug>.mdx (Korean) + <slug>.en.mdx (English), loaded by src/lib/journal.ts, rendered through components/journal/. Images in public/journal/<slug>/.
  • Static data in code — src/app/roadmap/roadmap-data.ts, src/app/editions/editions-data.ts (the shelf), src/app/features/features-data.ts (the feature catalogue and its try-out images), src/components/sections/build-cases-data.ts (the Build panel's cases), src/components/sections/InsideGlobe/builds.ts (the build map), src/lib/atlas/data.ts (the pattern-exploration map). Hand-curated; edit the file.

Analytics

PostHog (src/providers/PostHogProvider.tsx, event helpers in src/lib/posthogEvents.ts) plus Vercel Analytics / Speed Insights. All no-ops when the env vars are absent.

Environment variables

web/.env.example is the complete, commented list. The site builds and runs with none set; each group switches on one deployment concern:

Group Vars
Analytics NEXT_PUBLIC_POSTHOG_KEY, NEXT_PUBLIC_POSTHOG_HOST
Roadmap GITHUB_TOKEN
Community host COMMUNITY_ENABLED, BETTER_AUTH_SECRET, BETTER_AUTH_URL, COMMUNITY_DB_PATH, COMMUNITY_ALLOWED_ORIGINS, COMMUNITY_ADMIN_USERNAMES
Every deployment with a community NEXT_PUBLIC_COMMUNITY_URL
Build service BUILD_ENABLED, NEXT_PUBLIC_BUILD_ENABLED, FIRMWARE_SRC_DIR, BUILD_WORK_DIR, BUILD_ARTIFACT_DIR, ARDUINO_CLI_PATH, BUILD_FQBN, PYTHON_PATH, WORKER_ID, BUILD_POLL_MS
Edition images VARIANT_BIN_DIR
Local dev NEXT_DEV_ORIGINS

Conventions

  • Styling: Tailwind v4 (@import "tailwindcss" in globals.css) + global custom CSS; larger components use CSS Modules (*.module.css).
  • Layering: lib/ is the bottom. app/ and components/ import from it; it never imports from them (ESLint enforces it). A type a component and a serializer share belongs in lib/ — lib/community/cardTypes.ts is the precedent.
  • Adding a preset: add the JS file under src/lib/presets/, register it in index.ts, then generate the firmware .h with the Pattern Lab's To hardware → Copy the conversion prompt flow.
  • Tests are the check:* scripts in package.json — bespoke smoke suites under scripts/*-smoke.ts, one per subsystem — plus check:panels, a Vitest + jsdom + Testing Library harness (vitest.config.ts, test/setup.ts) for the Pattern Lab panels, where pointer gestures, keyboard shortcuts, undo and the Director's keyframes need a DOM. Those tests live beside the panels (panels/*.test.tsx) and read the store's buffers rather than the canvas, which jsdom does not have. npm run check:ci runs every one that needs nothing a CI runner lacks; check:module wants the Xtensa toolchain and stays local. The drift checks are check:* scripts too, one file each in scripts/drift/. Each compares two things the tree keeps by hand, the BOM and the build guide's parts table for one, and check:drift, the last step of check:ci, runs them all. Five hold a contract against a running panel instead. check:rest-vectors, check:osc-vectors, check:midi-vectors, check:mqtt-vectors and check:audio-vectors replay the vectors beside each spec in docs/ against the virtual panel in tools/virtual-panel/, which is lib/guide/'s DeviceSim behind the firmware's ports. A vector also fails when the spec sentence it quotes is gone. check:virtual-panel is not a vector suite: it runs every preset on that panel. A file there whose name starts with _ is a library the suites share. _broker.ts among them is a test broker, because a Patternflow is a broker's client and the MQTT suite has to bring one.
  • CI (.github/workflows/web-ci.yml) lints, typechecks (npm run typecheck, which reads tools/virtual-panel/ too), builds and runs check:ci on every PR touching web/. .github/workflows/drift.yml runs check:drift on every PR, whatever it touches, and check:human-lock against the PR's base. It sets PF_DRIFT_STRICT=1: at a desk a check skips a side it cannot read (a tag the clone lacks, no pdftotext), and in CI that skip fails.
  • Deployment weight: Vercel meters storage per retained deployment — every one counts at its full output size, static files and function bundles both, with nothing shared between deployments — so size × push count is what the Hobby allowance (10 GB each) is spent on. Only main deploys: vercel.json turns deployments off for dev, which was three quarters of them and whose previews nobody opened. public/ is ~90 % of a deployment's static weight (journal and build-map photos). Server functions must carry none of it: outputFileTracingExcludes in next.config.ts keeps public/ out of every function, and a build that prints "Dynamic filesystem access causes tracing of the whole project" has found a new way in — the comment there has the history.