From 4cb9126bf1c207345f732004d93dcc8947bcfad0 Mon Sep 17 00:00:00 2001 From: Thomas Jung Date: Sat, 8 Aug 2026 17:03:18 -0400 Subject: [PATCH 01/12] docs(#1520): design spec for client-side generative bg + cartoonify --- .../2026-08-08-selfie-generative-bg-design.md | 195 ++++++++++++++++++ 1 file changed, 195 insertions(+) create mode 100644 docs/superpowers/specs/2026-08-08-selfie-generative-bg-design.md diff --git a/docs/superpowers/specs/2026-08-08-selfie-generative-bg-design.md b/docs/superpowers/specs/2026-08-08-selfie-generative-bg-design.md new file mode 100644 index 00000000..c7be484a --- /dev/null +++ b/docs/superpowers/specs/2026-08-08-selfie-generative-bg-design.md @@ -0,0 +1,195 @@ +# Selfie: AI generative background + style transfer (Tier 3, #1520) — Design + +**Issue:** #1520 (Tier 3 stretch in epic #1512) · **Date:** 2026-08-08 · **Status:** approved for planning + +## Summary + +Add two client-side "generative" capabilities to the Devtoberfest selfie composer: + +1. **Themed background swap** — lift the person off their real background (reusing the + Tier-1 imgly cutout) and drop them onto a curated themed scene (pumpkin patch, TechEd + stage, terminal/code, autumn gradient, starfield). "Generative" = curated vector art, + not a runtime model. +2. **Cartoonify style transfer** — a real neural style-transfer preset (AnimeGANv2 + `face_paint_512_v2`, ONNX, run in-browser via `onnxruntime-web`) applied to the whole + composite at export. + +Both run **entirely in the browser**. No photo ever leaves the device, so the selfie's +existing privacy promise — *"Your photo is processed entirely on your device — it never +leaves your browser"* (`Selfie.vue`) — remains literally true and is left unchanged. + +## Decision record (satisfies #1520 AC #1) + +- **Hosted vs client-side:** **client-side**. AI Core infra exists (`@cap-js/ai`, + `tutorials-aicore` binding) but a server round-trip would break the "never leaves your + browser" promise, carry per-request cost, and expose the anonymous flow to abuse. None + of that is acceptable for a fun community tool. **Cost ceiling: $0** — no hosted + inference. +- **Why client-side is viable here:** the tool already runs `onnxruntime-web` + WASM in + the browser for imgly background removal (`segment.ts`, self-hosted at `/vendor/imgly/`, + ~76 MB, lazy-loaded). The approuter CSP already permits the eval tokens onnx needs + (`'wasm-unsafe-eval'` + `'unsafe-eval'`, established in #1546). Cartoonify reuses that + proven pattern with its own small model. + +## Architecture — Approach A: two independent layers + +Each feature is a bounded unit reusing an existing seam. The Konva stage internals and the +six existing effect presets (#1516) are not modified in their behavior. + +``` +[bgLayer: themed scene] ← new, bottom-most Konva layer (below cutout) +[cutoutLayer: person] ← existing +[frameLayer] ← existing (advocate frame) +[overlaysLayer] ← existing (stickers / emoji / caption), topmost + + export: stage.toCanvas() + → await applyEffectAsync(composite, effect) // cartoon = ONNX pass; else sync CSS bake + → paintPolaroid(composite, border) // if bordered + → toBlob('image/png') +``` + +No network egress at any step. Model + WASM are same-origin static assets; inference is +local WebGPU→wasm. + +## Components + +### 1. Themed background swap + +**`hugo-apps/src/selfie/backgrounds.ts`** (new) — mirrors `stickers.ts`. +- `interface BackgroundDef { id: string; label: string; file: string }` +- `BACKGROUNDS: BackgroundDef[]` — the scene list. +- `BACKGROUND_IDS: string[]` — picker order, `'none'` first. +- `backgroundUrl(imgBase, file): string` → `${imgBase}/backgrounds/${file}.png`. + +**`scripts/gen-backgrounds.mjs`** (new) — hand-authored SVG → 1024×1024 (or stage-aspect) +transparent/opaque PNGs via `sharp`, same pipeline and Devtoberfest palette as +`scripts/gen-stickers.mjs`. Output → `hugo/static/images/devtoberfest/selfie/backgrounds/`. +Scenes: `pumpkin-patch`, `teched-stage`, `terminal`, `autumn-gradient`, `starfield`. + +**`hugo-apps/src/selfie/BackgroundPicker.vue`** (new) — a scene picker mirroring +`EffectPicker.vue`/`StickerPicker.vue`. Renders a `None` + one thumbnail per scene, marks +the active one, emits `pick(id)`. + +**`compose.ts` — new stage method `setBackground(img: HTMLImageElement | null): void`.** +- A new `bgLayer = new Konva.Layer()` is added to the stage **first** (bottom-most), before + `cutoutLayer`, in `buildStage`. This holds one `Konva.Image` sized to fill the stage + (`x:0, y:0, width:stageW, height:stageH`), `listening(false)`. +- `setBackground(img)` adds/replaces that node and `bgLayer.batchDraw()`. +- `setBackground(null)` removes the node (clears the scene). +- Added to the `SelfieStage` interface. + +**`Composer.vue` wiring.** +- Renders `BackgroundPicker`, tracks `backgroundId` ref. +- On pick: loads the scene image (`blobToImage`-style `Image` load of the static URL), + calls `stage.setBackground(img)`; `none` → `setBackground(null)`. +- **Picking a non-none background forces `removeBg` on**: emits `update:removeBg` true if + not already on, so the person is cut out. If no cutout is cached, reuses the existing + on-demand `segment` emit + "segmenting…" state (same path as toggling removeBg on today). +- Live preview: the themed scene is a real Konva layer, so it shows live in the stage with + no CSS approximation needed (unlike effects). + +### 2. Cartoonify style transfer + +**`hugo-apps/src/selfie/stylize.ts`** (new). +- `export async function cartoonify(canvas: HTMLCanvasElement): Promise` +- Lazy `await import('onnxruntime-web')` (new direct hugo-apps dependency) — never at page + load, mirroring `segment.ts`. +- Creates an inference session from the self-hosted model at `/vendor/animegan/` (WASM/model + path set on `ort.env`, no CDN). Prefers the `webgpu` execution provider, falls back to + `wasm`. +- Pipeline: draw `canvas` into a 512×512 offscreen canvas (contain-fit), read pixels → + normalized `Float32Array` NCHW tensor → `session.run` → denormalize output tensor → put + onto a 512² canvas → upscale (`drawImage`) onto a copy of the input dimensions → return. +- Fail-soft: **any** failure (import throw, fetch fail, no EP available, inference throw) + → returns the **input canvas unchanged**. + +**`scripts/vendor-animegan.mjs`** (new) — downloads + self-hosts the AnimeGANv2 +`face_paint_512_v2` ONNX model to `hugo/static/vendor/animegan/`, plus a +`.animegan-vendored-version` sentinel for idempotency. Same discipline as +`vendor-imgly.cjs`: no CDN at runtime, structural guard, wiped + re-fetched on version +drift. Wired into the `setup`/build scripts alongside `vendor:imgly`. + +**License gate (blocking, in the plan):** the AnimeGANv2 model binary is **not committed** +until a plan step verifies its license is permissive (MIT/Apache/CC-family) and records the +source URL + license in the vendor script header. If the pinned model's license does not +clear, the fallback is another small face-oriented cartoon ONNX with a clear permissive +license, or — last resort — ship the background-swap half and land cartoonify in a +follow-up. The vendoring step gates the commit either way. + +**`effects.ts` — async effect path.** +- `EffectId` gains `'cartoon'`. `EFFECT_IDS` appends `'cartoon'`. +- The `cartoon` entry has `label: 'Cartoon'`, an empty `preview` (no CSS approximation — + cartoonify only materializes at export, consistent with the "preview is an approximation" + contract already documented in the file header), and **no sync `apply`** (its bake is async). +- New `export async function applyEffectAsync(canvas, id): Promise`: + - `id === 'cartoon'` → `try { return await cartoonify(canvas) } catch { return canvas }`. + - otherwise → `return applyEffect(canvas, id)` (the existing sync dispatcher). + - The sync `applyEffect` is unchanged and never routes `'cartoon'` (returns input). + +**`compose.ts` — `exportPng`.** +- `needsCanvas` also true when `effect === 'cartoon'`. +- The single `composite = applyEffect(composite, effect)` line becomes + `composite = await applyEffectAsync(composite, effect)`. The surrounding Promise executor + becomes `async` (or the pixel-pass branch is extracted to an async helper). Bake order + (effect before border) is preserved. + +**`Composer.vue` — UX during inference.** +- The cartoon picker button shows a spinner while inference runs; the Export button is + disabled during the export bake (already async). +- On cartoonify failure a non-blocking note ("couldn't apply the cartoon effect — exported + without it") is shown; export still completes with the un-stylized composite. + +## Privacy + +Unchanged. `Selfie.vue`'s privacy note stays. No consent UX, no server upload, no reword — +this is the entire rationale for choosing client-side over hosted. + +## Error handling (fail-soft — matches Tier 1/2 posture) + +| Failure | Behavior | +|---|---| +| Background image 404 / decode fail | `setBackground` logs, scene stays empty, compositing proceeds | +| `onnxruntime-web` import fail | `cartoonify` returns input canvas; export un-stylized | +| Model fetch fail / WebGPU+wasm both unavailable | `cartoonify` returns input canvas | +| Inference throws | `cartoonify` returns input canvas | +| Cartoonify slow (seconds) | spinner on effect button; Export disabled during bake | + +A bad background or a failed cartoonify never crashes the island and never blocks export. + +## Testing + +- `backgrounds.test.ts` — def-list shape, `backgroundUrl` builder, `none` handling. +- `BackgroundPicker.test.ts` — renders scenes, marks active, emits `pick`, forces removeBg. +- `stylize.test.ts` — `cartoonify` fail-soft paths with a mocked `onnxruntime-web` + (import throws → input; session.run throws → input); dimension round-trip with a fake + session returning a known tensor. +- `effects.test.ts` — extend: `applyEffectAsync('cartoon')` calls `cartoonify` and is + fail-soft; `applyEffectAsync` delegates sync presets to `applyEffect`; sync `applyEffect` + ignores `'cartoon'`. +- `compose.test.ts` — `setBackground` adds/replaces/clears the bottom bgLayer node; export + awaits `applyEffectAsync`; cartoon export takes the canvas path. +- `Composer.test.ts` — background picker wiring; picking a scene forces `update:removeBg`; + picking cartoon forwards `effect: 'cartoon'` to `exportPng`; failure shows the note. + +All new + existing selfie unit tests pass; `vite build` clean. Run from repo root: +`npm test -- --project unit hugo-apps/src/selfie`. + +## Global constraints + +- **Client-side only** — no network egress; cost ceiling $0. +- **Fail-soft everywhere** — no failure path crashes the island or blocks export. +- **Self-host all assets** — no CDN; CSP-clean (reuses the imgly `'wasm-unsafe-eval'` + + `'unsafe-eval'` posture, #1546). +- **Lazy-load** `onnxruntime-web` + the AnimeGAN model — never on page load (imgly precedent). +- **Devtoberfest palette** for all generated background art (OG `#e8791a`, DK `#2b1a0f`). +- **Konva stage never mutated by effects** — effects bake on the exported canvas only; + the themed background is a real stage layer (that IS the point of the feature). +- **License gate** — the model binary is committed only after its permissive license is + verified and recorded. + +## Out of scope + +- Hosted/server-side generation (rejected — see decision record). +- Text-prompt "imagine any background" generation (needs a hosted model; violates $0 + privacy). +- More than one style-transfer preset (one cartoonify preset now; others are follow-ups). +- Photographic (raster) themed backgrounds (vector art now; real art can swap in later). From cf61763a04f77d44e49450f99c7e34de12fd418e Mon Sep 17 00:00:00 2001 From: Thomas Jung Date: Sat, 8 Aug 2026 17:14:25 -0400 Subject: [PATCH 02/12] docs(#1520): implementation plan for selfie generative background + cartoonify --- .../plans/2026-08-08-selfie-generative-bg.md | 1092 +++++++++++++++++ 1 file changed, 1092 insertions(+) create mode 100644 docs/superpowers/plans/2026-08-08-selfie-generative-bg.md diff --git a/docs/superpowers/plans/2026-08-08-selfie-generative-bg.md b/docs/superpowers/plans/2026-08-08-selfie-generative-bg.md new file mode 100644 index 00000000..3631cec0 --- /dev/null +++ b/docs/superpowers/plans/2026-08-08-selfie-generative-bg.md @@ -0,0 +1,1092 @@ +# Selfie Generative Background + Cartoonify Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Add two client-side "generative" capabilities to the Devtoberfest selfie composer — a themed background swap (curated vector scenes behind the cut-out person) and a cartoonify style-transfer effect (AnimeGANv2 ONNX run in-browser) — baked into the exported PNG, with nothing ever leaving the browser. + +**Architecture:** Two independent, bounded units reusing existing seams. The themed background is a new bottom-most Konva layer (`bgLayer`) added in `buildStage`, driven by a `setBackground` stage method and a `BackgroundPicker.vue` mirroring the sticker/effect pickers. Cartoonify is a new async effect: `EffectId` gains `'cartoon'`, a new `applyEffectAsync` awaits an ONNX pass for cartoon and delegates all six existing CSS presets to the unchanged synchronous `applyEffect`; `exportPng` awaits `applyEffectAsync` so the bake order (effect before border) is preserved. Both fail soft to the un-modified composite. + +**Tech Stack:** Vue 3 SFCs (` + +``` + +- [ ] **Step 4: Add the picker styles** + +Append to `hugo-apps/src/selfie/styles.css`: + +```css +/* ---- Background scene picker (#1520) ---- */ +.selfie-bg-controls { display: inline-flex; flex-wrap: wrap; align-items: center; gap: .35rem; } +.selfie-bg-btn { background: var(--sapButton_Background, #fff); color: var(--sapButton_TextColor, #0070f2); border: 1px solid var(--sapButton_BorderColor, #0070f2); } +.selfie-bg-btn.is-active { background: var(--sapButton_Emphasized_Background, #0070f2); color: #fff; } +.selfie-bg-thumb { cursor: pointer; padding: 0; width: 48px; height: 48px; border: 2px solid transparent; border-radius: 8px; overflow: hidden; background: var(--sapTile_Background, #fff); } +.selfie-bg-thumb.is-active { border-color: var(--sapButton_Emphasized_Background, #0070f2); } +.selfie-bg-thumb img { display: block; width: 100%; height: 100%; object-fit: cover; } +``` + +- [ ] **Step 5: Run test to verify it passes** + +Run: `npm test -- --project unit hugo-apps/src/selfie/__tests__/BackgroundPicker.test.ts` +Expected: PASS (4 tests). + +- [ ] **Step 6: Commit** + +```bash +git add hugo-apps/src/selfie/BackgroundPicker.vue hugo-apps/src/selfie/__tests__/BackgroundPicker.test.ts hugo-apps/src/selfie/styles.css +git commit -m "feat(#1520): BackgroundPicker.vue scene picker" +``` + +--- + +## Task 5: Vendor the AnimeGAN model + add onnxruntime-web (license gate) + +**Files:** +- Create: `scripts/vendor-animegan.mjs` +- Modify: `hugo-apps/package.json` (add `onnxruntime-web` dep + `vendor:animegan` script; chain into `vendor`/`setup`) +- Create (build output, committed after license clears): `hugo/static/vendor/animegan/model.onnx` + `.animegan-vendored-version` sentinel + +**Interfaces:** +- Consumes: nothing at code level. +- Produces: model served at `/vendor/animegan/model.onnx`; `onnxruntime-web` installed for Task 6. + +**⚠️ LICENSE GATE — this task blocks committing the binary until the license is verified.** + +- [ ] **Step 1: Verify the model license (BLOCKING)** + +Research the AnimeGANv2 `face_paint_512_v2` ONNX export and confirm a permissive license (MIT / Apache-2.0 / CC-BY family). The commonly-used `bryandlee/animegan2-pytorch` weights are **MIT**; several ONNX re-exports carry it forward. Record the exact source URL, the license name, and the SHA-256 of the downloaded file. + +**Decision point:** +- License clears (MIT/Apache/CC) → proceed to Step 2 with that URL. +- License does NOT clear → do NOT commit a binary. Substitute another small (<25 MB), 512×512-ish, face-oriented style-transfer ONNX with a verified permissive license. If none is found, STOP and report BLOCKED: the background-swap half (Tasks 1-4) ships alone and cartoonify (Tasks 6-7) becomes a documented follow-up. Do not proceed to Task 6 without a licensed model on disk. + +- [ ] **Step 2: Add onnxruntime-web + the vendor script to package.json** + +In `hugo-apps/package.json`, add to `dependencies` (pin a current 1.x): `"onnxruntime-web": "^1.20.0"`. Add to `scripts`: `"vendor:animegan": "node ../scripts/vendor-animegan.mjs"`. Chain it wherever `vendor:imgly` is invoked (the `setup`/`vendor` script) so a fresh worktree fetches both. + +- [ ] **Step 3: Write the vendor script** + +Create `scripts/vendor-animegan.mjs` following the `scripts/vendor-imgly.cjs` discipline (header documenting source URL + license + SHA; idempotent via a `.animegan-vendored-version` sentinel; wipe-and-refetch on version drift; no CDN reference left in runtime code). Concretely: + +```js +// Vendors the AnimeGANv2 face_paint_512_v2 ONNX model for self-hosting (#1520). +// +// Source: +// License: (verified permissive before commit) +// SHA-256: +// +// Runtime never fetches from a CDN — the model is served same-origin at +// /vendor/animegan/model.onnx (approuter CSP), exactly like the imgly assets. +// +// Idempotency: a .animegan-vendored-version sentinel records the model version; +// a version change (or a missing sentinel) wipes and re-fetches. + +import { createWriteStream } from 'fs' +import { mkdir, readFile, writeFile, rm } from 'fs/promises' +import { createHash } from 'crypto' +import path from 'path' + +const MODEL_URL = '' +const EXPECTED_SHA256 = '' +const VERSION = 'face_paint_512_v2' +const DEST_DIR = path.resolve(process.cwd(), 'hugo/static/vendor/animegan') +const MODEL_PATH = path.join(DEST_DIR, 'model.onnx') +const SENTINEL = path.join(DEST_DIR, '.animegan-vendored-version') + +async function sentinelMatches() { + try { return (await readFile(SENTINEL, 'utf8')).trim() === VERSION } catch { return false } +} + +async function main() { + if (await sentinelMatches()) { console.log('animegan: up to date'); return } + await rm(DEST_DIR, { recursive: true, force: true }) + await mkdir(DEST_DIR, { recursive: true }) + const res = await fetch(MODEL_URL) + if (!res.ok) throw new Error(`animegan fetch failed: ${res.status}`) + const buf = Buffer.from(await res.arrayBuffer()) + const sha = createHash('sha256').update(buf).digest('hex') + if (EXPECTED_SHA256 && sha !== EXPECTED_SHA256) { + throw new Error(`animegan SHA mismatch: got ${sha}`) + } + await writeFile(MODEL_PATH, buf) + await writeFile(SENTINEL, VERSION + '\n') + console.log(`animegan: vendored model.onnx (${(buf.length / 1e6).toFixed(1)} MB)`) +} +main().catch((e) => { console.error(e); process.exit(1) }) +``` + +Replace the three `` placeholders with the real values from Step 1. + +- [ ] **Step 4: Install + run the vendor script** + +Run (from the repo root): +```bash +cd hugo-apps && npm install && cd .. +node scripts/vendor-animegan.mjs +ls -l hugo/static/vendor/animegan/ +``` +Expected: `model.onnx` present (single-digit-to-~25 MB) + `.animegan-vendored-version`. + +- [ ] **Step 5: Commit** + +```bash +git add scripts/vendor-animegan.mjs hugo-apps/package.json hugo-apps/package-lock.json hugo/static/vendor/animegan/ +git commit -m "feat(#1520): vendor AnimeGANv2 model + onnxruntime-web (license verified)" +``` + +--- + +## Task 6: `stylize.ts` cartoonify + async effect path + +**Files:** +- Create: `hugo-apps/src/selfie/stylize.ts` +- Modify: `hugo-apps/src/selfie/effects.ts` (add `'cartoon'` to `EffectId` line 6 + `EFFECT_IDS` line 118; add `cartoon` table entry; add `applyEffectAsync`) +- Test: `hugo-apps/src/selfie/__tests__/stylize.test.ts` +- Modify test: `hugo-apps/src/selfie/__tests__/effects.test.ts` + +**Interfaces:** +- Consumes: `applyEffect` (existing sync dispatcher). +- Produces: + - `stylize.ts`: `export async function cartoonify(canvas: HTMLCanvasElement): Promise` — returns a stylized canvas, or the **input canvas unchanged** on any failure. + - `effects.ts`: `EffectId` includes `'cartoon'`; `EFFECT_IDS` ends with `'cartoon'`; `export async function applyEffectAsync(canvas: HTMLCanvasElement, id: EffectId): Promise`. + +- [ ] **Step 1: Write the failing stylize test** + +Create `hugo-apps/src/selfie/__tests__/stylize.test.ts`: + +```ts +// @vitest-environment happy-dom +import { describe, it, expect, vi, beforeEach } from 'vitest' + +// Mock onnxruntime-web so no real WASM/model loads in unit tests. +const runMock = vi.fn() +const createMock = vi.fn(async () => ({ run: runMock })) +vi.mock('onnxruntime-web', () => ({ + InferenceSession: { create: createMock }, + Tensor: class { constructor(public type: string, public data: unknown, public dims: number[]) {} }, + env: { wasm: {} }, +})) + +import { cartoonify } from '../stylize' + +// A canvas stub whose 2D context yields predictable pixel data. +function fakeCanvas(w = 256, h = 256): HTMLCanvasElement { + const ctx = { + drawImage: vi.fn(), + getImageData: vi.fn(() => ({ data: new Uint8ClampedArray(512 * 512 * 4).fill(128), width: 512, height: 512 })), + putImageData: vi.fn(), + } + return { width: w, height: h, getContext: vi.fn(() => ctx) } as unknown as HTMLCanvasElement +} + +beforeEach(() => { runMock.mockReset(); createMock.mockClear() }) + +describe('cartoonify fail-soft', () => { + it('returns the INPUT canvas unchanged when session creation throws', async () => { + createMock.mockRejectedValueOnce(new Error('no wasm')) + const c = fakeCanvas() + expect(await cartoonify(c)).toBe(c) + }) + + it('returns the INPUT canvas unchanged when inference throws', async () => { + runMock.mockRejectedValueOnce(new Error('run failed')) + const c = fakeCanvas() + expect(await cartoonify(c)).toBe(c) + }) + + it('returns the INPUT canvas unchanged when the 2D context is null', async () => { + const c = { width: 256, height: 256, getContext: vi.fn(() => null) } as unknown as HTMLCanvasElement + expect(await cartoonify(c)).toBe(c) + }) +}) +``` + +Note: the success-path pixel round-trip is hard to assert meaningfully against a mock without pinning the exact tensor layout; the fail-soft paths are the behavioral contract that matters and are fully covered here. If the implementer wants a success assertion, have `runMock` resolve `{ output: new Tensor('float32', new Float32Array(3*512*512).fill(0), [1,3,512,512]) }` and assert the return is a NEW canvas (not `c`) — but keep it only if the tensor key/layout in the implementation is stable. + +- [ ] **Step 2: Run test to verify it fails** + +Run: `npm test -- --project unit hugo-apps/src/selfie/__tests__/stylize.test.ts` +Expected: FAIL — cannot resolve `../stylize`. + +- [ ] **Step 3: Implement stylize.ts** + +Create `hugo-apps/src/selfie/stylize.ts`: + +```ts +// Cartoonify style transfer for the selfie composer (#1520). Runs the +// self-hosted AnimeGANv2 face_paint_512_v2 ONNX model in-browser via +// onnxruntime-web — the same in-browser posture as imgly background removal. +// Lazy-imported so neither the runtime nor the model touch the page-load path. +// Fail-soft: ANY failure returns the input canvas unchanged. + +const MODEL_URL = '/vendor/animegan/model.onnx' +const SIZE = 512 // model's fixed square I/O + +let sessionPromise: Promise | null = null + +async function getSession(ort: typeof import('onnxruntime-web')) { + if (!sessionPromise) { + // Self-hosted WASM binaries live alongside the app bundle (no CDN). + ort.env.wasm.wasmPaths = '/vendor/onnxruntime/' + sessionPromise = ort.InferenceSession.create(MODEL_URL, { + executionProviders: ['webgpu', 'wasm'], + }) + } + return sessionPromise +} + +export async function cartoonify(canvas: HTMLCanvasElement): Promise { + try { + const ort = await import('onnxruntime-web') + const session = (await getSession(ort)) as import('onnxruntime-web').InferenceSession + + // Downscale the composite into a SIZE×SIZE working canvas. + const work = document.createElement('canvas') + work.width = SIZE; work.height = SIZE + const wctx = work.getContext('2d') + if (!wctx) return canvas + wctx.drawImage(canvas, 0, 0, SIZE, SIZE) + const { data } = wctx.getImageData(0, 0, SIZE, SIZE) + + // RGBA uint8 → planar RGB float32 in [-1,1], NCHW. + const chw = new Float32Array(3 * SIZE * SIZE) + const plane = SIZE * SIZE + for (let i = 0; i < plane; i++) { + chw[i] = data[i * 4] / 127.5 - 1 + chw[plane + i] = data[i * 4 + 1] / 127.5 - 1 + chw[2 * plane + i] = data[i * 4 + 2] / 127.5 - 1 + } + const input = new ort.Tensor('float32', chw, [1, 3, SIZE, SIZE]) + const feeds: Record = { [session.inputNames[0]]: input } + const out = await session.run(feeds as never) + const outTensor = out[session.outputNames[0]] as import('onnxruntime-web').Tensor + const od = outTensor.data as Float32Array + + // Planar RGB float32 [-1,1] → RGBA uint8. + const rgba = new Uint8ClampedArray(plane * 4) + for (let i = 0; i < plane; i++) { + rgba[i * 4] = (od[i] + 1) * 127.5 + rgba[i * 4 + 1] = (od[plane + i] + 1) * 127.5 + rgba[i * 4 + 2] = (od[2 * plane + i] + 1) * 127.5 + rgba[i * 4 + 3] = 255 + } + wctx.putImageData(new ImageData(rgba, SIZE, SIZE), 0, 0) + + // Upscale the stylized result back onto a canvas of the input's dimensions. + const outCanvas = document.createElement('canvas') + outCanvas.width = canvas.width; outCanvas.height = canvas.height + const octx = outCanvas.getContext('2d') + if (!octx) return canvas + octx.drawImage(work, 0, 0, canvas.width, canvas.height) + return outCanvas + } catch (e) { + console.warn('[selfie] cartoonify failed; exporting without it', e) + return canvas + } +} +``` + +Note: `ort.env.wasm.wasmPaths` points at `/vendor/onnxruntime/`. If Task 5 vendored the ORT WASM binaries elsewhere, set this to the actual served path. If ORT's default packaged WASM works under the approuter CSP without a custom path, delete that line — but verify no CDN fetch happens (Network tab shows same-origin only). + +- [ ] **Step 4: Run stylize test to verify it passes** + +Run: `npm test -- --project unit hugo-apps/src/selfie/__tests__/stylize.test.ts` +Expected: PASS (3 fail-soft tests). + +- [ ] **Step 5: Extend effects.ts for the async cartoon path** + +In `hugo-apps/src/selfie/effects.ts`: + +1. Line 6 — add `'cartoon'`: +```ts +export type EffectId = 'none' | 'duotone' | 'warm' | 'mono' | 'vignette' | 'joule' | 'cartoon' +``` + +2. Add a `cartoon` entry to the `EFFECTS` table (after `joule`). Its sync `apply` returns the input untouched (the real bake is async, in `applyEffectAsync`); its preview is empty (no CSS approximation): +```ts + cartoon: { + label: 'Cartoon', + preview: {}, // no CSS approximation — the ONNX bake only materializes at export + apply: (canvas) => canvas, // sync no-op; async bake lives in applyEffectAsync/cartoonify + }, +``` + +3. Line 118 — append `'cartoon'` to the picker order: +```ts +export const EFFECT_IDS: EffectId[] = ['none', 'duotone', 'warm', 'mono', 'vignette', 'joule', 'cartoon'] +``` + +4. Add the async dispatcher (after `applyEffect`), importing `cartoonify`: +```ts +import { cartoonify } from './stylize' + +// Async effect dispatcher. 'cartoon' runs the in-browser ONNX style transfer; +// every other id delegates to the synchronous applyEffect. Fail-soft: a failed +// cartoonify returns the input canvas (cartoonify already guards internally; the +// try/catch is belt-and-braces, matching applyEffect's outer guard). +export async function applyEffectAsync(canvas: HTMLCanvasElement, id: EffectId): Promise { + if (id === 'cartoon') { + try { return await cartoonify(canvas) } catch { return canvas } + } + return applyEffect(canvas, id) +} +``` + +- [ ] **Step 6: Update the existing effects.test.ts assertions for the new id** + +In `hugo-apps/src/selfie/__tests__/effects.test.ts`, the `EFFECT_IDS` exact-match test (line 36-43) now fails because the list grew. Update it and add async coverage. Replace the `'lists none first…'` test body's array and keep the per-entry `apply` check (cartoon's `apply` is a function too): + +```ts + it('lists none first and exposes every preset in order', () => { + expect(EFFECT_IDS).toEqual(['none', 'duotone', 'warm', 'mono', 'vignette', 'joule', 'cartoon']) + expect(EFFECT_IDS[0]).toBe('none') + for (const id of EFFECT_IDS) { + expect(typeof EFFECTS[id].label).toBe('string') + expect(typeof EFFECTS[id].apply).toBe('function') + } + }) +``` + +Add a new `describe` for `applyEffectAsync` at the end of the file. Mock `../stylize` so no ONNX loads: + +```ts +import { applyEffectAsync } from '../effects' +import { vi as _vi } from 'vitest' + +vi.mock('../stylize', () => ({ cartoonify: vi.fn(async (c: HTMLCanvasElement) => ({ ...c, _cartooned: true } as unknown as HTMLCanvasElement)) })) + +describe('applyEffectAsync', () => { + it('routes cartoon through cartoonify', async () => { + const { cartoonify } = await import('../stylize') + const c = composite() + await applyEffectAsync(c, 'cartoon') + expect(cartoonify).toHaveBeenCalledWith(c) + }) + + it('delegates non-cartoon ids to the sync applyEffect (mono returns via drawImage path)', async () => { + const c = composite() + const out = await applyEffectAsync(c, 'none') + expect(out).toBe(c) // none is a no-op in both paths + }) + + it('fail-soft: returns the input canvas when cartoonify throws', async () => { + const { cartoonify } = await import('../stylize') + ;(cartoonify as unknown as { mockRejectedValueOnce: (e: Error) => void }).mockRejectedValueOnce(new Error('boom')) + const c = composite() + expect(await applyEffectAsync(c, 'cartoon')).toBe(c) + }) +}) +``` + +Note: the `vi.mock('../stylize', …)` call must sit at the top of the file with the other imports (hoisted), not inside the describe. Move it up if vitest complains about hoist ordering. + +- [ ] **Step 7: Run the effects test to verify it passes** + +Run: `npm test -- --project unit hugo-apps/src/selfie/__tests__/effects.test.ts` +Expected: PASS (existing + updated + 3 new async tests). + +- [ ] **Step 8: Commit** + +```bash +git add hugo-apps/src/selfie/stylize.ts hugo-apps/src/selfie/effects.ts hugo-apps/src/selfie/__tests__/stylize.test.ts hugo-apps/src/selfie/__tests__/effects.test.ts +git commit -m "feat(#1520): cartoonify style transfer via async effect path" +``` + +--- + +## Task 7: `exportPng` awaits the async effect path + +**Files:** +- Modify: `hugo-apps/src/selfie/compose.ts` (`exportPng` ~line 128-171) +- Test: `hugo-apps/src/selfie/__tests__/compose.test.ts` + +**Interfaces:** +- Consumes: `applyEffectAsync` from Task 6. +- Produces: `exportPng` bakes `'cartoon'` (and all effects) via the async dispatcher; bake order unchanged (effect before border). + +- [ ] **Step 1: Write the failing test** + +Add to `hugo-apps/src/selfie/__tests__/compose.test.ts` (mock `../effects` `applyEffectAsync` in that file's existing mock setup, or add a `vi.mock('../effects', …)`): + +```ts +describe('exportPng cartoon path', () => { + it('takes the canvas (not fast) path for cartoon and awaits applyEffectAsync', async () => { + // build a stage, call exportPng({ effect: 'cartoon' }); assert applyEffectAsync + // was called with the toCanvas() result and the id 'cartoon', and a Blob resolves. + }) + + it('bakes the effect BEFORE the polaroid border (order preserved)', async () => { + // exportPng({ effect: 'mono', border: {style,name} }): assert applyEffectAsync + // resolves before paintPolaroid is invoked (spy call order). + }) +}) +``` + +Fill the bodies against the file's mock (spy on `applyEffectAsync` and `paintPolaroid`; assert `.mock.invocationCallOrder`). + +- [ ] **Step 2: Run test to verify it fails** + +Run: `npm test -- --project unit hugo-apps/src/selfie/__tests__/compose.test.ts` +Expected: FAIL — cartoon not treated as needing the canvas path / `applyEffectAsync` not called. + +- [ ] **Step 3: Implement in compose.ts** + +1. Update the import (line 5): +```ts +import { applyEffect, applyEffectAsync, type EffectId } from './effects' +``` +(Keep `applyEffect` imported only if still referenced; otherwise import just `applyEffectAsync`.) + +2. In `exportPng`, `needsCanvas` already covers `effect !== 'none'`, which includes `'cartoon'` — no change needed there. Make the pixel-pass branch async. Replace the synchronous `applyEffect` line and its surrounding executor. The current executor is `new Promise((resolve, reject) => { ... })`; change the `if (needsCanvas)` block to run async: + +```ts + if (needsCanvas) { + void (async () => { + try { + let composite = stage.toCanvas() as HTMLCanvasElement + // Effect bakes BEFORE the border so the white matte stays untinted. + if (effect && effect !== 'none') composite = await applyEffectAsync(composite, effect) + const finalCanvas = border ? paintPolaroid(composite, border) : composite + finalCanvas.toBlob((b: Blob | null) => { + restore() + b ? resolve(b) : reject(new Error('export failed')) + }, 'image/png') + } catch (e) { + restore() + reject(e as Error) + } + })() + return + } +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `npm test -- --project unit hugo-apps/src/selfie/__tests__/compose.test.ts` +Expected: PASS (existing + Task 3 setBackground + the 2 new cartoon-path tests). + +- [ ] **Step 5: Commit** + +```bash +git add hugo-apps/src/selfie/compose.ts hugo-apps/src/selfie/__tests__/compose.test.ts +git commit -m "feat(#1520): exportPng awaits applyEffectAsync (cartoon bake)" +``` + +--- + +## Task 8: Wire BackgroundPicker + cartoon UX into Composer.vue + +**Files:** +- Modify: `hugo-apps/src/selfie/Composer.vue` +- Test: `hugo-apps/src/selfie/__tests__/Composer.test.ts` + +**Interfaces:** +- Consumes: `BackgroundPicker.vue` (Task 4), `backgroundUrl`/`BACKGROUNDS` (Task 1), `setBackground` (Task 3), `urlToImage` (already in Composer, line 62-69). +- Produces: background picking (forces `removeBg` on for a scene), cartoon inference spinner, failure note. No new emits beyond the existing set. + +- [ ] **Step 1: Write the failing tests** + +Add to `hugo-apps/src/selfie/__tests__/Composer.test.ts`. The file's `buildStage` mock (in the `vi.hoisted` block) must gain a `setBackground` spy — add `const setBackground = vi.fn()` there, include it in the resolved stage object, and export it on `h`. + +```ts + it('renders the background picker', async () => { + const w = mount(Composer, { props: { rawPhoto: raw, cutout: cut, removeBg: true, segmenting: false, ...base } }) + await flushPromises() + expect(w.find('[data-testid="bg-none"]').exists()).toBe(true) + expect(w.find('[data-testid="bg-terminal"]').exists()).toBe(true) + }) + + it('picking a scene sets it on the stage and forces removeBg on', async () => { + const w = mount(Composer, { props: { rawPhoto: raw, cutout: cut, removeBg: false, segmenting: false, ...base } }) + await flushPromises() + await w.find('[data-testid="bg-terminal"]').trigger('click') + await flushPromises() + expect(h.setBackground).toHaveBeenCalledTimes(1) // an was loaded + set + expect(w.emitted('update:removeBg')?.[0]?.[0]).toBe(true) // forced on + }) + + it('picking None clears the stage background', async () => { + const w = mount(Composer, { props: { rawPhoto: raw, cutout: cut, removeBg: true, segmenting: false, ...base } }) + await flushPromises() + await w.find('[data-testid="bg-terminal"]').trigger('click'); await flushPromises() + h.setBackground.mockClear() + await w.find('[data-testid="bg-none"]').trigger('click'); await flushPromises() + expect(h.setBackground).toHaveBeenCalledWith(null) + }) + + it('picking cartoon forwards effect: cartoon to exportPng', async () => { + const w = mount(Composer, { props: { rawPhoto: raw, cutout: cut, removeBg: true, segmenting: false, ...base } }) + await flushPromises() + await w.find('[data-testid="effect-cartoon"]').trigger('click') + await w.find('[data-testid="export"]').trigger('click') + await flushPromises() + expect(h.exportPng).toHaveBeenCalledWith({ effect: 'cartoon', border: undefined }) + }) +``` + +Note: the `Image` stub already in `Composer.test.ts`'s `beforeEach` resolves `onload` on the next microtask, so `urlToImage` for the scene resolves in tests — `setBackground` is reached. + +- [ ] **Step 2: Run test to verify it fails** + +Run: `npm test -- --project unit hugo-apps/src/selfie/__tests__/Composer.test.ts` +Expected: FAIL — `bg-none` not found / `setBackground` not called / no `effect-cartoon`. + +- [ ] **Step 3: Implement in Composer.vue** + +1. Imports (after the EffectPicker import, line 9-10): +```ts +import BackgroundPicker from './BackgroundPicker.vue' +import { backgroundUrl, BACKGROUNDS } from './backgrounds' +``` + +2. State (after `effectId`, line 35): +```ts +const backgroundId = ref('none') +const cartoonBusy = ref(false) +const effectNote = ref('') +``` + +3. Handler (after `doExport`, or near the other `on*` handlers): +```ts +async function onPickBackground(id: string) { + backgroundId.value = id + if (!stage) return + if (id === 'none') { stage.setBackground(null); return } + // A themed background is meaningless without the cutout — force removeBg on. + if (!props.removeBg) emit('update:removeBg', true) + const scene = BACKGROUNDS.find((b) => b.id === id) + if (!scene) return + try { + const img = await urlToImage(backgroundUrl(props.imgBase, scene.file)) + stage.setBackground(img) + } catch (e) { + console.warn('[selfie] background load failed', e) + } +} +``` + +4. `doExport` — reflect cartoon busy + surface a note if cartoon silently no-ops is out of scope (cartoonify is fail-soft internally); wrap the export in the busy flag: +```ts +async function doExport() { + if (!stage) return emit('fallback', effectiveBlob()) + effectNote.value = '' + if (effectId.value === 'cartoon') cartoonBusy.value = true + try { + const blob = await stage.exportPng({ + effect: effectId.value, + border: borderEnabled.value ? { style: borderStyle.value, name: borderName.value } : undefined, + }) + emit('export', blob) + } catch { emit('fallback', effectiveBlob()) } + finally { cartoonBusy.value = false } +} +``` + +5. Template — add the `BackgroundPicker` next to the `EffectPicker` (after line 188): +```html + +``` +And a busy hint + note near the export button (before/after the export `