feat(self-host): full Docker Compose self-hosting via a Miniflare runtime - #450
Open
tlvenn wants to merge 5 commits into
Open
feat(self-host): full Docker Compose self-hosting via a Miniflare runtime#450tlvenn wants to merge 5 commits into
tlvenn wants to merge 5 commits into
Conversation
…time Adds a complete, working path for running Maple outside Cloudflare — no Tinybird, no managed platform. apps/api, apps/alerting, and apps/electric-sync are Cloudflare Workers with no plain-HTTP entry point. deploy/workerd/ runs their Wrangler bundles under Miniflare in one container, deriving every binding and cron schedule from each worker's own wrangler.jsonc at boot (so it tracks upstream config with no manual mirroring), reaching Postgres through the Hyperdrive binding and running the drizzle migrations (incl. the Electric publication) on startup. docker-compose.selfhost.yml wires the full stack: postgres + electric (control plane / shape sync), clickhouse + ch-migrate + collector (telemetry), the workerd container, the web SPA, and a Caddy reverse proxy giving one origin (/api -> api worker, /sync -> electric-sync, /* -> SPA). docs/self-hosting.md is the guide; .env.selfhost.example lists the config. Also removes the dead apps/api/Dockerfile (its `bun run start` CMD has no matching script; the api is a Worker) and reverts docker-compose.yml to the postgres+electric dev-DB role its header documents. Validated end-to-end from a clean boot: ClickHouse + Postgres migrations, self-hosted login, and the Electric dashboards shape all succeed through the proxy; a smoke trace reaches the collector. Runtime note: Miniflare is the working path today; celld is the eventual target once it can run the api (TCP/Hyperdrive, cron, KV/Queues), per MapleTechLabs#352.
…tarts Review follow-ups on the Miniflare runtime: - Dockerfile: copy `deploy/workerd/bun.lock` into the runner stage and install `--frozen-lockfile`. Without the lock the image floated on the `^` ranges, so an unchanged commit could build a drizzle-orm whose migrator disagrees with the generated `meta/_journal.json`. - Proxy: strip connection/content-encoding/content-length/transfer-encoding when re-wrapping the worker response. The body is written already-decoded and re-chunked, so copying those framed the bytes wrong — the same strip `apps/electric-sync/src/routes/headers.ts` does one layer down. - Proxy: await `drain` instead of ignoring `write()` backpressure, and tie an AbortSignal to the client socket, so a large shape snapshot no longer buffers in heap and a closed tab cancels its `live=true` invocation. - Proxy: once headers are sent a 502 is no longer expressible, so destroy the socket rather than splicing `bad gateway: …` into the shape body the client is mid-parse on. - Bindings: let the operator's environment override wrangler `vars` rather than the reverse, so the dev-local defaults baked into wrangler.jsonc stop winning in a self-hosted deploy. - Env: forward the EMAIL_/HAZEL_/SLACK_/APNS_/GITHUB_/CLOUDFLARE_/ PLANETSCALE_ prefixes the api's Env reads. They are optional there, so the missing prefixes failed silently. - Shutdown: stop crons, drain the sync server, then dispose Miniflare, and exit non-zero if dispose throws. - Compose: `restart: unless-stopped` on the long-lived services, so a failed boot migration or a host reboot doesn't leave the stack down.
`deploy/workerd/runtime.ts` is a container entrypoint in a self-contained package with its own package.json and lockfile, which root `bun install` never resolves. knip reported it as an unused file and failed test-packages. Declaring it as a root entry instead trades that one failure for four unlisted-dependency errors (cron, jsonc-parser, miniflare, postgres), since those live in deploy/workerd/package.json rather than the root manifest. Ignoring the directory matches how oxfmt and the lint/typecheck globs already treat it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #365, delivering the full self-hosted path discussed in #352 — run Maple outside Cloudflare, no Tinybird.
apps/api,apps/alerting, andapps/electric-syncare Workers with no plain-HTTP entry.deploy/workerd/runs their Wrangler bundles under Miniflare in one container — deriving every binding and cron from each worker's ownwrangler.jsoncat boot, so it tracks config changes here with no manual mirroring. It reaches Postgres via the Hyperdrive binding and runs the drizzle migrations (incl. the Electric publication) on startup.docker-compose.selfhost.ymlwires the whole stack — postgres + electric, clickhouse + ch-migrate + collector, workerd, the web SPA, and a Caddy proxy giving one origin (/api→api,/sync→electric-sync,/*→SPA).docs/self-hosting.mdis the guide;.env.selfhost.examplethe config. Also removes the deadapps/api/Dockerfile(itsbun run startCMD has no matching script; the api is a Worker) and revertsdocker-compose.ymlto the postgres+electric dev-DB role its header documents.Validated end-to-end from a clean
docker compose -f docker-compose.selfhost.yml up: ClickHouse + Postgres migrations, self-hosted login, and the Electricdashboardsshape all succeed through the proxy; a smoke trace reaches the collector.Runtime note: Miniflare is the working path today; celld is the natural target once it can run the api (TCP/Hyperdrive→Postgres, cron/
scheduled, KV/Queues) — happy to help move there when those land.Structural choices I'd welcome your steer on:
docker-compose.selfhost.ymlrather than reworking the rootdocker-compose.yml(kept as yourwrangler devdev-DB), so the two don't conflate. Happy to merge them if you'd prefer one.http://localhost:3471experience; drop it if you'd rather document a bring-your-own ingress.Need help on this PR? Tag
@codesmith-botwith what you need. Autofix is disabled.