Maintenance/refactor to ninja

.env.example

@@ -70,7 +70,9 @@ LOG_LEVEL=INFO
 # Bearer token required to access the Prometheus-style /metrics endpoint.
 METRICS_TOKEN=
 
-CELERY_TASK_ALWAYS_EAGER=false
+RABBITMQ_URL=amqp://digest-engine:digest-engine@rabbitmq:5672/
+TASKIQ_ALWAYS_EAGER=false
+TASKIQ_SCHEDULER_SKIP_FIRST_RUN=true
 
 DJANGO_SUPERUSER_USERNAME=admin
 DJANGO_SUPERUSER_EMAIL=admin@example.com

.env.test

@@ -6,8 +6,7 @@ OLLAMA_URL=http://ollama:11434
 REDDIT_CLIENT_ID=client
 REDDIT_CLIENT_SECRET=secret
 REDDIT_USER_AGENT=digest-engine/test
-CELERY_BROKER_URL=memory://
-CELERY_RESULT_BACKEND=cache+memory://
+TASKIQ_ALWAYS_EAGER=false
 ALLOWED_HOSTS=localhost,127.0.0.1,nginx,testserver
 NEWSLETTER_API_INTERNAL_URL=http://127.0.0.1:8080
 NEWSLETTER_PUBLIC_URL=http://127.0.0.1:8080

.github/copilot-instructions.md

@@ -1,6 +1,6 @@
 # Digest Engine Project Instructions
 
-You are working in Digest Engine, a Django + DRF + Celery + Qdrant backend with a Next.js App Router frontend.
+You are working in Digest Engine, a Django + Ninja API + Taskiq + Qdrant backend with a Next.js App Router frontend.
 
 ## Repository Shape
 
@@ -22,14 +22,14 @@ You are working in Digest Engine, a Django + DRF + Celery + Qdrant backend with
 
 - Project scoping is a core invariant. Most API resources are nested under `/api/v1/projects/{project_id}/...`.
 - Treat `core/` as the home for genuine cross-cutting concerns only. New app-owned runtime logic should live with its owning app rather than expanding `core/`.
-- Reuse the established DRF patterns in `core/api.py`, `core/api_urls.py`, and `core/serializer_mixins.py`:
-	- `ProjectOwnedQuerysetMixin` for nested viewsets
-	- serializer context containing `project`
-	- explicit validation for cross-project foreign keys
-- Keep viewsets and views thin. Put operational logic in `core/tasks.py`, `core/pipeline.py`, `ingestion/plugins/`, `newsletters/intake.py`, or nearby helpers owned by the feature's app.
+- Reuse the established Ninja patterns in `projects/ninja_api.py` and the app `*_ninja_api.py` modules:
+	- keep most resources nested under `/api/v1/projects/{project_id}/...`
+	- validate cross-project foreign keys explicitly in schema helpers or endpoint-local validation code
+	- keep path wiring thin and put workflow logic in app-owned helpers
+- Keep route handlers and views thin. Put operational logic in `core/tasks.py`, `core/pipeline.py`, `ingestion/plugins/`, `newsletters/intake.py`, or nearby helpers owned by the feature's app.
 - Preserve existing API field shapes. Backend serializers and frontend types currently use `snake_case`; do not introduce ad hoc `camelCase` transforms.
-- When API behavior changes, update drf-spectacular schema metadata in `core/api.py`.
-- When changing ingestion, newsletter intake, AI processing, or embeddings, preserve the handoff between database state, Celery tasks, and Qdrant state.
+- When API behavior changes, update the Ninja schema, response typing, and docs metadata in the owning `*_ninja_api.py` module.
+- When changing ingestion, newsletter intake, AI processing, or embeddings, preserve the handoff between database state, Taskiq tasks, and Qdrant state.
 
 ## Frontend Conventions
 
@@ -85,4 +85,4 @@ Use the workspace skills in `.github/skills/` when they match the task:
 - `bridge-scaffolder`: features that span Django API work and Next.js consumption.
 - `project-api-patterns`: adding or changing project-scoped DRF endpoints.
 - `source-plugin-patterns`: adding or changing ingestion plugins or source-config behavior.
-- `ai-pipeline-patterns`: changing embeddings, relevance scoring, newsletter intake, or Celery-driven AI workflow behavior.
+- `ai-pipeline-patterns`: changing embeddings, relevance scoring, newsletter intake, or Taskiq-driven AI workflow behavior.

.github/instructions/backend-python.instructions.md

@@ -1,6 +1,6 @@
 ---
 name: "Backend Python Guidelines"
-description: "Use when editing Django, DRF, Celery, plugin, management command, or backend test code in Python. Covers project scoping, workflow placement, docstrings, and focused validation for core/, digest_engine/, tests/, and manage.py."
+description: "Use when editing Django, DRF, Taskiq, plugin, management command, or backend test code in Python. Covers project scoping, workflow placement, docstrings, and focused validation for core/, digest_engine/, tests/, and manage.py."
 applyTo:
   - "core/**/*.py"
   - "digest_engine/**/*.py"
@@ -11,13 +11,13 @@ applyTo:
 # Backend Python Guidelines
 
 - Preserve `project` as the scoping boundary. Do not reintroduce `tenant` naming.
-- Keep Django views and DRF viewsets thin. Put operational logic in nearby helpers such as `core/tasks.py`, `core/pipeline.py`, `core/newsletters.py`, `core/plugins/`, or focused modules next to the owning workflow.
-- For nested API resources, follow the patterns in `core/api.py`, `core/api_urls.py`, and `core/serializers.py`.
-- Enforce cross-project relationship validation in serializers instead of trusting the client.
+- Keep Django views and Ninja route handlers thin. Put operational logic in nearby helpers such as `core/tasks.py`, `core/pipeline.py`, `core/newsletters.py`, `core/plugins/`, or focused modules next to the owning workflow.
+- For nested API resources, follow the patterns in `projects/ninja_api.py` and the owning app `*_ninja_api.py` modules.
+- Enforce cross-project relationship validation in request schemas or endpoint-local validation instead of trusting the client.
 - Preserve existing API field names in `snake_case` unless the contract is intentionally changing across backend and frontend.
 - Use Google-style docstrings with PEP 257 conventions for public modules, classes, functions, and non-obvious helpers.
 - Keep changes small and local. Do not create generic `services.py` or `utils.py` files unless the repo already needs that extraction.
-- When changing ingestion, embeddings, newsletter intake, or review behavior, keep the database, Celery, and Qdrant handoff coherent.
+- When changing ingestion, embeddings, newsletter intake, or review behavior, keep the database, Taskiq, and Qdrant handoff coherent.
 
 ## Validation
 
@@ -30,7 +30,7 @@ applyTo:
 ## Good Anchors
 
 - `core/models.py`
-- `core/serializers.py`
+- `projects/ninja_api.py`
 - `core/api.py`
 - `core/tasks.py`
 - `core/pipeline.py`

.github/instructions/dev-server-workflow.instructions.md

@@ -11,4 +11,4 @@ applyTo:
 - Assume the user usually already has the relevant dev server running.
 - Before running commands such as `pnpm dev`, `next dev`, `vite`, `npm run dev`, `just <app>-dev`, or similar long-lived local servers, ask the user to start the server or confirm that no server is already running.
 - Only start a dev server yourself when the user explicitly asks you to do that.
-- When you need runtime verification and no server is available, prefer asking the user to start the correct app server instead of launching one on their behalf.
+- When you need runtime verification and no server is available, prefer asking the user to start the correct app server instead of launching one on their behalf.

.github/skills/ai-pipeline-patterns/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: ai-pipeline-patterns
-description: "Use when changing the AI workflow, embeddings, Qdrant integration, newsletter intake, relevance scoring, summarization, classification, Celery handoff, or Anymail inbound processing. Trigger phrases include pipeline, relevance scoring, embeddings, Qdrant, newsletter intake, summarization, classification, OpenRouter, and review queue."
+description: "Use when changing the AI workflow, embeddings, Qdrant integration, newsletter intake, relevance scoring, summarization, classification, Taskiq handoff, or Anymail inbound processing. Trigger phrases include pipeline, relevance scoring, embeddings, Qdrant, newsletter intake, summarization, classification, OpenRouter, and review queue."
 ---
 
 # AI Pipeline Patterns Skill

.github/skills/bridge-scaffolder/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: bridge-scaffolder
-description: "Use when creating or changing a feature that spans both the Django API and the Next.js frontend. Trigger phrases include full-stack feature, add endpoint and UI, wire frontend to backend, project dashboard change, and bridge serializer to frontend types."
+description: "Use when creating or changing a feature that spans both the Django API and the Next.js frontend. Trigger phrases include full-stack feature, add endpoint and UI, wire frontend to backend, project dashboard change, and bridge backend schemas to frontend types."
 ---
 
 # Bridge Scaffolder Skill
@@ -9,20 +9,20 @@ Use this skill when a change genuinely crosses the backend and frontend boundary
 
 ## Rules
 
-- **Django Side:** Follow the existing patterns in `core/api.py`, `core/api_urls.py`, and `core/serializers.py`.
+- **Django Side:** Follow the existing patterns in `projects/ninja_api.py` and the app `*_ninja_api.py` modules.
 - Most nested resources should stay project-scoped under `/api/v1/projects/{project_id}/...`.
-- Keep business logic out of viewsets. Use `core/tasks.py`, `core/pipeline.py`, `core/newsletters.py`, `core/plugins/`, or nearby helpers for real workflow logic.
+- Keep business logic out of route handlers. Use `core/tasks.py`, `core/pipeline.py`, `core/newsletters.py`, `core/plugins/`, or nearby helpers for real workflow logic.
 - **Next.js Side:** Update `frontend/src/lib/types.ts`, `frontend/src/lib/api.ts`, and the relevant pages, components, or route handlers under `frontend/src/app/`.
 - Preserve the existing `snake_case` payload shape unless the backend contract is intentionally changing.
 
 ## Implementation Guidance
 
-- Check `core/api_urls.py` for the current route topology and `core/api.py` for the schema helper patterns.
-- Keep serializer validation aligned with project scoping and cross-project relationship rules.
+- Check `digest_engine/ninja_api.py`, `projects/ninja_api.py`, and the relevant app `*_ninja_api.py` files for the current route topology and schema patterns.
+- Keep schema validation aligned with project scoping and cross-project relationship rules.
 - If the frontend consumes the new field or endpoint, reflect it in `frontend/src/lib/types.ts` and the corresponding API helpers.
 - Update docs when the feature changes a core workflow or user-facing behavior.
 
 ## Related Guidance
 
-- Use `project-api-patterns` when the backend portion is primarily a new or changed DRF resource.
+- Use `project-api-patterns` when the backend portion is primarily a new or changed project-scoped API resource.
 - Use `coverage-auditor` immediately after scaffolding to add targeted backend and frontend tests.

.github/skills/coverage-auditor/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: coverage-auditor
-description: "Use when adding or updating tests for Django, DRF, Celery, admin, serializer, plugin, or Next.js code, or when closing a coverage gap. Trigger phrases include add tests, improve coverage, pytest, vitest, missing branch, serializer test, admin test, and route handler test."
+description: "Use when adding or updating tests for Django, Ninja API, Taskiq, admin, plugin, or Next.js code, or when closing a coverage gap. Trigger phrases include add tests, improve coverage, pytest, vitest, missing branch, schema validation test, admin test, and route handler test."
 ---
 
 # Coverage Auditor Skill
@@ -24,6 +24,7 @@ Use this skill to add the smallest effective tests around the changed behavior.
 	- `core/tests/test_tasks.py`
 	- `core/tests/test_newsletters.py`
 	- `core/tests/test_pipeline.py`
+	- the owning app's `tests/` package for Ninja route coverage
 - For frontend work, add or extend the nearest colocated `*.test.ts` or `*.test.tsx` file beside the owning route, page, or component.
 - When adding new frontend tests, keep imports sorted to satisfy the repo's ESLint import-order rules. If you hit `Run autofix to sort these imports!`, fix the import block or run file-scoped ESLint before moving on.
 - After changing tests, run the narrowest relevant validation command first.

.github/skills/project-api-patterns/SKILL.md

@@ -1,33 +1,32 @@
 ---
 name: project-api-patterns
-description: "Use when adding or changing Django REST Framework serializers, viewsets, nested routes, schema docs, or project-scoped endpoints in core/api.py, core/api_urls.py, or core/serializers.py. Trigger phrases include project API, nested route, DRF viewset, serializer validation, project_id endpoint, and drf-spectacular docs."
+description: "Use when adding or changing project-scoped Ninja routes, request or response schemas, nested routes, or API docs in `projects/ninja_api.py` and related app `*_ninja_api.py` modules. Trigger phrases include project API, nested route, schema validation, project_id endpoint, and OpenAPI docs."
 ---
 
 # Project API Patterns Skill
 
-Use this skill when changing the project-scoped REST API.
+Use this skill when changing the project-scoped API.
 
 ## Rules
 
 - Treat `Project` as the isolation boundary.
 - Top-level project resources live on the base router; most other resources are nested under `/api/v1/projects/{project_id}/...`.
-- Reuse `ProjectOwnedQuerysetMixin` in `core/api.py` for nested resources.
-- Pass `project` through serializer context and enforce cross-project relationship validation in serializers.
-- Keep API field names in `snake_case` to match current serializers and frontend types.
-- Update drf-spectacular metadata in `core/api.py` when the endpoint contract changes.
+- Reuse the existing nested router structure in `projects/ninja_api.py` and the app `*_ninja_api.py` modules for nested resources.
+- Enforce cross-project relationship validation explicitly in schema helpers or endpoint-local validation code.
+- Keep API field names in `snake_case` to match current backend payloads and frontend types.
+- Update Ninja request schemas, response schemas, and OpenAPI metadata in the owning `*_ninja_api.py` module when the endpoint contract changes.
 
 ## Implementation Guidance
 
-- Add or update serializers in `core/serializers.py`.
-- Add or update viewsets and schema decorators in `core/api.py`.
-- Register routes in `core/api_urls.py` using the existing nested router pattern.
+- Add or update request and response schemas in the owning `*_ninja_api.py` module or nearby helpers when the logic is shared.
+- Add or update route handlers and OpenAPI metadata in the owning `*_ninja_api.py` module.
+- Register routes through `projects/ninja_api.py` or `digest_engine/ninja_api.py` using the existing nested router pattern.
 - If the frontend consumes the API, update `frontend/src/lib/types.ts` and `frontend/src/lib/api.ts`.
-- Add or update focused tests near the changed behavior, usually under `core/tests/`.
+- Add or update focused tests near the changed behavior, usually under the owning app's `tests/` package.
 
 ## References
 
-- `core/api.py`
-- `core/api_urls.py`
-- `core/serializers.py`
+- `digest_engine/ninja_api.py`
+- `projects/ninja_api.py`
 - `frontend/src/lib/types.ts`
 - `frontend/src/lib/api.ts`

.github/skills/source-plugin-patterns/SKILL.md

@@ -20,9 +20,9 @@ Use this skill when working on the ingestion plugin system.
 
 - Update `core/plugins/base.py` only when the shared plugin contract must change.
 - Add or update concrete plugins in `core/plugins/`.
-- Update `core/plugins/registry.py` and any related enum or serializer validation paths.
-- Confirm `SourceConfig` validation still works through both the serializer and admin paths.
-- Add or update focused tests in `core/tests/test_tasks.py`, `core/tests/test_admin.py`, `core/tests/test_serializers.py`, or a new nearby plugin test module.
+- Update `core/plugins/registry.py` and any related enum or request-schema validation paths.
+- Confirm `SourceConfig` validation still works through both the API validation layer and admin paths.
+- Add or update focused tests in `core/tests/test_tasks.py`, `core/tests/test_admin.py`, or a new nearby plugin test module.
 
 ## References
 

.gitignore

@@ -12,7 +12,6 @@ __pycache__/
 **/.turbo/
 *storybook.log
 
-celerybeat-schedule*
 coverage/
 db.sqlite3
 docs/_internal_only/

.python-version

@@ -1 +1 @@
-3.13
+3.13

.vscode/extensions.json

@@ -3,4 +3,4 @@
     "astral-sh.uv-vscode",
     "ms-pyright.pyright"
   ]
-}
+}

.vscode/settings.json

@@ -6,6 +6,7 @@
     "abangser",
     "Aptos",
     "ASGI",
+    "asyncio",
     "botocore",
     "bsky",
     "buildx",
@@ -60,6 +61,7 @@
     "pyasn",
     "pylint",
     "pyopenssl",
+    "pyproject",
     "PYTHONDONTWRITEBYTECODE",
     "PYTHONUNBUFFERED",
     "pytokens",
@@ -78,6 +80,7 @@
     "solomonstre",
     "srgb",
     "svix",
+    "Taskiq",
     "topicv",
     "txaio",
     "ujson",
@@ -96,5 +99,6 @@
     "xrpc",
     "xxhash",
     "zope"
-  ]
+  ],
+  "git.ignoreLimitWarning": true
 }