Skip to content

MERGE_PLAN.md

Latest commit

 

History

History
295 lines (206 loc) · 9.37 KB

MERGE_PLAN.md

File metadata and controls

295 lines (206 loc) · 9.37 KB
Raw

DataOps Merge Plan

Direction

Build DataTalksClub/dataops as the single operations portal for DataTalks.Club.

The first milestone is a code and process merge, not a full product rewrite. The repo should bring the existing systems under one roof, keep them runnable, and create clear integration points.

Source systems:

  • ../dtc-operations: process docs, SOP editor, search, Lambda docs app
  • ../datatasks: task execution engine, templates, cards, recurring work
  • ../podcast-assistant: Telegram intake and AI drafting for podcast prep

Product Result

The merged portal will have these top-level areas:

  • Home: today, overdue, active workflows, inbox, assistant jobs
  • Work: tasks, cards, workflows, recurring work, assignments
  • Processes: SOPs, templates, references, playbooks, prompts, lint status
  • Assistants: podcast assistant first, later process-doc and newsletter helpers
  • Artifacts: files, generated docs, event links, transcripts, invoices, reports
  • Search: one search across work, process docs, artifacts, and assistant output
  • Admin: users, integrations, imports, secrets, diagnostics

Repository Structure

Use this target structure after the merge:

dataops/
  docs/                  repo process, architecture, merge notes
  content/                DataTalks.Club process docs from dtc-operations
  frontend/               shared portal frontend
  backend/                Python backend copied from dtc-operations Lambda app
  work-engine/            DataTasks TypeScript task engine during transition
  assistants/
    podcast/              podcast-assistant code and prompts
  scripts/                import, migration, lint, dev tooling
  tests/                  cross-system tests
  playwright_tests/       portal E2E tests

Transition rule:

  • Keep source systems mostly intact at first.
  • Integrate through APIs and explicit import/sync scripts.
  • Refactor only after one workflow runs end to end in the merged repo.

Technology Choices

Backend

Superseded -- TypeScript is the consolidated backend. The analysis below was the original merge-period direction; it was reversed in favor of one TypeScript backend (see docs/TARGET_ARCHITECTURE.md for the decision and rationale). The Python docs/SOP/search backend has been retired; its content-validation tooling lives in tools/content_tools/.

Original direction (historical context):

  • Use Python as the long-term portal backend.
  • The dtc-operations docs Lambda, podcast-assistant, and process-doc parsing/linting are already Python.
  • Python was a better fit for assistant jobs, document parsing, transcription, and background processing.

Why this was reversed:

  • The work engine (the larger, stateful, actively-developed core) is TypeScript.
  • The Python backend's only runtime third-party deps were minsearch and python-frontmatter, both with drop-in Node equivalents.
  • minsearch pulled in scikit-learn + numpy + scipy + pandas -- the single largest contributor to Lambda package size and cold-start latency.
  • Replacing it with zerosearch-node (zero-dependency BM25-lite) both removes the heavy scientific-Python stack and gives the lighter Lambda.

Current state: backend/ is the single TypeScript backend. The only remaining Python is the podcast assistant worker and the content-validation CI tools.

Frontend

Use the existing dtc-operations vanilla JavaScript frontend as the first portal shell.

Reasons:

  • It already has docs navigation, search, filters, editor, drafts, lint status, image upload, keyboard shortcuts, and mobile layout.
  • It avoids a framework rewrite before the merge proves the product model.

Do not introduce React or another frontend framework in the merge milestone.

Data Stores

Use separate stores for separate truth:

  • GitHub markdown is the source of truth for process knowledge.
  • DynamoDB stores execution state: users, tasks, cards, workflows, recurring work, assistant jobs, artifacts, sessions, and notifications.
  • S3 or GitHub stores generated files depending on the artifact type.
  • Lambda /tmp remains cache only, not durable storage.

Search

Superseded -- search now uses zerosearch-node (zero-dependency BM25-lite). The original minsearch (TF-IDF via scikit-learn) has been retired; see docs/TARGET_ARCHITECTURE.md.

Original direction (historical context):

Then add indexed records for:

  • workflows
  • task templates
  • active cards
  • tasks
  • artifacts
  • assistant outputs

Deployment

Keep the dtc-operations Lambda deployment model for the first merge.

Required paths:

  • Content-only changes validate docs and refresh the deployed index.
  • Code changes run backend, frontend, and Playwright tests before deploy.
  • Assistant jobs can run locally first, then move to a Lambda, container, or queued worker once the portal job model is stable.

Package Management

Use both ecosystems during transition:

  • Python: uv
  • Node/TypeScript: npm

Add one top-level Makefile later to wrap common commands.

Merge Phases

Phase 0: Repo Setup

Outcome: DataTalksClub/dataops is ready for issue-driven work.

Tasks:

  • Create public GitHub repo.
  • Add planning docs.
  • File initial merge issues.

Phase 1: Import DTC Operations

Outcome: the process-doc portal runs from dataops.

Tasks:

  • Copy content/, frontend/, lambda-functions/, scripts/, templates/, and docs-app tests from ../dtc-operations.
  • Preserve commit history only if we decide it is worth the added merge complexity. Otherwise record the source commit in docs/import-log.md.
  • Rename lambda-functions/ to backend/ after import, or leave it in place until tests pass.
  • Keep docs search, editor, lint, image handling, and GitHub-backed save working.
  • Update repo names and deployment configuration from dtc-operations to dataops.

Validation:

  • Local docs app starts.
  • Search works.
  • SOP lint still runs.
  • A document can be edited and saved in local mode.

Phase 2: Import DataTasks

Outcome: task execution code lives in the merged repo and still runs.

Tasks:

  • Copy DataTasks into work-engine/.
  • Keep its TypeScript build, tests, and Playwright tests passing from that subdirectory.
  • Add an adapter contract between portal backend and work engine.
  • Expose tasks, templates, cards, recurring configs, users, files, and notifications as portal concepts.
  • Add instructionDocId, instructionStepId, phase, systems, and required artifact fields to the task/template model.

Validation:

  • npm test, npm run typecheck, and focused Playwright tests pass in work-engine/.
  • A task can link to a process doc by ID.
  • A card can show related process docs.

Phase 3: Import Podcast Assistant

Outcome: podcast-assistant is a portal assistant module.

Tasks:

  • Copy local ../podcast-assistant into assistants/podcast/.
  • Keep its tests passing with uv.
  • Replace local-only documents/ output with portal artifact records.
  • Add assistant job records: queued, running, needs review, done, failed.
  • Keep Telegram intake.
  • Add a portal screen for inbox, job logs, generated draft review, and approval.

Validation:

  • Telegram or local input can create a podcast assistant inbox item.
  • The assistant can draft a podcast prep document.
  • A reviewed draft becomes a podcast-doc artifact.

Phase 4: First End-to-End Workflow

Outcome: one real workflow runs across docs, tasks, artifacts, and assistant output.

Use Podcast first.

Tasks:

  • Add stable IDs to priority podcast SOPs and templates.
  • Define the Podcast workflow from the DataTasks/Trello template.
  • Link each workflow task to a process doc or template.
  • Preserve operator-flow links through stable IDs, with external assistant process references called out until the content registry supports them.
  • Add required artifacts: podcast doc, Luma link, Meetup link, YouTube link, transcript, Spotify link, Apple link, DTC webpage link.
  • Add stages: prep, announced, live, post-production, published, follow-up, done.
  • Add Playwright coverage for the operator flow.

Validation:

  • Create Podcast card.
  • Generate tasks.
  • Open a task and jump to its SOP.
  • Run podcast assistant.
  • Approve generated podcast doc.
  • Attach artifact to card.
  • Complete required-link tasks only after outputs exist.

Phase 5: Consolidate Backend

Outcome: the portal has one backend implementation for work and docs.

Tasks:

  • Move task execution from TypeScript into Python only after the first workflow proves the model.
  • Migrate DataTasks data access to the portal backend.
  • Retire work-engine/ when parity tests pass.

Do not start this phase before Phase 4 succeeds.

Initial GitHub Issues

Create these as the first issues in DataTalksClub/dataops:

  1. Import DTC Operations into dataops.
  2. Keep DTC Operations docs app running from merged repo.
  3. Import DataTasks into work-engine/.
  4. Add document registry IDs and resolver.
  5. Add task-to-process-doc links.
  6. Import Podcast Assistant into assistants/podcast/.
  7. Define Podcast workflow.
  8. Build Podcast end-to-end slice.
  9. Add first portal dashboard.

Key Decisions

Base App

Use dtc-operations as the base app during the merge because it already owns the process docs and portal-like frontend.

Repo

Use DataTalksClub/dataops as the new public repo.

Issues

Use GitHub Issues in DataTalksClub/dataops.

First Workflow

Use Podcast as the first real workflow because it touches all three source systems.