Skip to content

feat(skills): publish sending-transactional-emails for the Wizard - #456

Open
Silthus wants to merge 15 commits into
mainfrom
feat/publish-sending-transactional-emails
Open

Silthus wants to merge 15 commits into
mainfrom
feat/publish-sending-transactional-emails

Conversation

@Silthus

@Silthus Silthus commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Problem

The Wizard's transactional-email command (PostHog/wizard#1507) runs a skill from the context-mill menu. The skill it should run, sending-transactional-emails, lives in PostHog/posthog, and no menu publishes it.

With this PR, every release publishes sending-transactional-emails. The build fetches the skill from PostHog/posthog, so PostHog's copy stays the only one to edit, and adds what the Wizard needs to run it. The skill keeps the email in the app's code by default (PostHog/posthog#128870). In the Wizard, the person is asked once: "Keep the email in your code (recommended) or design it in PostHog?"

Merge order

  1. ✅ PostHog/posthog#127123 merged.
  2. ✅ posthog-js#5509 and posthog-python#1171 merged and released.
  3. ✅ workflows-email-from-code is at 100%.
  4. ✅ PostHog/posthog#128870 merged on 2026-11-13, so the build check now fetches its files from master.
  5. This PR merges and is released.
  6. PostHog/wizard#1507 merges.

Merge Danger

Note

🔵 Merge risk: low, 🔴 once released · one-way door once a context-mill release publishes the skill: a public skill whose runs change people's apps and send email

  • At merge nothing changes for anyone. Only a release-please merge publishes, and the build fetches the skill from PostHog/posthog master then.
  • After a release, wizard skill sending-transactional-emails and the MCP skill resources offer the skill, and the Wizard's transactional-email command (PostHog/wizard#1507) runs it. A run writes email code into the app and, with the person's yes, sends one test email.
  • Covered at this head: the code path to a delivered first send, and the no-server-code abort, in the live walk. Prod shape: most Wizard runs with a known framework have server code, and about a third take the abort.
  • Uncovered: a project with no verified sender yet, which prod shape shows is the common case; a project with engagement events off, also the common case (the walk's project has them on); the template path, where nearly every project creates its first transactional template; a failing wizard_ask or a missing tool. Residual incident: a first send that fails or waits for the person, or engagement events the person still has to turn on, visible in the run and its report. No row sends or changes a setting without the person's yes.
  • API not enabled: both transactional email API flags are at 100% in US, so the [ABORT] transactional email api not enabled branch reaches no US project today. Where it fires, the run ends on its outro and sends nothing.
  • Upstream: each release publishes whatever SKILL.md is on PostHog/posthog master, unreviewed here. The test pins the two [ABORT] reasons the Wizard matches only against this repo's section. A moved or renamed upstream file fails the build with a 404.
  • Switch-off: a release without the skill. The Wizard reads the menu from the latest release at run time. Code already written into apps and emails already sent stay.
  • Tier: critical (c, f: one-way once a context-mill release publishes the skill)

Rollout

  • Stages: this PR merges, the next context-mill release publishes the skill, then PostHog/wizard#1507 merges and the Wizard release exposes the command.
  • Watch: Wizard runs of program transactional-email: wizard: agent completed against wizard: agent aborted by reason. No alert pages on it.

Impact

Feeds the Wizard's first-send command, which moves weekly orgs outside PostHog's own org that send email from code, on the Code-sending orgs by client insight. It rolls up to Workflows revenue through billed workflows_emails.

Changes

  • context/skills/sending-transactional-emails/, one all variant, so the skill ID is sending-transactional-emails. It has no cli: block, so it is a role: skill entry, like data-warehouse-source.
    • config.yaml fetches PostHog/posthog's SKILL.md as references/sending-transactional-emails.md, and its references/moving-from-other-providers.md. The shared docs are the transactional emails page and its HTTP API page.
    • description.md tells the agent to read the PostHog skill first and follow it. Its "Inside the PostHog Wizard" section covers what the Wizard changes. It names the PostHog skill's steps by their bold titles, so an upstream reorder can't move them:
      • Server code first. With no server code that can send, the agent emits [ABORT] no server code found before anything is asked.
      • One question, code first. The first wizard_ask call asks which email to send and "Keep the email in your code (recommended) or design it in PostHog?", with code as the first choice, once. When wizard_ask fails, the email stays in code, nothing is sent, and the report lists the skipped steps.
      • Missing tools. The Wizard's login may not carry the workflows-* and integrations-* scopes. A missing tool means the person does that step in PostHog: the sender, engagement events for every send, and for a template only, the skill's No marketing category. and Make it transactional. steps.
      • The key. The project secret API key is asked with sensitive: true and written by set_env_values as POSTHOG_PROJECT_SECRET_API_KEY, so it never shows in the run.
      • The test. A test send only with the person's consent, through the app's new send code, never reading the key. A one-off test script takes the address as an argument and is deleted after.
      • [ABORT] transactional email api not enabled when Transactional isn't in the project, or a template send answers 404 for a checked template ID.
  • scripts/lib/skill-generator.js: a doc entry { url, title, filename } writes the fetched file under filename, so PostHog's SKILL.md doesn't land next to this skill's own SKILL.md. The build fails when a filename isn't a plain <name>.md, or clashes with another doc's name.
  • config.yaml display_name: "Transactional Emails", so the skill menu titles the entry (97c703d7).

Evidence

  • Tests (scripts/lib/tests/):
    • skill-generator-doc-filename.test.js runs generateSkill with fetch stubbed. A doc entry with filename: 'upstream-skill.md' writes references/upstream-skill.md and no references/SKILL.md. A nested path, ../, a name without .md, .md alone, COMMANDMENTS.md and a duplicate name each fail the build.
    • sending-transactional-emails-skill.test.js (6e675d18) builds the real skill config with fetch stubbed and reads the package it writes: PostHog's skill lands as references/sending-transactional-emails.md and not as references/SKILL.md, every references/… file the built SKILL.md names is in the package (the two shared docs included, since b58090a2), and the built SKILL.md emits only the two [ABORT] reasons PostHog/wizard#1507 matches.
  • Build: CI's build check fetches PostHog/posthog master, which holds #128870's files since 2026-11-13.
  • Live run (2026-11-16, at b58090a2, served by this repo's dev server to the Wizard of PostHog/wizard#1507 at 7c9de7e4, in project 2): on an Express app, the run wrote the welcome email and its send at signup, and a signup sent it through posthog.emails.send. Its status read delivered 10 s later, and it arrived in the PR author's inbox. On a frontend-only Vite and React app, the run ended on [ABORT] no server code found before any question. Walks of the code and template paths on 2026-11-05 also delivered both sends.
  • Prod shape (2026-11-16, read only, numbers in the private proof store): most Wizard runs with a known framework have server code, and about a third have none and take the abort. Most projects have no email sender yet, so the sender setup branch is the common one; the walk's project already had one. Few teams emit engagement events or hold a transactional template, and both API flags are at 100% in US.

This revision (2026-11-16, b58090a2, unchanged code; live run and prod shape added, and the risk callout now names the release as the one-way door): bdecf5ba merges main (the skill menu now takes a title, description and tags), 97c703d7 titles the skill in that menu, 6e675d18 replaces the test that read the source text with one that reads the built package, and b58090a2 names the two shared docs by their references/ path so that test covers them. 1fdd558f added a test that pinned the wording of the API key. step, after a review of PostHog/wizard#1507 asked for the sensitive: true ask to be guaranteed. The fleet judge rejected it as a test of wording, not behavior, so abcfc5e1 reverts it. 1b72edff merges main, whose generator now names every doc page in one function (docFilenames). The doc entry's filename: override moves into that function, and its tests pass unchanged. A review of the merge found that the move dropped the check against references the skill already wrote. b6d433f2 restores it in writeDocs, so a named doc can't overwrite a local workflow step or an example. It also makes the [ABORT] test read each reason in full. Built with fetch stubbed, the skill package at b6d433f2 is byte-identical to the one at b58090a2.

Decisions

Question Answer Who Where
Which skill the Wizard's transactional-email command runs sending-transactional-emails from PostHog/posthog, published to the menu by a context-mill PR 👤 Human Silthus/posthog#461
Where the email lives by default In the app's code; a PostHog template is the alternative, asked once with code recommended 👤 Human Map conductor, 2026-11-04
When this releases After workflows-email-from-code reaches 100%; the skill has no fallback for projects without code sends 👤 Human Map conductor, 2026-11-04
A copy or a fetch Fetch PostHog/posthog's files at build time, so its copy is the one to edit 🤖 Agent comment
Where the Wizard section lives In this skill's description.md, around the fetched skill, not in PostHog/posthog, whose skill serves every agent 🤖 Agent comment
How the Wizard section points into the fetched skill By the steps' bold titles, never by number 🤖 Agent comment
Move the Wizard section into the fetched references/SKILL.md No: a second file named SKILL.md in the package reads as a second skill to tools that scan for it. Rejected: the move 🤖 Agent Lane #318

Involvement: no grilling rounds, no prototypes. Two Opus 5.5 review panels on the fork, GPT-6.1 Sol and Opus 5.5 final reads. On bdecf5ba: ReviewHog and a fleet proof panel (GPT-6.1 Sol and Opus 5.5); its findings on the menu title and tests that read source text are fixed in 97c703d7 and 6e675d18. Round 2 on the fixes (GPT-6.1 Sol correctness, Opus 5.5 fresh eyes): one minor and two nits, fixed in b58090a2. GPT-6.1 Sol final read: no findings.

🤖 Agent context

Agent: Claude Code, Claude Opus 5.5 (claude-opus-5-5), as the docs, skills and wizard slice of Silthus/posthog#461.

🤖 Generated with Claude Code

Build the sending-transactional-emails skill from PostHog/posthog's
SKILL.md and its reference, fetched at build time, with the
transactional emails docs and a section for running it inside the
Wizard. A docs_urls entry can now name its file, so the fetched
SKILL.md lands as references/sending-transactional-emails.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Nov 3, 2026

Copy link
Copy Markdown

🧙 Wizard CI

Run the Wizard CI and test your changes against wizard-workbench example apps by replying with a GitHub comment using one of the following commands:

Test all apps:

  • /wizard-ci all

Test all apps in a directory:

  • /wizard-ci ai-observability
  • /wizard-ci basic-integration
  • /wizard-ci feature-flags
  • /wizard-ci mcp-analytics
  • /wizard-ci replay-vision
  • /wizard-ci revenue
  • /wizard-ci self-driving
  • /wizard-ci warehouse
  • /wizard-ci warehouse-seeded

Test an individual app:

  • /wizard-ci ai-observability/anthropic
  • /wizard-ci ai-observability/google-adk
  • /wizard-ci ai-observability/groq
Show more apps
  • /wizard-ci ai-observability/manual-capture
  • /wizard-ci ai-observability/openai
  • /wizard-ci ai-observability/openai-agents
  • /wizard-ci ai-observability/opentelemetry
  • /wizard-ci ai-observability/vercel-ai
  • /wizard-ci basic-integration/android
  • /wizard-ci basic-integration/angular
  • /wizard-ci basic-integration/astro
  • /wizard-ci basic-integration/django
  • /wizard-ci basic-integration/fastapi
  • /wizard-ci basic-integration/flask
  • /wizard-ci basic-integration/flutter
  • /wizard-ci basic-integration/javascript-node
  • /wizard-ci basic-integration/javascript-web
  • /wizard-ci basic-integration/laravel
  • /wizard-ci basic-integration/next-js
  • /wizard-ci basic-integration/nuxt
  • /wizard-ci basic-integration/python
  • /wizard-ci basic-integration/rails
  • /wizard-ci basic-integration/react-native
  • /wizard-ci basic-integration/react-router
  • /wizard-ci basic-integration/sveltekit
  • /wizard-ci basic-integration/swift
  • /wizard-ci basic-integration/tanstack-router
  • /wizard-ci basic-integration/tanstack-start
  • /wizard-ci basic-integration/vue
  • /wizard-ci feature-flags/django
  • /wizard-ci feature-flags/next-js
  • /wizard-ci mcp-analytics/custom-dispatcher
  • /wizard-ci mcp-analytics/typescript-sdk
  • /wizard-ci replay-vision/javascript-node
  • /wizard-ci replay-vision/next-js
  • /wizard-ci replay-vision/react-vite
  • /wizard-ci revenue/stripe
  • /wizard-ci self-driving/astro
  • /wizard-ci self-driving/fastapi
  • /wizard-ci self-driving/nuxt
  • /wizard-ci self-driving/react-router
  • /wizard-ci self-driving/sveltekit
  • /wizard-ci warehouse/monorepo-env
  • /wizard-ci warehouse/multi-source-next
  • /wizard-ci warehouse/stripe-node
  • /wizard-ci warehouse/zero-source
  • /wizard-ci warehouse-seeded/next-stripe
  • /wizard-ci warehouse-seeded/next-stripe-declined

Test against a wizard branch:

  • /wizard-ci all wizard:my-branch

Add wizard:<branch> to any command above to pin the wizard branch. It defaults to main.

Results will be posted here when complete.

Silthus and others added 2 commits November 3, 2026 10:51
…, and check doc filenames

From the review panel:
- Check for server code before any question or PostHog step, so a
  frontend-only project aborts before the person sets anything up.
- Name the PostHog skill's steps by their bold titles, not by number or
  heading, so an upstream reorder can't move them.
- Say what to do when wizard_ask fails: skip the steps that need answers,
  list them in the report, and send nothing.
- Send the test through the app's own send code, never reading the key.
- Keep Protect sensitive data, the no-marketing-category rule and
  engagement events on the manual path.
- A doc entry's filename must be a plain .md name no other reference uses.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The filename check covered only entries that name a filename, so a later
doc whose name comes from its URL could still overwrite one. Now no doc
may take a filename an entry chose. Docs that both take their names from
their URLs keep today's behaviour.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Silthus

Silthus commented Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

Decisions on this PR, for the record:

Question Answer Who
A copy of the PostHog skill or a fetch Fetch PostHog/posthog's SKILL.md and its reference at build time, so PostHog's copy is the only one to edit. A copy here would drift. 🤖 Agent
Where the Wizard section lives In this skill's description.md, around the fetched skill. PostHog/posthog's skill serves every agent, so Wizard-only rules stay out of it. 🤖 Agent
How the Wizard section points into the fetched skill By the steps' bold titles (API key., Test.), never by number or heading, after the review panel showed an upstream reorder would silently move numbered steps. 🤖 Agent
Which filename clashes fail the build Any clash with a filename an entry names. Two docs that both take their names from their URLs keep today's overwrite, because the current build already has such a pair (error-tracking-ruby-on-rails) and failing it is out of scope here. 🤖 Agent

@Silthus
Silthus marked this pull request as ready for review November 3, 2026 17:41
@Silthus
Silthus requested a review from a team as a code owner November 3, 2026 17:41
@Silthus
Silthus marked this pull request as draft November 4, 2026 07:09
Silthus and others added 3 commits November 4, 2026 07:16
The Wizard section now asks once, with wizard_ask, "Keep the email in
your code (recommended) or design it in PostHog?", with code first.
The template steps a missing tool hands to the person apply only when
the person picks a template. When the test of a code send answers 404
not_found, the project can't send from code yet: the agent says so in
one line, moves the email into a PostHog template, and tests again,
rather than aborting. The skill description says code is the default.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…te section

- The Wizard section names the upstream skill's Design it in PostHog
  steps, which replace the Template: steps.
- The rule for projects without code sends goes. This skill publishes
  only after workflows-email-from-code reaches every project.
- The test send's one-off script takes the address as an argument and
  is deleted after the test.
- A doc filename needs a name before .md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… eight SDKs

PostHog/posthog#128870 adds references/sdks.md, which covers each SDK's
client setup, sends and status read. The build fetches it with the
skill's other files, and the description names all eight SDKs.

The build answers 404 for it until #128870 merges into master.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Silthus
Silthus marked this pull request as ready for review November 5, 2026 08:42
…ery send

PostHog records engagement events for code sends only when the
project's setting is on, as for template sends. The Wizard section
asked about the setting only for a template.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Silthus
Silthus marked this pull request as draft November 14, 2026 17:46
Silthus and others added 2 commits November 14, 2026 19:02
The menu's new title field comes from the variant's display_name, which
read 'all supported platforms'.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@posthog posthog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PostHog Review · Standard · reviewed 97c703d: no findings.

…ot its source text

Build the skill with fetch stubbed and assert what the package holds: the
PostHog skill as a reference, no dangling reference, and exactly the
[ABORT] reasons the Wizard program matches. Drop the tests that pinned
the wording and order of the Wizard section.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@posthog posthog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PostHog Review · Standard · reviewed 6e675d1: no findings.

The test that checks every named reference file now covers both docs.
Its [ABORT] pattern no longer needs a closing backtick, and the header
comment goes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@posthog posthog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PostHog Review · Standard · reviewed b58090a: no findings.

@Silthus
Silthus marked this pull request as ready for review November 16, 2026 08:31
@Silthus
Silthus requested a review from dmarchuk November 16, 2026 08:31
Silthus and others added 2 commits November 16, 2026 08:36
The Wizard's transactional-email command keeps the project secret key out
of the model only because this skill tells the agent to ask for it with
sensitive: true. Fail the build when that step stops saying so.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…PI key"

This reverts commit 1fdd558. The test
checked wording, not that the key stays out of the model.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@posthog posthog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PostHog Review · Standard · reviewed abcfc5e: no findings.

…transactional-emails

# Conflicts:
#	scripts/lib/skill-generator.js
The merge with main moved the filename override into docFilenames, which
sees only the docs, so a named doc could overwrite a local workflow step
or example. writeDocs now refuses it before fetching. The [ABORT] test
also reads each reason in full, so an added reason fails it whatever its
first letter.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@posthog posthog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PostHog Review · Standard · reviewed b6d433f: no findings.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants