From 68ade36fe6faee3c799e562ba2f84081d8a36439 Mon Sep 17 00:00:00 2001 From: Matt Miermans Date: Thu, 30 Jul 2026 13:41:11 -0400 Subject: [PATCH 01/51] feat(newtab): Add hnt-backend-investigation skill --- plugins/newtab/README.md | 8 + .../skills/hnt-backend-investigation/SKILL.md | 312 ++++++++++++++++++ .../references/data-sources.md | 262 +++++++++++++++ .../references/failure-modes.md | 146 ++++++++ 4 files changed, 728 insertions(+) create mode 100644 plugins/newtab/skills/hnt-backend-investigation/SKILL.md create mode 100644 plugins/newtab/skills/hnt-backend-investigation/references/data-sources.md create mode 100644 plugins/newtab/skills/hnt-backend-investigation/references/failure-modes.md diff --git a/plugins/newtab/README.md b/plugins/newtab/README.md index 407457f..e13bd86 100644 --- a/plugins/newtab/README.md +++ b/plugins/newtab/README.md @@ -9,3 +9,11 @@ Skills and tools for Firefox newtab development. Adds `@nova-cleanup` comments to Firefox newtab code for Project Nova parallel implementation tracking. Use this skill when adding cleanup comments to Nova-related changes. See [`skills/nova-cleanup-comments/SKILL.md`](skills/nova-cleanup-comments/SKILL.md) for details. + +### `hnt-backend-investigation` + +Diagnoses Home New Tab **backend** errors, outages, and data-quality defects — Merino, the curated corpus, admin-api, the article crawler, and the ML section pipeline. + +Use this skill when a Sentry alert fires, an editor reports something broken, or recommendations look empty, wrong, or stale. It confirms the symptom independently, probes competing hypotheses in parallel, stratifies metrics, tries to break its own conclusion, and writes an evidence-backed `FINDINGS.md` naming the check that would have caught the problem earlier. See [`skills/hnt-backend-investigation/SKILL.md`](skills/hnt-backend-investigation/SKILL.md) for details. + +**Needs:** `gh`; `gcloud`/`bq` with a billing project; the `aws` CLI with a read-only SSO profile; a MySQL client plus Mozilla VPN; `uv` or `python3`; the Sentry MCP server if you have it. The skill runs on whatever subset you have — anything unreachable is raised as a task and reported under "Could not measure". diff --git a/plugins/newtab/skills/hnt-backend-investigation/SKILL.md b/plugins/newtab/skills/hnt-backend-investigation/SKILL.md new file mode 100644 index 0000000..a4437b7 --- /dev/null +++ b/plugins/newtab/skills/hnt-backend-investigation/SKILL.md @@ -0,0 +1,312 @@ +--- +name: hnt-backend-investigation +description: Investigates Home New Tab backend errors, outages, and data-quality problems across Merino, the curated corpus, admin-api, the article crawler, and the ML section pipeline. Confirms the symptom independently, probes competing hypotheses in parallel, stratifies metrics, tries to break its own conclusion, then writes an evidence-backed FINDINGS doc naming the check that would have caught it. Use when a Sentry alert fires, an editor reports something broken, or recommendations look empty, wrong, or stale. +--- + +# New Tab (HNT) Backend Investigation + +Diagnose a problem in the services behind Firefox Home New Tab recommendations — Merino, the +curated corpus, admin-api, the article crawler, the ML section pipeline, the data pipelines. Not +for in-tree `browser/extensions/newtab` frontend bugs. + +Your product is a **diagnosis**: a root cause, the evidence for it, the impact quantified, and the +check that would have caught it earlier. Two populations can be affected, and every finding should +say which: **Firefox New Tab clients** (everyone who sees recommendations) and **editors/curators** +(the internal editorial team working through curation-admin-tools and admin-api). + +**How to reach each system, and the traps that will silently give you wrong answers** — read before +your first probe in step 1: [references/data-sources.md](references/data-sources.md) + +**How these systems fail, the invariants worth asserting, and falsification moves that work** — read +before you enumerate hypotheses in step 3, because it will change which ones you take seriously: +[references/failure-modes.md](references/failure-modes.md) + +## How to think about this + +Four habits do most of the work. They matter more than any specific query below. + +**Confirm the symptom before explaining it.** Independently reproduce or measure the reported +problem first, whatever its source. An alert can be miscalibrated, mis-scoped, or watching the +wrong layer, and a description passed through two people can drift. Until you have seen the +symptom yourself in data you pulled, you do not know what you are diagnosing. + +**Form the hypothesis before you run the check.** Write down what you expect to see, and what you +would expect to see if the hypothesis were false. A query designed to confirm and a query designed +to discriminate look similar and are not. The characteristic failure in this domain is a correct +query paired with a wrong inference, stated confidently. + +**Work several lines at once.** When more than one explanation is plausible, probe them in parallel +rather than following the most attractive one to its end. Breadth is cheap in the unmetered planes — +Sentry, repo reads, CloudWatch alarm history, one live Merino request — and tunnel vision is the +expensive failure. It is not free in BigQuery or Zyte: dry-run and select narrow columns before you +fan out. Keep a list of every line you considered and record which you dropped and why. + +**Prefer measuring to reasoning.** Reproduce the request. Run the model on real inputs. Count the +actual rows. Read the code path instead of assuming its behaviour. A plausible mechanism becomes a +number, and only numbers survive review. + +Write like the evidence is asymmetric, because it usually is. Avoid "always", "never", "in every +case"; say what each count is a count of, and whether it is a floor, a ceiling, or a sample. + +## Don't stall on the developer + +Every pause costs wall-clock, because the developer may not be watching. Default to proceeding: +pick the sensible option, say which one you picked, and keep going. Ask a blocking question only +when proceeding on any assumption would waste the whole investigation. + +When a source you cannot reach looks promising, raise it **once** as a pending task carrying the +whole errand in its own text, prefixed `DEV:` so it reads as theirs rather than yours — e.g. +`DEV: connect to VPN so corpus MySQL is reachable — settles whether the DE items exist at all`. +Give the exact command, not a description of the problem; for an interactive login use the `!` +in-session form so the output lands here, e.g. +`! aws --profile sso login`. If the answer lives in a dashboard +or console you have no credential for, do not ask for access at all — name the view, the filter, and +the single number or shape you need, and ask them to read it back. + +**Then keep investigating the other hypotheses immediately. Do not wait.** + +Restate it exactly once more — whichever comes first: they confirm, the unblocked lines run out, or +you are three probes into a line you had already judged weaker than the blocked one. That is the +second and last ask. If there is still no response, finish steps 7 and 8 with `Status: blocked`, +name the one unblock under "Could not measure", and leave the access task pending. + +Stopping outright is a last resort: only when the blocked source is the only thing that can settle +the question **and** it cannot be reduced to a question they can answer for you — a dashboard, +console, or explore is almost never a real stop. A request that quietly disappears is worse than one +that was never made. + +## Workspace + +One directory holds the whole investigation: `~/investigation/-
-`, e.g. +`~/investigation/2026-jul29-empty-de-sections`. Reuse a directory only when its FINDINGS.md is about +the same symptom. Say which directory you chose; do not ask. + +- `FINDINGS.md` — one living document. Create it from the step-7 skeleton **before your first + probe**; steps 2, 4 and 6 write into it as they go, so step 7 is a final pass over a document that + already exists. Rewrite it in place, not as `FINDINGS-v2.md`. +- `queries/` and `results/` — every query as a file, its output alongside under the same basename. +- `api_responses/` — raw JSON from live calls. Save them even when they look boring; metered APIs + cost money to re-hit and the data may be gone tomorrow. +- Aggregate before saving. Counts, rates, and shapes answer nearly every question these + investigations ask; this directory sits outside any repo and gets linked into tickets, so prefer a + distribution over a dump of rows carrying editor identities or user data. +- Rewrite only the FINDINGS.md you wrote and delete only files this session created — anything else + in the directory belongs to another investigation. +- Python work goes in an isolated venv inside this directory: `uv venv` if `uv` is installed, + otherwise `python3 -m venv`. + +## Step 1 — pin down the report + +Establish these five facts. Derive what you can — the alert or issue carries the symptom and its +volume, stratifying the data gives you the surface and locale, a live request tells you whether it +is still happening. Ask only for what is genuinely undiscoverable, in one batch, and keep working +while you wait. + +| Fact | Why it matters | +|---|---| +| Exact symptom, verbatim | "Empty section" and "wrong items in section" have disjoint causes | +| Who noticed, and how | Alert / editor report / spotted by hand — sets what evidence exists | +| First and last seen, **with timezone** | Anchors the timeline; reports usually arrive late | +| Surface, locale, section, client version | The stratum is very often the diagnosis | +| Still happening right now? | Live reproduction vs. historical forensics | + +Then confirm it yourself. + +**If it is still happening, capture the perishable evidence first** — a live +`curated-recommendations` request, current logs, current per-section counts — and save the raw +responses under `api_responses/`. Build the step-2 prior while those probes are in flight; history +keeps, live signal does not. + +**An error-rate signal is not yet a symptom.** Measure the erroring stage's *output* to fix the +blast radius — items per section and freshest item timestamp per surface for assembly, a live Merino +request for anything client-facing — so you can say whether Firefox clients are affected at all. An +error floor that never reaches the output is a not-incident; errors flat with output at zero is +worse than the alert says. When the report is a Sentry alert, resolve it to a concrete error and a +volume, and decompose the issue by error message before trusting its title or its trend — the +alert-quality checks are in `references/failure-modes.md`. + +**If you cannot reproduce it, that is a result, not a blocker.** Record the exact attempt — surface, +locale, time, request, what you saw instead — then switch the question to why the reporter saw it +and you do not: a stratum you did not hit, a window that has closed, a cache, a client version, or a +monitor measuring something other than what it claims. Classify as `not-incident` only after that +second question has an answer; unconfirmed is not refuted. + +A report relayed from an editor is ambiguous between what editors see in curation-admin-tools and +what clients see on the surface. Probe both in parallel and state which population you confirmed. + +The moment you confirm client-visible impact that is still happening, state it in one line — what, +how big, since when — and keep investigating. + +## Step 2 — establish what "normal" is + +You do not know what is currently normal in this system, so you have no basis on which to reject +your own inference. Build that prior yourself before asking for it — most of it is in reach: + +- **Which surfaces are serving** — surfaces present and not disabled in the section data, or a live + Merino request for the locale in question. +- **Whether the crawl is feeding them** — per-surface, per-source volume over the last day from the + crawl data. These two sets differ, and a crawled surface with no sections is not automatically a + bug. +- **What shipped recently** — recent commits in the service repos, releases in Sentry, timestamps on + model or config artifacts. +- **Whether this is normally noisy or seasonal** — a trailing profile of the same metric, by day of + week. Do not ask; compute it. +- **Whether it has happened before** — search Sentry for the same signature, and the service repos' + GitHub issues. + +What is left is genuinely in the developer's head, and it is worth one short, non-blocking ask: +anything in flight that would not show up in a repo or a dashboard, and whether a postmortem or +incident register exists for this area. An existing write-up can answer in a paragraph what costs +hours to reconstruct, so ask early — then carry on without waiting. + +Record what you derive and what they tell you in FINDINGS.md, keeping the two apart. When a later +query contradicts the prior, treat your own inference as the suspect first, and say so. + +## Step 3 — enumerate hypotheses, then probe them in parallel + +List every plausible explanation before testing any of them. Widen the list from +`references/failure-modes.md` — its failure-class tells cover classes the symptom table below does +not, and "Why these failures are silent" names the mechanisms that make a broken pipeline return a +valid-looking empty result rather than an error. + +| Symptom | Likely layer | Probe | +|---|---|---| +| Recommendations 500 / timeout / stale | Merino, or its upstream corpus API | Sentry + live `curated-recommendations` request | +| Section empty or short | ML section assembly → corpus → Merino | Active items per section; freshest item timestamp per section | +| Section missing entirely | Section config / section manager | Section rows for the surface; section-manager errors | +| Wrong / stale / duplicate items | Corpus content or ranking | Trace one item end to end by `corpusItemId` | +| Bad title, image, author, date | Crawl + hydration | Compare the stored item against a live re-extraction | +| Whole publisher or domain absent | Crawl discovery / domain block | Per-domain counts over time, then the vendor's own status data | +| Errors up, no reported user impact | The erroring stage itself | Assert that stage's output invariant downstream, then decompose the errors | +| Editor-facing error or lockout | admin-api → corpus API | Sentry for the editor-facing services; reproduce the mutation | +| Metrics or dashboard wrong | ETL / derived tables | Compare the derived table against its source | + +Before firing, write beside each hypothesis the result that would kill it. Then issue the probes as +independent tool calls in a single batch, backgrounding anything slow and handing a line that needs +several dependent steps to a subagent so the batch still returns together. Cap the first wave at +four or five cheap, independent probes, and hold metered calls and large scans for the second wave. +Name each `queries/` and `results/` pair after its hypothesis, so a result cannot be attributed to +the wrong line. + +Two moves consistently break cases open: **leave the tool you started in** — when a source stops +yielding, measure the same thing somewhere else, since the failure is often invisible in the plane +you began with — and **compare against sibling strata**, which localises a fault faster than reading +code. Both are expanded in `references/failure-modes.md`. + +## Step 4 — build the timeline + +An explicit `timestamp (UTC) | observation | source` table. + +- Normalise every timestamp to UTC and label it. Timezone mismatch is a common source of phantom + gaps, and a subtraction error can manufacture an outage that never happened. +- Extend the window to weeks, not hours — these failures are frequently much older than the report. + Widen freely on the cheap planes; on a billed plane, widen only after a narrow window has shown + you something. +- Distinguish *first occurrence* from *first noticed*, and state both. +- Line the window up against the deploys and config changes from step 2. +- Treat retention limits as limits: a "first seen" date can be the edge of a retention window rather + than onset. See `references/data-sources.md`. + +## Step 5 — stratify before concluding anything + +An aggregate that looks fine is the normal way these problems hide. Slice every metric by +**domain, locale, region, surface, section, experiment branch, and client/addon version**, and look +for a single stratum at zero or down sharply against its own trailing median rather than against +yesterday. When the report says "some users" with no stratum attached, treat experiment branch, +region, and rollout state as the first cuts — some surfaces are reachable only through experiment +enrolment, so an enrolled/unenrolled split is invisible to a locale slice. + +Strata are not interchangeable, and the volumes are strongly seasonal; both traps, and the +per-surface source mix, are detailed in `references/data-sources.md`. Compare each stratum against +its own history. + +**Liveness is not health.** "The job ran successfully" is compatible with total data loss in every +one of these pipelines. Count what came out, compare it against what went in, and never accept a +green run as evidence. + +If the measurements exonerate the backend, stop at the service boundary. Read far enough to name the +owning team and the contract that is being broken, then hand it over rather than continuing into a +system this skill does not cover. A second, unrelated anomaly you trip over on the way is a one-line +note plus a task, not a second investigation. + +## Step 6 — try to break your own conclusion + +Before writing anything down as fact, attack it: + +- Check the boring explanations first: timezone; a filter in your own query; a `LIMIT` silently + truncating; an unrepresentative code path; sampling; a column that does not mean what its name + says; a partially-launched feature. +- Then reach for the falsification moves in `references/failure-modes.md` — the code path, exact + enumeration, repetition, the real limit, the config's history — picking the ones that would break + your specific mechanism. +- **Measure the effect, not just the mechanism.** A guard that provably runs is not evidence that + the outcome is correct; verify the outcome separately. +- Cross-check against a second, independent source. One source is a hypothesis. +- Re-read the step-2 prior. If your conclusion implies that something the prior says is working is + broken, re-check your own measurement first — a filter in your query, a stratum mismatch, or a + wrong surface identifier is the likelier explanation. If the measurement survives that re-check, + the measurement wins: state the contradiction explicitly in FINDINGS.md, note it to the developer + in one line, and keep going. + +Mark every claim **verified**, **inferred**, or **refuted**. Keep the refuted ones in the document; +they stop the next person re-running them. + +## Step 7 — write FINDINGS.md + +```markdown +# — investigation + +**Status:** investigating | root cause identified | blocked +**Class:** outage | degradation | data-quality defect | user-facing bug | near-miss | not-incident +**Severity:** critical | high | medium | low +**Affected:** Firefox clients | editors | both — with a magnitude, not an adjective + +## Summary +Three sentences: what broke, since when, what the user-visible effect is. + +## Timeline (UTC) +| Time | Observation | Source | + +## Prior +**Derived** — per-surface volumes, recent deploys and model artifacts, trailing profile, prior +occurrences. Each with the query or link. +**Reported** — verbatim answers from the developer, or `asked
-`, e.g. -`~/investigation/2026-jul29-empty-de-sections`. Reuse a directory only when its FINDINGS.md is about -the same symptom. Say which directory you chose; do not ask. +Each investigation gets its own subdirectory inside one root directory that the developer keeps their +investigations in. Do not assume where that root is. Establish it in this order: + +1. **Look in user memory.** `~/.claude/CLAUDE.md` is where the root is recorded, because it is the + only memory scope that applies across every repository you might be launched from. If it names a + root, use it and do not ask. +2. **Otherwise ask, once, before your first write.** Where the developer keeps investigations cannot + be derived, and guessing puts files somewhere they did not choose. Ask it as a single question at + the start, and run the step-1 probes while you wait rather than idling. +3. **Then record it.** Append one line to `~/.claude/CLAUDE.md`, creating that file if it does not + exist: `Investigations live in (one subdirectory per investigation).` Tell the developer + you saved it, so they know they will not be asked again. Auto memory is the wrong home for this: + it is keyed to the repository you were launched from, and investigations get started from + whichever service repo is to hand. + +Then create this investigation's subdirectory inside that root. If the root already holds +investigations, follow the naming they use; otherwise `
-`, e.g. +`jul29-empty-de-sections`. Say which directory you created. Reuse an existing one only when its +FINDINGS.md is about the same symptom. - `FINDINGS.md` — one living document. Create it from the step-7 skeleton **before your first probe**; steps 2, 4 and 6 write into it as they go, so step 7 is a final pass over a document that From 346a75af3493867a0744302bf56a52ba13d5240e Mon Sep 17 00:00:00 2001 From: Matt Miermans Date: Thu, 30 Jul 2026 14:59:10 -0400 Subject: [PATCH 10/51] newtab/hnt-backend-investigation: add Slack MCP and channel posting rule --- plugins/newtab/README.md | 1 + .../skills/hnt-backend-investigation/SKILL.md | 22 +++++++++++++++---- .../references/data-sources.md | 22 ++++++++++++++++--- 3 files changed, 38 insertions(+), 7 deletions(-) diff --git a/plugins/newtab/README.md b/plugins/newtab/README.md index 9674925..5feec90 100644 --- a/plugins/newtab/README.md +++ b/plugins/newtab/README.md @@ -24,5 +24,6 @@ Use this skill when a Sentry alert fires, an editor reports something broken, or - a MySQL client, plus Mozilla VPN - `uv` or `python3` - the Sentry MCP server +- the Slack MCP server, for `#hnt-dev-be-alerts` Anything unreachable is raised as a task and reported under "Could not measure". diff --git a/plugins/newtab/skills/hnt-backend-investigation/SKILL.md b/plugins/newtab/skills/hnt-backend-investigation/SKILL.md index 623506c..ebd5ed0 100644 --- a/plugins/newtab/skills/hnt-backend-investigation/SKILL.md +++ b/plugins/newtab/skills/hnt-backend-investigation/SKILL.md @@ -54,8 +54,9 @@ pick the sensible option, say which one you picked, and keep going. Ask a blocki when proceeding on any assumption would waste the whole investigation. When a source you cannot reach looks promising, raise it **once** as a pending task carrying the -whole errand in its own text, prefixed `DEV:` so it reads as theirs rather than yours — e.g. -`DEV: connect to VPN so corpus MySQL is reachable — settles whether the DE items exist at all`. +whole errand in its own text, prefixed `User action:` so it reads as theirs rather than yours — e.g. +`User action: connect to VPN so corpus MySQL is reachable — settles whether the DE items exist at +all`. Give the exact command, not a description of the problem; for an interactive login use the `!` in-session form so the output lands here, e.g. `! aws --profile sso login`. If the answer lives in a dashboard @@ -156,7 +157,8 @@ A report relayed from an editor is ambiguous between what editors see in curatio what clients see on the surface. Probe both in parallel and state which population you confirmed. The moment you confirm client-visible impact that is still happening, state it in one line — what, -how big, since when — and keep investigating. +how big, since when — and keep investigating. If that deserves a heads-up in `#hnt-dev-be-alerts`, +follow the posting rule at the end of step 8. ## Step 2 — establish what "normal" is @@ -172,7 +174,8 @@ your own inference. Build that prior yourself before asking for it — most of i model or config artifacts. - **Whether this is normally noisy or seasonal** — a trailing profile of the same metric, by day of week. Do not ask; compute it. -- **Whether it has happened before** — search Sentry for the same signature, and the service repos' +- **Whether it has happened before** — search Sentry for the same signature, recent alerts in + `#hnt-dev-be-alerts`, and the service repos' GitHub issues. What is left is genuinely in the developer's head: anything in flight that a repo or a dashboard @@ -319,3 +322,14 @@ Any telemetry plane you could not reach, and the access that would unblock it. - **Leave the follow-up as a task**, owned by whoever will act: confirm the fix shipped and the metric actually recovered. State that you have done so rather than asking whether to. - Hand back a two-line verdict plus the FINDINGS.md path. Do not paste the document into chat. + +### Posting to `#hnt-dev-be-alerts` + +Updates worth sharing go to `#hnt-dev-be-alerts`. Search the channel first: if the investigation +started from an alert or message posted there in the last day or two, reply **in that thread**, so the +diagnosis stays attached to the alert people already saw. Only start a new thread when nothing recent +relates. + +**Ask before every single message, showing the exact text and where it will go.** Approval for one +post is not approval for the next, and this is a shared channel colleagues act on. If approval does +not come, keep the draft in FINDINGS.md and carry on; an unposted update is not a reason to stop. diff --git a/plugins/newtab/skills/hnt-backend-investigation/references/data-sources.md b/plugins/newtab/skills/hnt-backend-investigation/references/data-sources.md index c28afe2..1ced8f1 100644 --- a/plugins/newtab/skills/hnt-backend-investigation/references/data-sources.md +++ b/plugins/newtab/skills/hnt-backend-investigation/references/data-sources.md @@ -7,6 +7,7 @@ Reference for the `hnt-backend-investigation` skill. - The pipeline, and which repo owns each stage - Vocabulary: surface, locale, section, and the id to trace items by - Sentry — projects and the traps in reading them +- Slack — `#hnt-dev-be-alerts`, as alert history and as where updates go - Merino — live requests and GCP projects - BigQuery — the tables worth knowing and the traps in reading them - Curated corpus MySQL — access and schema shape @@ -73,10 +74,10 @@ alone will not decompose an umbrella fingerprint. **If no `mcp__sentry__` tools are available, the server is not set up in this session, and you should say so immediately.** Do not weigh this one against your other lines first; most symptoms here are error-shaped, and steps 1, 2 and 6 all read from Sentry, so an investigation without it is working -half-blind. Raise it as a `DEV:` task with the command: +half-blind. Raise it as a `User action:` task with the command: ``` -! claude mcp add --transport http sentry https://mcp.sentry.dev/mcp +! claude mcp add --scope user --transport http sentry https://mcp.sentry.dev/mcp ``` followed by `/mcp` to authenticate in the browser. Servers load when a session starts, so the tools @@ -106,6 +107,20 @@ Traps: - Absence of events is weak evidence: a process that dies at startup, or a job that drops items while reporting success, emits nothing. +## Slack + +`#hnt-dev-be-alerts` carries the backend alerts for this stack, which makes it two things at once: a +record of what has already fired, worth searching in step 2 before you conclude something is new, and +the place investigation updates go. Reachable through the Slack MCP server, whose tools are prefixed +`mcp__slack__`: + +``` +! claude mcp add --scope user --transport http slack https://mcp.slack.com/mcp +``` + +Reading it is free. Posting needs the developer's approval for **each** message, and prefers a reply +in the existing alert thread over a new one: the rule is at the end of step 8 in SKILL.md. + ## Merino Reproducing the client call is often the fastest confirmation of a client-visible symptom: @@ -273,8 +288,9 @@ Each is a short errand: give the developer the command, not a description of the | No billing project configured | Name one they can bill, usually `moz-fx-dev--sandbox`, or set it with `gcloud config set project ` | | Permission denied on a dataset or a Merino project | Request read access, or viewer on the project; say meanwhile whether the question is about payload shape, which stage can answer | | Zyte key missing from the environment | Create one at https://app.zyte.com/o/612928/zyte-api/api-access, `export ZYTE_API_KEY=` in the shell they launch from, and restart the session. Or have them run the single extraction and paste back the JSON, not the key | -| No `mcp__sentry__` tools at all | Run `! claude mcp add --transport http sentry https://mcp.sentry.dev/mcp`, then `/mcp` to authenticate; the tools appear after a session restart | +| No `mcp__sentry__` tools at all | Run `! claude mcp add --scope user --transport http sentry https://mcp.sentry.dev/mcp`, then `/mcp` to authenticate; the tools appear after a session restart | | Sentry connected but unauthenticated or scoped too narrowly | Run `/mcp` and authenticate for the `mozilla` org, or read back the issue's event counts broken down by error message | +| No `mcp__slack__` tools, and an update needs posting | Run `! claude mcp add --scope user --transport http slack https://mcp.slack.com/mcp`, then `/mcp` to authenticate; or post the drafted message to `#hnt-dev-be-alerts` themselves | | Editor-facing symptom needs an authenticated session | Reproduce the click themselves and report the exact error text and time | | The answer is in a dashboard you cannot reach | Open it, apply the specific filter you name, and read back the one number or shape you asked for | From 5e5e41464127984a9ee51f1972cfb98b0222ce3a Mon Sep 17 00:00:00 2001 From: Matt Miermans Date: Fri, 7 Aug 2026 09:26:26 -0700 Subject: [PATCH 11/51] newtab/hnt-backend-investigation: correct data traps, add missing telemetry planes --- plugins/newtab/README.md | 7 +- .../skills/hnt-backend-investigation/SKILL.md | 36 ++- .../references/data-sources.md | 273 ++++++++++++------ .../references/failure-modes.md | 80 +++-- 4 files changed, 264 insertions(+), 132 deletions(-) diff --git a/plugins/newtab/README.md b/plugins/newtab/README.md index 5feec90..dbf8418 100644 --- a/plugins/newtab/README.md +++ b/plugins/newtab/README.md @@ -12,11 +12,11 @@ Use this skill when adding cleanup comments to Nova-related changes. See [`skill ### `hnt-backend-investigation` -Diagnostic workflow for Home New Tab **backend** incidents, covering Merino, the curated corpus, admin-api, the article crawler, and the ML section pipeline. Its output is an evidence-backed `FINDINGS.md` carrying a root cause and quantified impact. +Diagnostic workflow for Home New Tab **backend** incidents, covering Merino, the curated corpus, admin-api, the article crawler, the ML section pipeline, and the New Tab data pipelines. Its output is an evidence-backed `FINDINGS.md` carrying a root cause and quantified impact. Use this skill when a Sentry alert fires, an editor reports something broken, or recommendations look empty, wrong, or stale. See [`skills/hnt-backend-investigation/SKILL.md`](skills/hnt-backend-investigation/SKILL.md) for details. -**Optional tools:** +**Optional tools.** Anything unreachable is raised as a task and reported under "Could not measure": - `gh` - `gcloud` and `bq` @@ -25,5 +25,4 @@ Use this skill when a Sentry alert fires, an editor reports something broken, or - `uv` or `python3` - the Sentry MCP server - the Slack MCP server, for `#hnt-dev-be-alerts` - -Anything unreachable is raised as a task and reported under "Could not measure". +- Zyte API keys, for reproducing an extraction or reading the vendor's stats diff --git a/plugins/newtab/skills/hnt-backend-investigation/SKILL.md b/plugins/newtab/skills/hnt-backend-investigation/SKILL.md index ebd5ed0..6309324 100644 --- a/plugins/newtab/skills/hnt-backend-investigation/SKILL.md +++ b/plugins/newtab/skills/hnt-backend-investigation/SKILL.md @@ -1,6 +1,6 @@ --- name: hnt-backend-investigation -description: Investigates Home New Tab backend errors, outages, and data-quality problems across Merino, the curated corpus, admin-api, the article crawler, and the ML section pipeline. Confirms the symptom independently, probes competing hypotheses in parallel, stratifies metrics, tries to break its own conclusion, then writes an evidence-backed FINDINGS doc with a root cause and quantified impact. Use when a Sentry alert fires, an editor reports something broken, or recommendations look empty, wrong, or stale. +description: Investigates Home New Tab backend errors, outages, and data-quality problems across Merino, the curated corpus, admin-api, the article crawler, the ML section pipeline, and the New Tab data pipelines. Confirms the symptom independently, probes competing hypotheses in parallel, stratifies metrics, tries to break its own conclusion, then writes an evidence-backed FINDINGS doc with a root cause and quantified impact. Use when a Sentry alert fires, an editor reports something broken, or recommendations look empty, wrong, or stale. For backend investigation, not in-tree browser/extensions/newtab frontend work. --- # New Tab (HNT) Backend Investigation @@ -37,8 +37,8 @@ query paired with a wrong inference, stated confidently. **Work several lines at once.** When more than one explanation is plausible, probe them in parallel rather than following the most attractive one to its end. Breadth is cheap in the unmetered planes — Sentry, repo reads, CloudWatch alarm history, one live Merino request — and tunnel vision is the -expensive failure. It is not free in BigQuery or Zyte: dry-run and select narrow columns before you -fan out. Keep a list of every line you considered and record which you dropped and why. +expensive failure. It is not free in BigQuery, Zyte, or CloudWatch Logs Insights: dry-run, narrow the +columns, and narrow the window before you fan out. Keep a list of every line you considered and record which you dropped and why. **Prefer measuring to reasoning.** Reproduce the request. Run the model on real inputs. Count the actual rows. Read the code path instead of assuming its behaviour. A plausible mechanism becomes a @@ -68,9 +68,9 @@ the single number or shape you need, and ask them to read it back. One standing exception to raising a source only when it looks promising: missing Sentry tools, which you flag on sight in step 1. -Restate it exactly once more — whichever comes first: they confirm, the unblocked lines run out, or -you are three probes into a line you had already judged weaker than the blocked one. That is the -second and last ask. If there is still no response, finish steps 7 and 8 with `Status: blocked`, +Restate it exactly once more — whichever comes first: the unblocked lines run out, or you are three +probes into a line you had already judged weaker than the blocked one. That is the second and last +ask, and it does not apply if they have already answered. If there is still no response, finish steps 7 and 8 with `Status: blocked`, name the one unblock under "Could not measure", and leave the access task pending. Stopping outright is a last resort: only when the blocked source is the only thing that can settle @@ -100,9 +100,11 @@ investigations, follow the naming they use; otherwise `
-`, e.g. `jul29-empty-de-sections`. Say which directory you created. Reuse an existing one only when its FINDINGS.md is about the same symptom. -- `FINDINGS.md` — one living document. Create it from the step-7 skeleton **before your first - probe**; steps 2, 4 and 6 write into it as they go, so step 7 is a final pass over a document that - already exists. Rewrite it in place, not as `FINDINGS-v2.md`. +- `FINDINGS.md` — one living document. Create it from the step-7 skeleton the moment the directory + exists, and in any case before you write a conclusion down; steps 2 through 6 write into it as they + go, so step 7 is a final pass over a document that already exists. If you are still waiting on the + root, run the step-1 probes anyway and hold their output until you have somewhere to put it. Rewrite + it in place, not as `FINDINGS-v2.md`. - `queries/` and `results/` — every query as a file, its output alongside under the same basename. - `api_responses/` — raw JSON from live calls. Save them even when they look boring; metered APIs cost money to re-hit and the data may be gone tomorrow. @@ -116,6 +118,9 @@ FINDINGS.md is about the same symptom. ## Step 1 — pin down the report +Before your first probe, read [references/data-sources.md](references/data-sources.md). It carries the +traps that silently return wrong answers, and several of them look like an outage. + Establish these five facts from the data. The alert or issue carries the symptom and its volume, stratifying gives you the surface and locale, and a live request tells you whether it is still happening. @@ -175,8 +180,9 @@ your own inference. Build that prior yourself before asking for it — most of i - **Whether this is normally noisy or seasonal** — a trailing profile of the same metric, by day of week. Do not ask; compute it. - **Whether it has happened before** — search Sentry for the same signature, recent alerts in - `#hnt-dev-be-alerts`, and the service repos' - GitHub issues. + `#hnt-dev-be-alerts`, and the service repos' GitHub issues. Where one of those is unreachable, record + it as unchecked rather than empty; "nothing similar has fired" is a claim about a channel you actually + read. What is left is genuinely in the developer's head: anything in flight that a repo or a dashboard would not show, and whether a postmortem or incident register already covers this area. Raise that @@ -225,6 +231,10 @@ An explicit `timestamp (UTC) | observation | source` table. - Line the window up against the deploys and config changes from step 2. - Treat retention limits as limits: a "first seen" date can be the edge of a retention window rather than onset. See `references/data-sources.md`. +- **The scheduling layer is not in UTC.** Scheduled dates and the assembly crons run in each surface's + own timezone, so a UTC comparison invents a one-day gap for part of every day on any surface offset + from UTC — worst for the Americas, and `en-US` is the largest surface. Convert per surface, and say + which timezone you used. ## Step 5 — stratify before concluding anything @@ -276,8 +286,8 @@ they stop the next person re-running them. # — investigation **Status:** investigating | root cause identified | blocked -**Class:** outage | degradation | data-quality defect | user-facing bug | near-miss | not-incident -**Severity:** critical | high | medium | low +**Incident type:** outage | degradation | data-quality defect | user-facing bug | near-miss | not-incident +**Severity:** critical | high | medium | low — anchored to the magnitude below, not chosen by feel **Affected:** Firefox clients | editors | both — with a magnitude, not an adjective ## Summary diff --git a/plugins/newtab/skills/hnt-backend-investigation/references/data-sources.md b/plugins/newtab/skills/hnt-backend-investigation/references/data-sources.md index 1ced8f1..256f266 100644 --- a/plugins/newtab/skills/hnt-backend-investigation/references/data-sources.md +++ b/plugins/newtab/skills/hnt-backend-investigation/references/data-sources.md @@ -5,14 +5,16 @@ Reference for the `hnt-backend-investigation` skill. ## Contents - The pipeline, and which repo owns each stage -- Vocabulary: surface, locale, section, and the id to trace items by +- Vocabulary: surface, locale, section, assembly, and the id to trace items by - Sentry — projects and the traps in reading them - Slack — `#hnt-dev-be-alerts`, as alert history and as where updates go -- Merino — live requests and GCP projects -- BigQuery — the tables worth knowing and the traps in reading them +- Merino — live requests, deployed revision, GCP projects, ranking inputs, the serve-stale cache +- BigQuery — corpus and crawl state, client telemetry, and the traps in reading them - Curated corpus MySQL — access and schema shape - The editor plane — admin-api and curated-corpus-api -- AWS — profiles, and what logs give you that Sentry cannot +- AWS — profiles, the SQS handoff, and what logs give you that Sentry cannot +- Assembly and cadence — what runs, where, and how often +- Experiment enrolment - Zyte — extraction API and the vendor's own stats - **Access requests** — the one-line ask for each gated source @@ -21,11 +23,16 @@ Access varies by developer. Verify a source is reachable before building a plan is not, follow "Don't stall on the developer" in SKILL.md — raise it once as a task and keep investigating, rather than stopping or silently substituting a weaker source. +If you are editing this file later: it holds non-derivable access facts and traps that silently +produce wrong answers. Worked incidents, current issue ids, canned queries, and symptom-to-cause +lookups belong nowhere in this skill. + ## The pipeline -**crawl / discovery → hydration (Zyte) → curated corpus → ML section assembly → curated-corpus-api → -client-api → Merino → Firefox New Tab.** Editors act on the corpus through curation-admin-tools → -admin-api → curated-corpus-api. Telemetry lands in BigQuery. +**crawl / discovery → hydration (Zyte) → curated corpus → ML section assembly → SQS → +corpus-scheduler / section-manager lambda → curated-corpus-api → client-api → Merino → Firefox New +Tab.** Editors act on the corpus through curation-admin-tools → admin-api → curated-corpus-api. +Telemetry lands in BigQuery. `client-api` is the Apollo federated router (`Pocket/pocket-monorepo`, `servers/client-api`) and it is easy to miss: it has **no Sentry project of its own in either organisation**, so a failure there @@ -36,12 +43,12 @@ environment, and the dev constant beside it is unused — so stage Merino reads | Repo | GitHub | Role | |---|---|---| | `merino-py` | mozilla-services/merino-py | Serves New Tab recommendations and Firefox Suggest | -| `content-monorepo` | Pocket/content-monorepo | Curated corpus, recommendations, section manager | +| `content-monorepo` | Pocket/content-monorepo | Curated corpus, recommendations, section manager, the SQS lambdas | | `content-ml-services` | mozilla/content-ml-services | Crawl, classification, section assembly (Metaflow, Cloud Functions) | | `pocket-monorepo` | Pocket/pocket-monorepo | client-api federated router, shared infrastructure | | `curation-admin-tools` | Pocket/curation-admin-tools | Editor-facing web app | | `admin-api` | Pocket/admin-api | Federated GraphQL gateway for the admin tools | -| `bigquery-etl` | mozilla/bigquery-etl | New Tab engagement and Merino recommendation ETL | +| `bigquery-etl` | mozilla/bigquery-etl | New Tab engagement and Merino export ETL, and the Airflow DAGs behind it | | `serverless-image-cache` | Pocket/serverless-image-cache | Thumbor image resize and cache | | `firefox` | mozilla-firefox/firefox | Client side of the contract (`browser/extensions/newtab`) | @@ -49,21 +56,28 @@ Locate clones rather than assuming paths: `find ~ -maxdepth 3 -type d -name .git -print0 2>/dev/null | xargs -0 -n1 dirname`. For a repo that is not cloned, read it through `gh api` or `gh search code`. -## Vocabulary: surface, locale, section +## Vocabulary: surface, locale, section, assembly A **surface** is one locale/market feed of the corpus, written `NEW_TAB_EN_US`. A **section** is a -topic row inside a surface. Merino's request `locale` is hyphenated (`en-US`) and the surface is -**derived** from language plus region — the mapping table is `SurfaceId` in -`merino/curated_recommendations/localization.py`, not a reformatting of the locale string. Some -surfaces are reachable only through experiment enrolment. Stratifying by locale and stratifying by -surface are therefore not the same slice. - -Merino appends `utm_source=firefox-newtab-` to item URLs (`get_utm_source` and -`update_url_utm_source` in `curated_recommendations/corpus_backends/utils.py`), so a URL taken from a -response will never match a stored `url`. Trace items by the stable id instead: the response's -`corpusItemId` is the corpus GraphQL item `id`, which lines up with `ApprovedItem.externalId` in -corpus MySQL and `approved_corpus_item_external_id` in BigQuery. Confirm that last hop on the first -item you trace rather than assuming it. +topic row inside a surface. **Assembly** is the ML stage that decides which corpus items sit in which +section for a surface; it runs as Metaflow flows in `content-ml-services` and reaches the corpus +through SQS, and the stage is called several things across these repos, so pin down which one a claim +refers to. + +Merino's request `locale` is hyphenated (`en-US`) and the surface is **derived** from language plus +region by `get_recommendation_surface_id` in `merino/curated_recommendations/utils.py`, which also +branches on experiment enrolment — it is not a reformatting of the locale string. The `SurfaceId` enum +itself lives in `merino/curated_recommendations/corpus_backends/protocol.py`. Stratifying by locale and +stratifying by surface are therefore not the same slice, and some surfaces are reachable only through +enrolment. The cheapest resolution is a live response: it echoes the surface it resolved to. + +Merino rewrites item URLs with `utm_source=firefox-newtab-` (`get_utm_source` +and `update_url_utm_source` in `curated_recommendations/corpus_backends/utils.py`), so a URL from a +response generally will not match a stored `url`. Do not reconstruct the parameter yourself: the +lookup covers only some surfaces, and where it has no entry the URL comes back unmodified. Trace items +by the stable id instead: the response's `corpusItemId` is the corpus GraphQL item `id`, which lines +up with `ApprovedItem.externalId` in corpus MySQL and `approved_corpus_item_external_id` in BigQuery. +Confirm that last hop on the first item you trace rather than assuming it. ## Sentry @@ -104,22 +118,26 @@ Traps: - Issue volume is dominated by long-standing error floors. Establish what was already there before attributing anything to today, and check whether a floor *changed* rather than whether it exists. - Decompose an issue by error message before trusting its title or trending it. -- Absence of events is weak evidence: a process that dies at startup, or a job that drops items - while reporting success, emits nothing. +- Absence of events is weak evidence: a process that dies at startup, a job that drops items while + reporting success, and a write path disabled by a config flag all emit nothing. ## Slack `#hnt-dev-be-alerts` carries the backend alerts for this stack, which makes it two things at once: a record of what has already fired, worth searching in step 2 before you conclude something is new, and -the place investigation updates go. Reachable through the Slack MCP server, whose tools are prefixed -`mcp__slack__`: +the place investigation updates go. Its tools are prefixed `mcp__slack__`. Install it through the +official plugin rather than a bare `mcp add`, because the server does not support dynamic client +registration and a plain add will connect-fail: ``` -! claude mcp add --scope user --transport http slack https://mcp.slack.com/mcp +/plugin install slack@claude-plugins-official ``` -Reading it is free. Posting needs the developer's approval for **each** message, and prefers a reply -in the existing alert thread over a new one: the rule is at the end of step 8 in SKILL.md. +The OAuth round trip needs a fixed local callback port, so it can collide with another session doing +the same thing at the same moment. Reading the channel is free. Posting needs the developer's approval +for **each** message, and prefers a reply in the existing alert thread over a new one: the rule is at +the end of step 8 in SKILL.md. With no Slack tools, alert history is **unchecked**, not empty — do not +record "nothing similar has fired" on the strength of a source you could not read. ## Merino @@ -130,73 +148,118 @@ optionally `experimentName` / `experimentBranch` to land on an experiment branch Firefox `User-Agent`. Save every response. First things to check on the payload: section count, items per section, presence of `followable` / `allowAds`, and the age of the newest item. -Keep it to the requests you need. If you find yourself writing a loop over locales or branches -against prod, query the telemetry instead. Stage answers payload-shape questions — but remember it -reads prod corpus data, so it cannot answer questions about prod corpus content. +`GET /__version__` returns the running commit and build URL. Use it before trusting a repo log: +a merged commit is not a deployed one, and a rollback is the change that most cleanly explains a +symptom that started or stopped on its own. The lambdas carry the same information as a `GIT_SHA` +environment variable. + +Keep live calls to the ones you need. If you find yourself looping over locales or branches against +prod, query the telemetry instead. Stage answers payload-shape questions only — it reads prod corpus +data — and its host is internal-only, so a request from outside the network returns 404 rather than an +auth error: `https://stage.merino.nonprod.webservices.mozgcp.net`. -GCP projects, for log reads, metrics, and GCS listings — confirm against -`merino/configs/production.toml` **and `merino/configs/stage.toml`** rather than trusting this table: +GCP projects, for log reads, metrics and GCS listings. Confirm these against the deployed values files +in the deployment repo rather than the in-repo TOMLs, which lag: | Environment | Project | |---|---| | Merino prod | `moz-fx-merino-prod-5de4`; some buckets in `moz-fx-merino-prod-1c2f` | -| Merino stage | `moz-fx-merino-nonprod-ee93`; also `moz-fx-merino-nonprod-db57` | +| Merino stage | `moz-fx-merino-nonprod-db57` (deployed); `moz-fx-merino-nonprod-ee93` is legacy and still named in `stage.toml` | +| Crawl and ML | `moz-fx-mozsoc-ml-prod`, with gen2 Cloud Functions `prod-crawl-handler`, `prod-page-crawler`, `prod-ingester`, `prod-parser` | When the logging API returns nothing useful, the BigQuery log sink for the same project usually does. That sink table is far larger than anything in the BigQuery list below — check whether it has a partition column and filter on it, aggregate inside the query rather than pulling rows, and `--dry_run` first. +**Ranking inputs arrive as GCS blobs on a timer, and a stalled producer is invisible everywhere else.** +Merino pulls `engagement/latest.json` and `priors/latest.json` from its exports bucket on a short +interval and feeds them to the Thompson-sampling ranker (see `[default.curated_recommendations.gcs]` +in `merino/configs/default.toml`). If the producing Airflow DAG in `bigquery-etl` stops, Merino keeps +serving with frozen ranking: it logs at INFO, raises nothing, and no row in any table below changes. +Compare the blob's update time against the producing DAG's current `schedule_interval` rather than +against a remembered cadence. + +**Merino serves stale on upstream failure, per pod.** When the corpus fetch fails and a warm cache +entry exists, it returns the stale entry *and* pushes the expiry out again, so the stale window has no +fixed bound while the upstream stays down. A corpus-api or client-api outage therefore reaches clients +as a successful 200 with old content, and only cold pods raise. One live request cannot refute an +upstream problem, and cannot establish its scope either. + ## BigQuery `bq` ships with the Google Cloud SDK and runs on your `gcloud` credentials, so a missing binary or an unauthenticated session is the first thing to rule out, ahead of any dataset permission: `gcloud auth list` shows whether there is an active account. The same credentials cover the `gcloud logging` and -GCS reads below. Confirm the billing project with a `--dry_run` before the first real query; a +GCS reads above. Confirm the billing project with a `--dry_run` before the first real query; a personal sandbox is usually `moz-fx-dev--sandbox`. Anything missing there is an access task, and authenticating is interactive so it belongs to the developer. +Corpus, section and crawl state: + | Table | What it is for | |---|---| | `moz-fx-mozsoc-ml-prod.prod_rss_news.rss_feed_items` | Articles discovered by crawl — per-surface, per-source (`PAGE` vs `RSS`) volume over time | -| `moz-fx-mozsoc-ml-prod.prod_articles.zyte_cache` | Hydrated article metadata keyed on `canonical_url` — what the pipeline believes an article says | +| `moz-fx-mozsoc-ml-prod.prod_articles.zyte_cache` | One row per hydration attempt, not per article — see the traps below before counting or joining it | | `moz-fx-data-shared-prod.snowflake_migration_derived.sections_v1` | Section existence, enable/disable state, and surface, as an event stream | | `moz-fx-data-shared-prod.snowflake_migration_derived.section_items_v1` | Which items sit in which section, and when each was last touched — the freshness check for a stalled section | -| `moz-fx-data-shared-prod.snowflake_migration_derived.corpus_items_current_v1` | Current-state view of corpus items — prefer this to reducing the event table yourself | -| `moz-fx-data-shared-prod.snowflake_migration_derived.approved_corpus_items` / `scheduled_corpus_items` | Maintained current-state views for approved and scheduled items | +| `moz-fx-data-shared-prod.snowflake_migration_derived.corpus_items_current_v1` | One row per corpus item, deduped — but it keeps the latest row even when that row is a removal, so filter status yourself | +| `moz-fx-data-shared-prod.snowflake_migration_derived.scheduled_corpus_items` | Scheduled items; one row per item in practice | + +Client-side telemetry, which is the only plane that answers "how many users" and the only one carrying +experiment branch or browser version: + +| Table | What it is for | +|---|---| +| `moz-fx-data-shared-prod.telemetry_derived.newtab_visits_v1` | The experiment and browser-version stratum (`experiments`, `browser_version`, locale, country). Very large and `requirePartitionFilter` is on | +| `moz-fx-data-shared-prod.firefox_desktop.newtab_content_live` | The live plane, minutes behind rather than a day — the only one that can confirm a symptom that is happening now | +| `moz-fx-data-shared-prod.firefox_desktop_derived.newtab_content_items_daily_v1` | Daily item-level impressions and clicks. Carries no locale and no experiment branch, and its version column is null for most volume, so do not use it for those cuts | -Confirm each table still exists before drawing a conclusion from an empty result. +The derived tables are T+1, so they cannot confirm a live symptom; reach for the live table when the +question is "is this happening right now". Confirm each object still exists before drawing a conclusion +from an empty result, and use `bq ls ` to settle whether something is a table or a view and +whether it is partitioned — a view has no partition column, so ordering a partition filter on one is a +query error. Traps that will silently give you a wrong answer: +- **`zyte_cache` is not keyed on `canonical_url`.** It holds one row per hydration attempt; a single + url can have hundreds. `rss_feed_items` is not unique on `canonical_url` either. So the natural + discovered-to-hydrated funnel join fans out several-fold and inflates every count in it. Dedupe + **both** sides to one row per url before joining, and note that the event-log warning below is about + a different set of tables — it does not make this one safe to count. - **There is no domain column on `rss_feed_items`.** Derive the domain from `canonical_url` (or `origin_url` for the page it was found on). A per-domain funnel is a computed grouping, not a lookup. -- **`source` is null for everything crawled before 2025-08-11** — 6.8M of 22.6M rows. A long +- **`source` is null for everything crawled before 2025-08-11**, several million rows. A long per-source series will show `PAGE` and `RSS` springing into existence on that date; that is the column being introduced, not the crawl changing. - **`crawled_date` and `published_date` are STRING; `crawled_at`, `published_at` and `loaded_at` are TIMESTAMP.** Use the timestamps for any time arithmetic. -- **Cost does not work the way you expect.** `rss_feed_items` (~23.6 GB) and `zyte_cache` (~128 GB) - are **unpartitioned and unclustered**, so a date predicate reduces nothing — only column selection - does. Never `SELECT *` on them and always `--dry_run` first. The `snowflake_migration_derived` - tables are day-partitioned on `happened_at` but do not require a partition filter, so supply one - yourself. +- **Cost does not work the way you expect.** `rss_feed_items` (tens of GB) and `zyte_cache` (over a + hundred GB) are **unpartitioned and unclustered**, so a date predicate reduces nothing — only column + selection does. Never `SELECT *` on them and always `--dry_run` first. The + `snowflake_migration_derived` tables are day-partitioned on `happened_at` but do not require a + partition filter, so supply one yourself. - **The `*_v1` event tables are event logs, not current state.** Rows accumulate per change, so a - plain `COUNT(*)` over-counts. Prefer the current-state views above; where you must reduce the log - yourself, take the latest row per id and check the inflation ratio rather than assuming it. -- **`sections_v1.source` is unusable for current sections** — null on 81% of all sections and on - **every** currently-active one. Section ownership (`MANUAL` vs `ML`) comes from + plain `COUNT(*)` over-counts. Where you must reduce the log yourself, take the latest row per id and + check the inflation ratio rather than assuming it. +- **`approved_corpus_items` is a filter, not a deduplication.** It excludes removed items but still + carries multiple rows per item, so it is not a current-state view despite the name. +- **`sections_v1.source` is unusable for current sections** — null on about four fifths of all + sections and on **every** currently-active one. Section ownership (`MANUAL` vs `ML`) comes from `Section.createSource` / `updateSource` / `deactivateSource` in corpus MySQL. A null `source` means unknown, never `MANUAL`. - **Surface identifiers differ by table.** Crawl data uses `en_US`; section data uses `NEW_TAB_EN_US`. Joining or comparing them naively produces empty results that look like an outage. -- **The crawl surface set and the serving surface set are not identical**, so a crawled surface with - no sections is not automatically a bug. +- **The crawl surface set, the served surface set, and the set of surfaces ML actually runs for are + three different sets**, so a surface missing from one of them is not automatically a bug. - **Source mix varies by surface** — some have no RSS-sourced content, some no page-crawled content. Judge each surface against its own history. - **Volume is strongly day-of-week seasonal.** Compare against a trailing median, never against yesterday. +- Row counts and byte sizes drift; treat any figure here as an order of magnitude and re-measure with + `bq show` when the number matters. - A zero in the crawl table is ambiguous between "found nothing" and "every request failed"; there is no attempt/error record to disambiguate it. @@ -209,15 +272,18 @@ an auth error is the usual symptom of being off it. Useful invocation guards: `--safe-updates` (caps returned rows at 1000 and aborts queries estimated to examine over a million) and `SET SESSION max_execution_time=10000` (10s server-side cap). The 1000-row cap **truncates silently**, so add an explicit `LIMIT` or raise the cap when you need a -complete set. +complete set. The million-row abort surfaces as `ERROR 1104`, and it is your own guard rather than +access or VPN — it fires on the first aggregate over `SectionItem`. Lift the examined-rows ceiling with +`SET SESSION sql_big_selects=1` or `--max-join-size`, which is a different control from the row cap. Schema `curation_corpus`. The tables that matter: `ApprovedItem` (the corpus itself, keyed by `externalId` and `url`), `SectionItem` and `Section` (placement and section config — `Section` also carries `createSource` / `updateSource` / `deactivateSource`, the authoritative ML-vs-manual -ownership), `ScheduledItem` (surface scheduling), `RejectedCuratedCorpusItem`, and the -`PublisherDomain` / `TrustedDomain` / `ExcludedDomain` domain lists. `SectionItem` runs into the -millions of rows and `ApprovedItem` into the hundreds of thousands — check indexes with `SHOW INDEX` -before filtering, since several obvious filter columns are unindexed. +ownership), `ScheduledItem` (surface scheduling, and its `scheduledDate` is a zoneless calendar date in +the surface's own timezone), `RejectedCuratedCorpusItem`, and the `PublisherDomain` / `TrustedDomain` / +`ExcludedDomain` domain lists. `SectionItem` runs into the millions of rows and `ApprovedItem` into the +hundreds of thousands — check indexes with `SHOW INDEX` before filtering, since several obvious filter +columns are unindexed. ## The editor plane @@ -233,49 +299,79 @@ content-monorepo infrastructure runs in AWS. Discover profiles with `grep -E '^\[profile' ~/.aws/config` and confirm one works with `sts get-caller-identity`; match prod against dev to the environment you are investigating. -CloudWatch Logs Insights is frequently the source that cracks a case — it will give you total -operation counts and per-error-type breakdowns that Sentry structurally cannot, because Sentry only -sees what was raised. It bills per GB scanned per query, so pass the narrowest +**The ML-to-corpus handoff is a queue, and it is where a day's candidates go missing quietly.** The +corpus-scheduler and section-manager lambdas are each fed by SQS with concurrency of one and a batch +size of one, and the shared construct gives each queue a dead-letter queue named after it. Check queue +depth, in-flight count, oldest-message age, and DLQ depth before concluding that ML produced nothing: +a backlog and a drained DLQ look identical in every corpus table. Two mechanics worth knowing: a +message whose handler outlives the queue's visibility timeout is redelivered while still in flight and +can exhaust its receive count without any error being raised, and the write path is gated by an +`ALLOWED_TO_SCHEDULE` flag that makes the lambda return normally while writing nothing. ML also has one +path that bypasses the queue and calls admin-api directly, so do not describe the handoff as +queue-only. + +CloudWatch Logs Insights is frequently the source that cracks a case for these lambdas — it will give +you total operation counts and per-error-type breakdowns that Sentry structurally cannot, because +Sentry only sees what was raised. It bills per GB scanned per query, so pass the narrowest `--start-time`/`--end-time` that could answer the question and `stats`-aggregate rather than dumping `fields`. Also useful: alarm history (transition timestamps and state-reason margins), and pulling an -anomaly band itself as a metric-math series to compare its predicted centre against reality. +anomaly band itself as a metric-math series to compare its predicted centre against reality. The crawl +runs in GCP, not AWS, so its equivalent plane is Cloud Logging or the BigQuery log sink. SSO sessions expire and the login is interactive, so it has to be the developer: see the table below. +## Assembly and cadence + +Freshness thresholds are a common thing to want and a common thing to invent. Derive the intended +cadence from the `@schedule` decorators on the Metaflow flows in `content-ml-services`, and record the +interval you used in FINDINGS.md. Two traps sit in that derivation: + +- The cron is wrapped so that it **only takes effect when a deploy-time environment variable is set**; + otherwise the decorator receives a deliberately impossible date that never fires. Reading the literal + cron out of the source and reporting "runs daily at 11:00" can be wrong twice over. +- The decorator also passes a per-surface timezone, so the schedule is surface-local, not UTC. + +Which flows exist at all is per-locale, listed in the deployed-locale and deployed-flow JSON manifests +in the same repo. A surface with no deployed flow is a third possibility alongside a crawl gap and a +serving gap. + +## Experiment enrolment + +Step 5 makes experiment branch one of the first cuts, and the branch names are not in any of the +tables above. The Experimenter API lists live and recent experiments without authentication: +`https://experimenter.services.mozilla.com/api/v6/experiments/`. Use it to get the real slug and branch +names before slicing telemetry, rather than inventing them or asking. + ## Zyte Two separate APIs, both **metered — every call costs money**. Save all responses; never bulk-crawl to -satisfy curiosity. Check for a key with `[ -n "$ZYTE_API_KEY" ] && echo present`. Keys are created at -https://app.zyte.com/o/612928/zyte-api/api-access. Reference the key as `$ZYTE_API_KEY` in anything -you save, so its value never lands in a query file or the transcript. - -**Extraction API** (`ZYTE_API_KEY`) reproduces what the crawler saw for a URL. Request `article` for -a single page or `articleList` for an index page — not both — and put `extractFrom: -"httpResponseBody"` inside `articleOptions` / `articleListOptions` to skip the browser. Check -**`statusCode` and the extraction probability before anything else**: a non-2xx, a bot wall, or a -"JavaScript is disabled" page still returns a populated object that reads like success. Probability -lives at `article.metadata.probability` for a single page and per item at -`articleList.articles[].metadata.probability` — `articleList.metadata` carries only +satisfy curiosity. Check for a key with `[ -n "$ZYTE_API_KEY" ] && echo present`. Reference keys by +variable name in anything you save, so no value lands in a query file or the transcript. + +**Extraction API** (`ZYTE_API_KEY`, created at https://app.zyte.com/o/612928/zyte-api/api-access) +reproduces what the crawler saw for a URL. Request `article` for a single page or `articleList` for an +index page — not both — and put `extractFrom: "httpResponseBody"` inside `articleOptions` / +`articleListOptions` to skip the browser. Check **`statusCode` and the extraction probability before +anything else**: a non-2xx, a bot wall, or a "JavaScript is disabled" page still returns a populated +object that reads like success. Probability lives at `article.metadata.probability` for a single page +and per item at `articleList.articles[].metadata.probability` — `articleList.metadata` carries only `dateDownloaded`. Compare `canonicalUrl` against the URL you requested; cross-domain canonicals pull unapproved domains into the corpus. -**Stats API** (`ZYTE_SECRET_KEY`, `https://zyte-api-stats.zyte.com/api/stats`) is the vendor's own -view of your traffic: per-domain response-code distribution over time. `organization_id` is -**required** and is `612928`; auth is HTTP basic with the secret key as the username and an empty -password, so a bare GET on the endpoint fails validation. Only `groupby_time` -and `groupby_domain` group; `response_codes` is a *filter*, not a grouping, and -`include_domain_health=true` is rejected without `groupby_domain=true`. This is the right source for -"did this domain start failing, and when" — your own logs will not show it if the pipeline discards -non-allowlisted status codes. - -Freshness thresholds are a common thing to want here and a common thing to invent. Derive the -intended cadence from the assembly schedule in `content-ml-services` and the corpus-scheduler-lambda -trigger config rather than guessing, and record the interval you used in FINDINGS.md. +**Stats API** (`https://zyte-api-stats.zyte.com/api/stats`) is the vendor's own view of your traffic: +per-domain response-code distribution over time. It takes a **different credential** — the Zyte +dashboard API key from the organisation's settings page, explicitly not the Zyte API key above — as the +HTTP basic username with an empty password, so the extraction key will be rejected here. +`organization_id` is **required** and is `612928`. Only `groupby_time` and `groupby_domain` group; +`response_codes` is a *filter*, not a grouping, and `include_domain_health=true` is rejected without +`groupby_domain=true`. This is the right source for "did this domain start failing, and when" — your +own logs will not show it if the pipeline discards non-allowlisted status codes. ## Access requests -Raise one of these only when that source is one of your strongest current lines, then keep working. -Each is a short errand: give the developer the command, not a description of the problem. +Raise one of these when the source looks promising, then keep working. The two Sentry rows are the +exception: raise those on sight, per step 1. Each is a short errand, so give the developer the command +rather than a description of the problem. | Blocked source | Ask them to | |---|---| @@ -287,10 +383,11 @@ Each is a short errand: give the developer the command, not a description of the | `gcloud` installed but not authenticated | Run `! gcloud auth login`, and `! gcloud auth application-default login` as well if you need the Python client libraries | | No billing project configured | Name one they can bill, usually `moz-fx-dev--sandbox`, or set it with `gcloud config set project ` | | Permission denied on a dataset or a Merino project | Request read access, or viewer on the project; say meanwhile whether the question is about payload shape, which stage can answer | -| Zyte key missing from the environment | Create one at https://app.zyte.com/o/612928/zyte-api/api-access, `export ZYTE_API_KEY=` in the shell they launch from, and restart the session. Or have them run the single extraction and paste back the JSON, not the key | +| Zyte extraction key missing | Create one at https://app.zyte.com/o/612928/zyte-api/api-access, `export ZYTE_API_KEY=` in the shell they launch from, and restart the session. Or have them run the single extraction and paste back the JSON, not the key | +| Zyte Stats key missing | Issue a **dashboard** API key from the Zyte organisation settings page and export it; the extraction key will not authenticate against the Stats API | | No `mcp__sentry__` tools at all | Run `! claude mcp add --scope user --transport http sentry https://mcp.sentry.dev/mcp`, then `/mcp` to authenticate; the tools appear after a session restart | | Sentry connected but unauthenticated or scoped too narrowly | Run `/mcp` and authenticate for the `mozilla` org, or read back the issue's event counts broken down by error message | -| No `mcp__slack__` tools, and an update needs posting | Run `! claude mcp add --scope user --transport http slack https://mcp.slack.com/mcp`, then `/mcp` to authenticate; or post the drafted message to `#hnt-dev-be-alerts` themselves | +| No `mcp__slack__` tools | Run `/plugin install slack@claude-plugins-official`, then `/mcp` to authenticate; or post the drafted message to `#hnt-dev-be-alerts` themselves | | Editor-facing symptom needs an authenticated session | Reproduce the click themselves and report the exact error text and time | | The answer is in a dashboard you cannot reach | Open it, apply the specific filter you name, and read back the one number or shape you asked for | diff --git a/plugins/newtab/skills/hnt-backend-investigation/references/failure-modes.md b/plugins/newtab/skills/hnt-backend-investigation/references/failure-modes.md index d0e8eb3..bf03fcf 100644 --- a/plugins/newtab/skills/hnt-backend-investigation/references/failure-modes.md +++ b/plugins/newtab/skills/hnt-backend-investigation/references/failure-modes.md @@ -1,8 +1,12 @@ # How New Tab (HNT) backend failures behave Reference for the `hnt-backend-investigation` skill: the shapes these failures take, the invariants -to check, and the moves that kill a wrong hypothesis. Read this before you take any hypothesis -seriously. +to check, and the moves that kill a wrong hypothesis. SKILL.md names the moments to read it; the +alert audit below applies as soon as a report comes from an alert. + +If you are editing this file later: it holds generalised mechanisms and techniques. It is deliberately +not a symptom-to-cause lookup, and it deliberately carries no worked incidents, issue ids, or canned +queries. Adding any of those changes what the file is for. ## Contents @@ -11,7 +15,7 @@ seriously. - Why these failures are silent — the mechanisms that turn a failure into a valid-looking result - Falsification moves that work — what to reach for when a story feels too clean - Structural gaps to report rather than infer past -- Invariants to check — hard checks, and the class each one points to +- Invariants to check — the hard ones, the ones relative to a stratum's own history, and the class each points to ## Failure classes @@ -28,11 +32,6 @@ defect · client or product defect · capacity near-miss Observability defects are expanded in the next section, and the invariants at the end of this file name the class each one points to. -A calibration note: symptoms that originate from a monitor's output or from a prevailing worry are -the ones that most often turn out not to be real, while content a human actually observed usually is. -This is a reason to confirm the signal independently in both directions, not a reason to dismiss -either. - ## If it came from an alert, audit the alert Alert quality is its own failure class, and worth a few minutes before chasing the system. This is an @@ -46,7 +45,8 @@ addition to confirming the symptom independently, not a substitute for it. margins under a percent, mean the band, not the system. - **Counting artefact** — a panel that counts per open period rather than per calendar day can be off by an order of magnitude. -- **Unroutable** — if you cannot tell prod from staging at a glance, fix that first. +- **Unroutable** — if you cannot tell prod from staging from the alert alone, establish which one it + is before reading anything else into it. ## Why these failures are silent @@ -61,6 +61,10 @@ gets converted into a valid-looking empty or partial result. Expect these: - An `INNER JOIN` against a frozen upstream catalog shedding rows quietly instead of failing. - A process that dies during startup, which emits no error event at all. - A bulk write with no row-count guard. +- A write path disabled by a config flag, so the job runs, returns normally, and writes nothing. +- A queued handoff backing up or dead-lettering, which looks identical to the producer sending nothing. +- A cache serving stale content on upstream failure and extending its own expiry, so an outage + upstream reaches clients as a successful response with old data. - An aggregate quality gate passing while one class collapses beneath it. - An alarm that treats missing data as "missing" rather than as a breach, sitting against an emitter that publishes nothing at zero — so it goes *quiet* during a total stop. @@ -88,6 +92,12 @@ too clean. limit. - **Go to the other side's telemetry.** A vendor's or upstream's own status data will show what your logs structurally cannot. +- **Measure the same thing in a different plane.** Each plane here is blind in a particular way, and + the pairs are what break structurally-invisible cases: Sentry sees only what was raised, while + CloudWatch Logs Insights sees every operation including the ones that failed quietly; the BigQuery + log sink sees requests the logging API will not surface; the vendor's stats see the responses your + own pipeline discarded before logging them; client telemetry sees what users got when every backend + table looks healthy. - **Compare against sibling strata** at the same layer — the healthy peers localise the fault faster than reading code does. - **Design a canary that would falsify** the hypothesis rather than a test that would confirm it. @@ -105,32 +115,48 @@ These make certain questions unanswerable from stored data alone. When one block - No lifecycle metadata or tombstones on the crawl target list, so an intentional retirement is indistinguishable from an outage. - No persisted ML filter funnel, so per-stage drop-off cannot be reconstructed afterwards. -- No queryable deployment history to line a timeline up against. +- No queryable deployment history. The running revision is available per service — Merino's + `/__version__`, the lambdas' `GIT_SHA` — but there is no log of past deploys to line a timeline up + against, so correlate against the current revision and the repo history instead. - No status code or extraction-probability stored alongside cached hydration results, so a stored row does not imply a successful extraction. ## Invariants to check -Hard equalities and floors rather than statistical thresholds, so a violation is unambiguous rather -than a matter of degree. Assert the ones that bear on your hypotheses against the data: a broken one -localises the fault immediately, and a holding one eliminates a line cheaply. +Checks worth asserting against the data when they bear on your hypotheses: a broken one narrows the +search, and a holding one eliminates a line cheaply. Read the two groups differently. The first group +is a hard equality or a structural fact, so a violation means something is wrong. The second group is +relative to the stratum's own history, so a violation means *look here*, not *this is the fault* — +healthy production violates a naive absolute version of every one of them. + +Hard: | Invariant | Class it points to | |---|---| -| Articles/hour > 0 per surface **per source**, judged against that surface's own mix | Upstream unavailability; capacity cliff | -| Active items per fixed-size section == configured N | Silent validation rejection; model failure | -| `items_written == items_received` per job invocation | Silent validation rejection | -| Input-validation reject rate == 0 | Silent validation rejection | -| Newest active item age per section within that surface's assembly interval | Model or routing failure; capacity cliff | -| Vendor non-2xx share per domain below a floor | Upstream unavailability; extraction defect | -| Content-shape ratio per domain within bounds (empty-body %, title length) | Extraction-quality defect | -| Per-domain funnel positive at every stage: discovered → hydrated → approved | Upstream unavailability; filter misconfiguration | -| Canonical domain ∈ approved list | Data-model / identity defect | -| Per-class model recall above a floor before publish | Model or routing failure | -| Row-count delta on key tables within bounds | Unscoped bulk operation | -| Every bound parameter in shared SQL supplied by every caller | Analytics correctness | -| Consumption below documented quota, alerting on the ratio | Capacity near-miss | -| The alarm itself emits a value at zero | Observability defect | +| Canonical domain ∈ approved list | Data-model or identity defect | +| Every bound parameter in shared SQL supplied by every caller | Analytics correctness defect | +| A derived dimension agrees with the entity it describes | Analytics correctness defect | +| The alarm itself emits a value at zero rather than nothing | Observability defect | +| Consumption against documented quota | Capacity near-miss | + +Relative to the stratum's own trailing behaviour: + +| Check | Class it points to | +|---|---| +| Articles per **day** > 0 per surface per source, plus the hourly count inside that stratum's own trailing range. An hour at zero is normal for low-volume strata and for a newly launched surface | Upstream unavailability; resource exhaustion or capacity cliff | +| Items served per section against that section's trailing count. There is no single configured size — the served count varies by layout and by request, so do not assert an equality | Silent validation rejection; ML model or routing failure | +| `items_written` against the job's own trailing ratio, not against `items_received`. Pipelines here dedupe by design, so a healthy ratio is far below one | Silent validation rejection | +| Input-validation reject count against its trailing floor. A steady non-zero floor is normal for third-party content; a step change is not | Silent validation rejection | +| Newest active item age per section against that surface's assembly interval | ML model or routing failure; resource exhaustion or capacity cliff | +| Vendor non-2xx share per domain below a **ceiling** | Upstream unavailability; vendor extraction-quality defect | +| Content-shape ratio per domain within bounds (empty-body share, title length) | Vendor extraction-quality defect | +| Per-domain funnel positive at every stage: discovered → hydrated → approved | Upstream unavailability; filter or threshold misconfiguration | +| Row-count delta on key tables within bounds | Unscoped bulk database operation | +| Queue depth, oldest-message age and dead-letter depth against their trailing values | Resource exhaustion or capacity cliff; silent validation rejection | + +For the relative group, name the plane you measured in: Cloud Logging or the BigQuery log sink for the +crawl and the GCP jobs, CloudWatch Logs Insights for the AWS lambdas, and client telemetry for anything +expressed in users. Stratify anything you assert. A rule that holds globally can be false for one surface, because the surfaces do not all draw content the same way. From 116454d2889bf09240ab29e2b5f29a5555e0dccd Mon Sep 17 00:00:00 2001 From: Matt Miermans Date: Fri, 7 Aug 2026 09:44:09 -0700 Subject: [PATCH 12/51] newtab/hnt-backend-investigation: cover New Tab features beyond recommendations --- plugins/newtab/README.md | 4 +- .../skills/hnt-backend-investigation/SKILL.md | 67 +++---- .../references/data-sources.md | 184 ++++++++++-------- .../references/failure-modes.md | 10 +- 4 files changed, 152 insertions(+), 113 deletions(-) diff --git a/plugins/newtab/README.md b/plugins/newtab/README.md index dbf8418..8f4123b 100644 --- a/plugins/newtab/README.md +++ b/plugins/newtab/README.md @@ -12,9 +12,9 @@ Use this skill when adding cleanup comments to Nova-related changes. See [`skill ### `hnt-backend-investigation` -Diagnostic workflow for Home New Tab **backend** incidents, covering Merino, the curated corpus, admin-api, the article crawler, the ML section pipeline, and the New Tab data pipelines. Its output is an evidence-backed `FINDINGS.md` carrying a root cause and quantified impact. +Diagnostic workflow for Home New Tab **backend** incidents, feature by feature: content recommendations (Merino, the curated corpus, admin-api, the article crawler, the ML section pipeline, the New Tab data pipelines), Picture of the Day, the daily crossword, and other New Tab features served the same way. Its output is an evidence-backed `FINDINGS.md` carrying a root cause and quantified impact. -Use this skill when a Sentry alert fires, an editor reports something broken, or recommendations look empty, wrong, or stale. See [`skills/hnt-backend-investigation/SKILL.md`](skills/hnt-backend-investigation/SKILL.md) for details. +Use this skill when a Sentry alert fires, an editor reports something broken, or a New Tab feature looks empty, wrong, or stale. See [`skills/hnt-backend-investigation/SKILL.md`](skills/hnt-backend-investigation/SKILL.md) for details. **Optional tools.** Anything unreachable is raised as a task and reported under "Could not measure": diff --git a/plugins/newtab/skills/hnt-backend-investigation/SKILL.md b/plugins/newtab/skills/hnt-backend-investigation/SKILL.md index 6309324..421f84c 100644 --- a/plugins/newtab/skills/hnt-backend-investigation/SKILL.md +++ b/plugins/newtab/skills/hnt-backend-investigation/SKILL.md @@ -1,29 +1,28 @@ --- name: hnt-backend-investigation -description: Investigates Home New Tab backend errors, outages, and data-quality problems across Merino, the curated corpus, admin-api, the article crawler, the ML section pipeline, and the New Tab data pipelines. Confirms the symptom independently, probes competing hypotheses in parallel, stratifies metrics, tries to break its own conclusion, then writes an evidence-backed FINDINGS doc with a root cause and quantified impact. Use when a Sentry alert fires, an editor reports something broken, or recommendations look empty, wrong, or stale. For backend investigation, not in-tree browser/extensions/newtab frontend work. +description: Investigates Home New Tab backend errors, outages, and data-quality problems in whichever New Tab feature the symptom lands in: content recommendations (Merino, the curated corpus, admin-api, the article crawler, the ML section pipeline, the New Tab data pipelines), Picture of the Day, the daily crossword, or a feature not named here. Confirms the symptom independently, probes competing hypotheses in parallel, stratifies metrics, tries to break its own conclusion, then writes an evidence-backed FINDINGS doc with a root cause and quantified impact. Use when a Sentry alert fires, an editor reports something broken, or a New Tab feature looks empty, wrong, or stale — recommendations, the picture of the day, the puzzle. For backend investigation, not in-tree browser/extensions/newtab frontend work. --- # New Tab (HNT) Backend Investigation -Diagnose a problem in the services behind Firefox Home New Tab recommendations — Merino, the -curated corpus, admin-api, the article crawler, the ML section pipeline, the data pipelines. Not -for in-tree `browser/extensions/newtab` frontend bugs. +Diagnose a problem in the backend services behind Firefox Home New Tab. The page is assembled from +features that are served separately and fail separately — content recommendations (Merino, the +curated corpus, admin-api, the article crawler, the ML section pipeline, the data pipelines), +Picture of the Day, the daily crossword — so establish which one the symptom belongs to before +probing. A New Tab feature with no notes in this skill is still yours to diagnose: work it the same +way, from the Merino provider and config that serve it. Not for in-tree `browser/extensions/newtab` +frontend bugs, and not for Merino's non-New-Tab consumers such as Firefox Suggest. Your product is a **diagnosis**: a root cause, the evidence for it, and the impact quantified. Two -populations can be affected, and every finding should say which: **Firefox New Tab clients** (everyone who sees recommendations) and **editors/curators** -(the internal editorial team working through curation-admin-tools and admin-api). +populations can be affected, and every finding should say which: **Firefox New Tab clients** +(everyone the affected feature reaches) and, where the feature has one, the **editors/curators** +working through curation-admin-tools and admin-api. -**How to reach each system, and the traps that will silently give you wrong answers** — read before -your first probe in step 1: [references/data-sources.md](references/data-sources.md) - -**How these systems fail, the invariants to check, and falsification moves that work** — read when a -probe comes back empty, when a story starts looking too clean, or when your hypothesis list has -narrowed to one: [references/failure-modes.md](references/failure-modes.md) +Access and traps, per system: [references/data-sources.md](references/data-sources.md) +Failure mechanisms, invariants, falsification moves: [references/failure-modes.md](references/failure-modes.md) ## How to think about this -Four habits do most of the work. They matter more than any specific query below. - **Confirm the symptom before explaining it.** Independently reproduce or measure the reported problem first, whatever its source. An alert can be miscalibrated, mis-scoped, or watching the wrong layer, and a description passed through two people can drift. Until you have seen the @@ -127,20 +126,21 @@ happening. | Fact | Why it matters | |---|---| -| Exact symptom, verbatim | "Empty section" and "wrong items in section" have disjoint causes | +| Exact symptom, verbatim, and which feature | "Empty section" and "wrong items in section" have disjoint causes | | Who noticed, and how | Alert / editor report / spotted by hand — sets what evidence exists | | First and last seen, **with timezone** | Anchors the timeline; reports usually arrive late | -| Surface, locale, section, client version | The stratum is very often the diagnosis | +| The stratum — surface, locale, section, client version, or for a daily artifact the date | The stratum is very often the diagnosis | | Still happening right now? | Live reproduction vs. historical forensics | -**If it is still happening, capture the perishable evidence first** — a live -`curated-recommendations` request, current logs, current per-section counts — and save the raw -responses under `api_responses/`. Build the step-2 prior while those probes are in flight; history +**If it is still happening, capture the perishable evidence first** — a live request for the feature, +current logs, and whatever counts or artifact it publishes — and save the raw responses under +`api_responses/`. Build the step-2 prior while those probes are in flight; history keeps, live signal does not. **An error-rate signal is not yet a symptom.** Measure the erroring stage's *output* to fix the -blast radius — items per section and freshest item timestamp per surface for assembly, a live Merino -request for anything client-facing — so you can say whether Firefox clients are affected at all. An +blast radius — items per section and per-surface freshness for assembly, the newest object under a +dated prefix for a publish job, a live Merino request for anything client-facing — so you can say +whether Firefox clients are affected at all. An error floor that never reaches the output is a not-incident; errors flat with output at zero is worse than the alert says. When the report is a Sentry alert, resolve it to a concrete error and a volume, and decompose the issue by error message before trusting its title or its trend — the @@ -170,11 +170,11 @@ follow the posting rule at the end of step 8. You do not know what is currently normal in this system, so you have no basis on which to reject your own inference. Build that prior yourself before asking for it — most of it is in reach: -- **Which surfaces are serving** — surfaces present and not disabled in the section data, or a live - Merino request for the locale in question. -- **Whether the crawl is feeding them** — per-surface, per-source volume over the last day from the - crawl data. These two sets differ, and a crawled surface with no sections is not automatically a - bug. +- **What the serving path returns now** — a live request for the feature; for recommendations, the + surfaces present and not disabled in the section data. +- **What the producer last wrote** — per-surface, per-source crawl volume over the last day for + recommendations, the newest object under the dated bucket prefix for a publish job. Producer and + serving sets differ, and a crawled surface with no sections is not automatically a bug. - **What shipped recently** — recent commits in the service repos, releases in Sentry, timestamps on model or config artifacts. - **Whether this is normally noisy or seasonal** — a trailing profile of the same metric, by day of @@ -238,10 +238,11 @@ An explicit `timestamp (UTC) | observation | source` table. ## Step 5 — stratify before concluding anything -An aggregate that looks fine is the normal way these problems hide. Slice every metric by -**domain, locale, region, surface, section, experiment branch, and client/addon version**, and look -for a single stratum at zero or down sharply against its own trailing median rather than against -yesterday. When the report says "some users" with no stratum attached, treat experiment branch, +An aggregate that looks fine is the normal way these problems hide. Slice every metric by the +dimensions the feature actually has — for recommendations **domain, locale, region, surface, section, +experiment branch, and client/addon version**, for a once-a-day global artifact barely more than the +date — and look for a single stratum at zero or down sharply against its own trailing median rather +than against yesterday. When the report says "some users" with no stratum attached, treat experiment branch, region, and rollout state as the first cuts — some surfaces are reachable only through experiment enrolment, so an enrolled/unenrolled split is invisible to a locale slice. @@ -283,7 +284,7 @@ they stop the next person re-running them. ## Step 7 — write FINDINGS.md ```markdown -# — investigation +# : — investigation **Status:** investigating | root cause identified | blocked **Incident type:** outage | degradation | data-quality defect | user-facing bug | near-miss | not-incident @@ -297,8 +298,8 @@ Three sentences: what broke, since when, what the user-visible effect is. | Time | Observation | Source | ## Prior -**Derived** — per-surface volumes, recent deploys and model artifacts, trailing profile, prior -occurrences. Each with the query or link. +**Derived** — the feature's own volumes or published artifacts, recent deploys and model artifacts, +trailing profile, prior occurrences. Each with the query or link. **Reported** — verbatim answers from the developer, or `asked
-`, e.g. `jul29-empty-de-sections`. Say which directory you created. Reuse an existing one only when its FINDINGS.md is about the same symptom. -- `FINDINGS.md` — one living document. Create it from the step-7 skeleton the moment the directory - exists, and in any case before you write a conclusion down; steps 2 through 6 write into it as they - go, so step 7 is a final pass over a document that already exists. If you are still waiting on the - root, run the step-1 probes anyway and hold their output until you have somewhere to put it. Rewrite +- `FINDINGS.md` — one living document. Create it from the skeleton in the write-up step the moment the directory + exists, and in any case before you write a conclusion down; every step that follows writes into it as + it goes, so the write-up step is a final pass over a document that already exists. If you are still waiting on the + root, run the first probes anyway and hold their output until you have somewhere to put it. Rewrite it in place, not as `FINDINGS-v2.md`. - `queries/` and `results/` — every query as a file, its output alongside under the same basename. - `api_responses/` — raw JSON from live calls. Save them even when they look boring; metered APIs @@ -129,7 +129,7 @@ happening. **If it is still happening, capture the perishable evidence first** — a live request for the feature, current logs, and whatever counts or artifact it publishes — and save the raw responses under -`api_responses/`. Build the step-2 prior while those probes are in flight; history +`api_responses/`. Establish what normal is while those probes are in flight; history keeps, live signal does not. **An error-rate signal is not yet a symptom.** Measure the erroring stage's *output* to fix the @@ -155,7 +155,7 @@ what clients see on the surface. Probe both in parallel and state which populati The moment you confirm client-visible impact that is still happening, state it in one line — what, how big, since when — and keep investigating. If that deserves a heads-up in `#hnt-dev-be-alerts`, -follow the posting rule at the end of step 8. +follow the posting rule under "Posting to `#hnt-dev-be-alerts`". ## Step 2 — establish what "normal" is @@ -185,8 +185,8 @@ query contradicts the prior, treat your own inference as the suspect first, and ## Step 3 — form hypotheses from the data, then probe them in parallel -Hypotheses come from what you have already measured, not from a catalogue. Start where step 1 put -you: the stage whose output is wrong, the stratum that is affected, the moment it changed. Ask what +Hypotheses come from what you have already measured, not from a catalogue. Start where pinning down +the report put you: the stage whose output is wrong, the stratum that is affected, the moment it changed. Ask what could produce exactly that, follow the data one hop upstream, and let each result generate the next question. A hypothesis you cannot tie to something you have observed is a guess competing for the same probe budget as one you can. @@ -219,7 +219,7 @@ An explicit `timestamp (UTC) | observation | source` table. Widen freely on the cheap planes; on a billed plane, widen only after a narrow window has shown you something. - Distinguish *first occurrence* from *first noticed*, and state both. -- Line the window up against the deploys and config changes from step 2. +- Line the window up against the deploys and config changes the prior turned up. - Treat retention limits as limits: a "first seen" date can be the edge of a retention window rather than onset. See `references/data-sources.md`. - **The scheduling layer is not in UTC.** Scheduled dates and the assembly crons run in each surface's @@ -263,7 +263,7 @@ Before writing anything down as fact, attack it: - **Measure the effect, not just the mechanism.** A guard that provably runs is not evidence that the outcome is correct; verify the outcome separately. - Cross-check against a second, independent source. One source is a hypothesis. -- Re-read the step-2 prior. If your conclusion implies that something the prior says is working is +- Re-read the prior you established. If your conclusion implies that something it says is working is broken, re-check your own measurement first — a filter in your query, a stratum mismatch, or a wrong surface identifier is the likelier explanation. If the measurement survives that re-check, the measurement wins: state the contradiction explicitly in FINDINGS.md, note it to the developer diff --git a/plugins/newtab/skills/hnt-backend-investigation/references/data-sources.md b/plugins/newtab/skills/hnt-backend-investigation/references/data-sources.md index 6f44e30..1ab20d9 100644 --- a/plugins/newtab/skills/hnt-backend-investigation/references/data-sources.md +++ b/plugins/newtab/skills/hnt-backend-investigation/references/data-sources.md @@ -358,7 +358,7 @@ serving gap. ### Experiment enrolment -Step 5 makes experiment branch one of the first cuts, and the branch names are not in any of the +Stratifying makes experiment branch one of the first cuts, and the branch names are not in any of the tables above. The Experimenter API lists live and recent experiments without authentication: `https://experimenter.services.mozilla.com/api/v6/experiments/`. Use it to get the real slug and branch names before slicing telemetry, rather than inventing them or asking. It returns every experiment, so @@ -385,10 +385,17 @@ unapproved domains into the corpus. per-domain response-code distribution over time. It takes a **different credential** — the Zyte dashboard API key from the organisation's settings page, explicitly not the Zyte API key above — as the HTTP basic username with an empty password, so the extraction key will be rejected here. -`organization_id` is **required** and is `612928`. Only `groupby_time` and `groupby_domain` group; -`response_codes` is a *filter*, not a grouping, and `include_domain_health=true` is rejected without -`groupby_domain=true`. This is the right source for "did this domain start failing, and when" — your -own logs will not show it if the pipeline discards non-allowlisted status codes. +`organization_id` is **required** and is `612928`. Only `groupby_time` (`hour|day|month|year`) and +`groupby_domain` group; `response_codes`, `domains`, `extraction_type` and `extraction_from` are +*filters*, not groupings, and `include_domain_health=true` is rejected without `groupby_domain=true`. + +Two defaults will quietly narrow an answer. **`start_time` defaults to seven days ago**, so a question +about when something began returns only the last week unless you pass an explicit window with +`end_time`. And results are **paginated, `page_size` maxing out at 500**, so a per-domain breakdown over +any real window is truncated unless you walk `page`. Both look like a complete answer. + +This is the right source for "did this domain start failing, and when" — your own logs will not show it +if the pipeline discards non-allowlisted status codes. ## Picture of the Day @@ -412,8 +419,8 @@ Sentry is the rest of the plane; `docs/providers/games/particle.md` has the deta `#hnt-dev-be-alerts` is where investigation updates go. Posting needs the developer's approval for **each** message and prefers a reply in the thread of the alert that started this, so locating that -one message is what the `mcp__slack__` tools are for here; the rule is at the end of step 8 in -SKILL.md. See Access requests if the tools are absent. +one message is what the `mcp__slack__` tools are for here; the rule is under "Posting to +`#hnt-dev-be-alerts`" in SKILL.md. See Access requests if the tools are absent. ## Access requests @@ -442,7 +449,7 @@ Roughly ordered by how often an investigation needs them, cheapest first. | `gcloud` or `bq` not installed | Install the Google Cloud SDK, which provides both | | No billing project configured | Confirm you can bill `mozdata-nonprod`, or name their personal sandbox, usually `moz-fx-dev--sandbox`; either can be set with `gcloud config set project ` | | No `mcp__sentry__` tools at all | `claude mcp add --scope user --transport http sentry https://mcp.sentry.dev/mcp`, then `/mcp` in the restarted session to authenticate | -| Sentry connected but unauthenticated or scoped too narrowly | `/mcp`, and authenticate for the `mozilla` org. Or read back the issue's event counts broken down by error message | +| Sentry connected but unauthenticated or scoped too narrowly | `/mcp`, and authenticate for the `mozilla` org | | Corpus MySQL hangs rather than erroring | Connect to Mozilla VPN, then say so; if it still hangs the login path itself is stale | | AWS SSO session expired | `aws --profile sso login` | | The answer is in a dashboard you cannot reach | Open it, apply the specific filter you name, and read back the one number or shape you asked for. Asking for *access* to a dashboard is usually the slower path; asking a precise question about what it shows is faster for both of you | From 65eed3a871ff9215bccdacb52f4ef3cf3a11fde8 Mon Sep 17 00:00:00 2001 From: Matt Miermans Date: Mon, 10 Aug 2026 11:51:04 -0700 Subject: [PATCH 27/51] newtab/hnt-backend-investigation: split VPN row by client install state --- plugins/newtab/README.md | 2 +- .../hnt-backend-investigation/references/data-sources.md | 7 ++++--- 2 files changed, 5 insertions(+), 4 deletions(-) diff --git a/plugins/newtab/README.md b/plugins/newtab/README.md index 8f4123b..5c215c7 100644 --- a/plugins/newtab/README.md +++ b/plugins/newtab/README.md @@ -21,7 +21,7 @@ Use this skill when a Sentry alert fires, an editor reports something broken, or - `gh` - `gcloud` and `bq` - the `aws` CLI, with a read-only SSO profile -- a MySQL client, plus Mozilla VPN +- a MySQL client, plus the AWS Client VPN - `uv` or `python3` - the Sentry MCP server - the Slack MCP server, for `#hnt-dev-be-alerts` diff --git a/plugins/newtab/skills/hnt-backend-investigation/references/data-sources.md b/plugins/newtab/skills/hnt-backend-investigation/references/data-sources.md index 1ab20d9..3375e54 100644 --- a/plugins/newtab/skills/hnt-backend-investigation/references/data-sources.md +++ b/plugins/newtab/skills/hnt-backend-investigation/references/data-sources.md @@ -279,8 +279,8 @@ Traps that will silently give you a wrong answer: ### Curated corpus MySQL Production database behind curated-corpus-api, reached through a preconfigured read-only login path. -Check what exists with `mysql_config_editor print --all`, and expect to need VPN — a hang rather than -an auth error is the usual symptom of being off it. +Check what exists with `mysql_config_editor print --all`, and expect to need the AWS Client VPN — a hang +rather than an auth error is the usual symptom of being off it. **Connect as a read-only user or not at all.** `pkt_curation_corpus` is the application's own account on the prod cluster and carries full write and schema privileges; decline it even for a `SELECT`, and @@ -450,7 +450,8 @@ Roughly ordered by how often an investigation needs them, cheapest first. | No billing project configured | Confirm you can bill `mozdata-nonprod`, or name their personal sandbox, usually `moz-fx-dev--sandbox`; either can be set with `gcloud config set project ` | | No `mcp__sentry__` tools at all | `claude mcp add --scope user --transport http sentry https://mcp.sentry.dev/mcp`, then `/mcp` in the restarted session to authenticate | | Sentry connected but unauthenticated or scoped too narrowly | `/mcp`, and authenticate for the `mozilla` org | -| Corpus MySQL hangs rather than erroring | Connect to Mozilla VPN, then say so; if it still hangs the login path itself is stale | +| Corpus MySQL hangs rather than erroring, and they have the VPN client | Connect to the AWS Client VPN, then say so; if it still hangs, the login path itself is stale | +| Corpus MySQL hangs and there is no VPN client installed | Install the AWS VPN Client from https://aws.amazon.com/vpn/client-vpn-download/, then download the client configuration from the endpoint page — https://us-east-1.console.aws.amazon.com/vpcconsole/home?region=us-east-1#ClientVPNEndpointDetails:clientVpnEndpointId=cvpn-endpoint-0d3c4e4a0121a5763 — load that profile into the client, connect, and say so | | AWS SSO session expired | `aws --profile sso login` | | The answer is in a dashboard you cannot reach | Open it, apply the specific filter you name, and read back the one number or shape you asked for. Asking for *access* to a dashboard is usually the slower path; asking a precise question about what it shows is faster for both of you | | No AWS profile at all | `aws configure sso` for a read-only role, or have them name a profile already in their `~/.aws/config` | From 947d6dcb9810c2764100dae8c7b3f57f2eb1255e Mon Sep 17 00:00:00 2001 From: Matt Miermans Date: Mon, 10 Aug 2026 11:55:40 -0700 Subject: [PATCH 28/51] newtab/hnt-backend-investigation: point at bigquery-etl for unlisted models --- .../hnt-backend-investigation/references/data-sources.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/plugins/newtab/skills/hnt-backend-investigation/references/data-sources.md b/plugins/newtab/skills/hnt-backend-investigation/references/data-sources.md index 3375e54..060d2d6 100644 --- a/plugins/newtab/skills/hnt-backend-investigation/references/data-sources.md +++ b/plugins/newtab/skills/hnt-backend-investigation/references/data-sources.md @@ -240,6 +240,14 @@ from an empty result, and use `bq ls ` to settle whether something is a whether it is partitioned — a view has no partition column, so ordering a partition filter on one is a query error. +These are the tables that come up most, not everything that is modelled. When they cannot answer the +question, `bigquery-etl` is the index of what else exists: each derived table has a directory at +`sql////` holding its `query.sql`, `schema.yaml` and `metadata.yaml`, so +grepping a table or column name there finds both what produces it and what sits beside it. If that repo +is cloned locally, hand the search to a subagent on the fastest model you have — it is a grep-and-read +job whose answer is a table name and a few columns, and delegating keeps a wide search out of your +context. + Traps that will silently give you a wrong answer: - **`zyte_cache` is not unique on `canonical_url`** — some urls appear many times over — and From e8adc6bf49a643caeae0b867b13703bc0225aae8 Mon Sep 17 00:00:00 2001 From: Matt Miermans Date: Mon, 10 Aug 2026 12:06:47 -0700 Subject: [PATCH 29/51] newtab/hnt-backend-investigation: document verified Zyte Stats auth encoding --- .../references/data-sources.md | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-) diff --git a/plugins/newtab/skills/hnt-backend-investigation/references/data-sources.md b/plugins/newtab/skills/hnt-backend-investigation/references/data-sources.md index 060d2d6..b52e35b 100644 --- a/plugins/newtab/skills/hnt-backend-investigation/references/data-sources.md +++ b/plugins/newtab/skills/hnt-backend-investigation/references/data-sources.md @@ -375,8 +375,8 @@ this file. ### Zyte -Two separate APIs, both **metered — every call costs money**. Save all responses; never bulk-crawl to -satisfy curiosity. Check for a key with `[ -n "$ZYTE_API_KEY" ] && echo present`. Reference keys by +Two separate APIs. The extraction API is **metered — every call costs money**, so save all responses and +never bulk-crawl to satisfy curiosity. Check for a key with `[ -n "$ZYTE_API_KEY" ] && echo present`. Reference keys by variable name in anything you save, so no value lands in a query file or the transcript. **Extraction API** (`ZYTE_API_KEY`, created at https://app.zyte.com/o/612928/zyte-api/api-access) @@ -391,8 +391,17 @@ unapproved domains into the corpus. **Stats API** (`https://zyte-api-stats.zyte.com/api/stats`) is the vendor's own view of your traffic: per-domain response-code distribution over time. It takes a **different credential** — the Zyte -dashboard API key from the organisation's settings page, explicitly not the Zyte API key above — as the -HTTP basic username with an empty password, so the extraction key will be rejected here. +dashboard API key from the organisation's settings page, explicitly not the Zyte API key above, held +here as `ZYTE_SECRET_KEY`. + +Authentication is HTTP basic with that key as the **username and no password**, which is fiddlier than +it sounds and is the usual reason a request that looks right fails. With curl, `-u "$ZYTE_SECRET_KEY:"` +— **the trailing colon is required**, because without it curl reads the whole string as a username and +stalls waiting for a password. Constructing the header by hand works too, as +`Authorization: Basic :">`, with the colon inside the encoded string. Read the status +code before anything else: `401` means no usable credential reached the server, and `403` means a valid +Zyte credential that is not the dashboard key, which in practice means the extraction key was used. + `organization_id` is **required** and is `612928`. Only `groupby_time` (`hour|day|month|year`) and `groupby_domain` group; `response_codes`, `domains`, `extraction_type` and `extraction_from` are *filters*, not groupings, and `include_domain_health=true` is rejected without `groupby_domain=true`. From aea7c1c1e3839c441190dc8c9cfc7774b765388a Mon Sep 17 00:00:00 2001 From: Matt Miermans Date: Mon, 10 Aug 2026 13:02:32 -0700 Subject: [PATCH 30/51] newtab/hnt-backend-investigation: name where the Zyte dashboard key comes from --- .../skills/hnt-backend-investigation/references/data-sources.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/plugins/newtab/skills/hnt-backend-investigation/references/data-sources.md b/plugins/newtab/skills/hnt-backend-investigation/references/data-sources.md index b52e35b..b69a7f2 100644 --- a/plugins/newtab/skills/hnt-backend-investigation/references/data-sources.md +++ b/plugins/newtab/skills/hnt-backend-investigation/references/data-sources.md @@ -477,5 +477,5 @@ Roughly ordered by how often an investigation needs them, cheapest first. | Permission denied on a dataset or a Merino project | Request read access, or viewer on the project; say meanwhile whether the question is about payload shape, which stage can answer | | No `mcp__slack__` tools | `/plugin install slack@claude-plugins-official` typed into Claude Code, then `/mcp` to authenticate. The server needs a fixed OAuth callback port, so it can clash with another session authenticating at the same moment. Or have them post the drafted message to `#hnt-dev-be-alerts` themselves | | Zyte extraction key missing | Create one at https://app.zyte.com/o/612928/zyte-api/api-access, then add it to the `env` block of `~/.claude/settings.json`: `"env": { "ZYTE_API_KEY": "" }`. That reaches every session and the commands it spawns, and it is the user-scope file rather than anything checked in. A shell `export` will not reach this session, which did not inherit it | -| Zyte Stats key missing | Issue a **dashboard** API key from the Zyte organisation settings page — the extraction key will not authenticate against the Stats API — and add it to the same `env` block as `ZYTE_SECRET_KEY` | +| Zyte Stats key missing | Get the **dashboard** API key from https://app.zyte.com/o/612928/settings, via the `Download your API key as .TXT` button — the extraction key will not authenticate against the Stats API — and add it to the same `env` block as `ZYTE_SECRET_KEY` | | Editor-facing symptom needs an authenticated session | Reproduce the click themselves and report the exact error text and time | From 6f8b712ce59b00a36538c8bd71448f9240eec795 Mon Sep 17 00:00:00 2001 From: Matt Miermans Date: Mon, 10 Aug 2026 13:27:27 -0700 Subject: [PATCH 31/51] newtab/hnt-backend-investigation: fix frontmatter YAML, drop failure-modes reference --- .../skills/hnt-backend-investigation/SKILL.md | 23 +-- .../references/failure-modes.md | 170 ------------------ 2 files changed, 12 insertions(+), 181 deletions(-) delete mode 100644 plugins/newtab/skills/hnt-backend-investigation/references/failure-modes.md diff --git a/plugins/newtab/skills/hnt-backend-investigation/SKILL.md b/plugins/newtab/skills/hnt-backend-investigation/SKILL.md index 2d5dd1c..500f754 100644 --- a/plugins/newtab/skills/hnt-backend-investigation/SKILL.md +++ b/plugins/newtab/skills/hnt-backend-investigation/SKILL.md @@ -1,6 +1,6 @@ --- name: hnt-backend-investigation -description: Investigates Home New Tab backend errors, outages, and data-quality problems in whichever New Tab feature the symptom lands in: content recommendations (Merino, the curated corpus, admin-api, the article crawler, the ML section pipeline, the New Tab data pipelines), Picture of the Day, the daily crossword, or a feature not named here. Confirms the symptom independently, probes competing hypotheses in parallel, stratifies metrics, tries to break its own conclusion, then writes an evidence-backed FINDINGS doc with a root cause and quantified impact. Use when a Sentry alert fires, an editor reports something broken, or a New Tab feature looks empty, wrong, or stale — recommendations, the picture of the day, the puzzle. For backend investigation, not in-tree browser/extensions/newtab frontend work. +description: Investigates Home New Tab backend errors, outages, and data-quality problems in whichever New Tab feature the symptom lands in, whether content recommendations (Merino, the curated corpus, admin-api, the article crawler, the ML section pipeline, the New Tab data pipelines), Picture of the Day, the daily crossword, or a feature not named here. Confirms the symptom independently, probes competing hypotheses in parallel, stratifies metrics, tries to break its own conclusion, then writes an evidence-backed FINDINGS doc with a root cause and quantified impact. Use when a Sentry alert fires, an editor reports something broken, or a New Tab feature looks empty, wrong, or stale — recommendations, the picture of the day, the puzzle. For backend investigation, not in-tree browser/extensions/newtab frontend work. --- # New Tab (HNT) Backend Investigation @@ -19,7 +19,6 @@ populations can be affected, and every finding should say which: **Firefox New T working through curation-admin-tools and admin-api. Access and traps, per system: [references/data-sources.md](references/data-sources.md) -Failure mechanisms, invariants, falsification moves: [references/failure-modes.md](references/failure-modes.md) ## How to think about this @@ -138,8 +137,9 @@ dated prefix for a publish job, a live Merino request for anything client-facing whether Firefox clients are affected at all. An error floor that never reaches the output is a not-incident; errors flat with output at zero is worse than the alert says. When the report is a Sentry alert, resolve it to a concrete error and a -volume, and decompose the issue by error message before trusting its title or its trend — the -alert-quality checks are in `references/failure-modes.md`. +volume, and decompose the issue by error message before trusting its title or its trend. An alert +can be mis-centred, watching the wrong layer, or counting per open period rather than per day, so +confirm the signal itself before chasing the system behind it. Sentry is the plane most of this reads from, so if the `mcp__sentry__` tools are absent, say so and raise it like any other blocked source, then carry on with the planes you do have. @@ -200,14 +200,13 @@ after its hypothesis, so a result cannot be attributed to the wrong line. Then look at what came back and do it again. Each round should either kill a line or sharpen the next question; when a round kills everything, the data has moved you upstream rather than left you stuck. -If you are down to a single surviving hypothesis, that is the moment to check the failure classes and -silent-failure mechanisms in `references/failure-modes.md` against what you have measured, as a guard -against having narrowed too early. +Being down to a single surviving hypothesis is a prompt, not an answer: ask what else could produce +exactly what you measured before you commit to it. Two moves consistently break cases open: **leave the tool you started in** — when a source stops yielding, measure the same thing somewhere else, since the failure is often invisible in the plane you began with — and **compare against sibling strata**, which localises a fault faster than reading -code. Both are expanded in `references/failure-modes.md`. +code. ## Step 4 — build the timeline @@ -257,9 +256,11 @@ Before writing anything down as fact, attack it: - Check the boring explanations first: timezone; a filter in your own query; a `LIMIT` silently truncating; an unrepresentative code path; sampling; a column that does not mean what its name says; a partially-launched feature. -- Then reach for the falsification moves in `references/failure-modes.md` — the code path, exact - enumeration, repetition, the real limit, the config's history — picking the ones that would break - your specific mechanism. +- Then attack the mechanism itself, with whichever of these would break yours: read the code path + rather than assuming it; enumerate exactly instead of sampling, since "0 of N" is a different claim + from "none in my sample"; repeat a suspicious observation before calling anything + non-deterministic; read the real limit rather than guessing which one binds; and check the history + of the constant or config you are relying on. - **Measure the effect, not just the mechanism.** A guard that provably runs is not evidence that the outcome is correct; verify the outcome separately. - Cross-check against a second, independent source. One source is a hypothesis. diff --git a/plugins/newtab/skills/hnt-backend-investigation/references/failure-modes.md b/plugins/newtab/skills/hnt-backend-investigation/references/failure-modes.md deleted file mode 100644 index 9a03517..0000000 --- a/plugins/newtab/skills/hnt-backend-investigation/references/failure-modes.md +++ /dev/null @@ -1,170 +0,0 @@ -# How New Tab (HNT) backend failures behave - -Reference for the `hnt-backend-investigation` skill: the shapes these failures take, the invariants -to check, and the moves that kill a wrong hypothesis. SKILL.md names the moments to read it; the -alert audit below applies as soon as a report comes from an alert. - -If you are editing this file later: it holds generalised mechanisms and techniques. It is deliberately -not a symptom-to-cause lookup, and it deliberately carries no worked incidents, issue ids, or canned -queries. Adding any of those changes what the file is for. - -## Contents - -- Failure classes — the vocabulary, for checking you have not narrowed too early -- If it came from an alert, audit the alert -- Why these failures are silent — the mechanisms that turn a failure into a valid-looking result -- Falsification moves that work — what to reach for when a story feels too clean -- Structural gaps to report rather than infer past -- Invariants to check — the hard ones, the ones relative to a stratum's own history, and the class each points to - -## Failure classes - -Vocabulary, not a lookup. Do not use this list to guess a cause from a symptom; derive the layer from -what you have measured. It is here so that when your hypothesis list has narrowed to one or two, you -can check it against the range of things that actually break in this stack: - -vendor extraction-quality defect · upstream unavailability · silent validation rejection · resource -exhaustion or capacity cliff · ML model or routing failure · data-model or identity defect · filter -or threshold misconfiguration · unscoped bulk database operation · cascading failure or retry -amplification · observability defect · analytics correctness defect · dedup failure · authorization -defect · publication or promotion failure · client or product defect · capacity near-miss - -Observability defects are expanded in the next section, and the invariants at the end of this file -name the class each one points to. - -## If it came from an alert, audit the alert - -Alert quality is its own failure class, and worth a few minutes before chasing the system. This is an -addition to confirming the symptom independently, not a substitute for it. - -- **Wrong layer** — the alert watches a gateway while the loss happens a hop upstream. -- **Umbrella fingerprint** — one issue collapses several unrelated failures; decompose by error - message before trusting the title or the trend, and anchor on stable issue identifiers rather than - numeric ids in the alert body. -- **Mis-centred or flapping band** — repeated transitions at a fixed offset within the hour, and - margins under a percent, mean the band, not the system. -- **Counting artefact** — a panel that counts per open period rather than per calendar day can be off - by an order of magnitude. -- **Unroutable** — if you cannot tell prod from staging from the alert alone, establish which one it - is before reading anything else into it. - -## Why these failures are silent - -Not under-monitored — structurally unobservable. The recurring mechanism is that an upstream failure -gets converted into a valid-looking empty or partial result. Expect these: - -- A pipeline that keeps only an allowlist of status codes and swallows request errors, so a total - block is indistinguishable from "nothing found today". -- A vendor non-2xx arriving as a partially-filled success object, so a total outage raises nothing. -- A job reporting invocation-level success while silently dropping every item it received. -- Input-validation rejections that are logged and never surfaced. -- An `INNER JOIN` against a frozen upstream catalog shedding rows quietly instead of failing. -- A process that dies during startup, which emits no error event at all. -- A bulk write with no row-count guard. -- A write path disabled by a config flag, so the job runs, returns normally, and writes nothing. -- A queued handoff backing up or dead-lettering, which looks identical to the producer sending nothing. -- A cache serving stale content on upstream failure and extending its own expiry, so an outage - upstream reaches clients as a successful response with old data. -- An aggregate quality gate passing while one class collapses beneath it. -- A publish that writes one success marker for two parts, so a part that failed beside a part that - succeeded reads as current and is never retried. -- A job whose success value means "something changed", so an ordinary no-change run and a failed run - report identically. -- An alarm that treats missing data as "missing" rather than as a breach, sitting against an emitter - that publishes nothing at zero — so it goes *quiet* during a total stop. - -The consequence for every one of these: **liveness is not health.** Count what came out and compare -it to what went in. - -## Falsification moves that work - -Each of these has killed a plausible, confidently-held hypothesis. Reach for them when a story feels -too clean. - -- **Read the code path.** Assumed behaviour of a resolver, filter, or job is the single most common - source of a wrong mechanism. -- **Enumerate the whole population** rather than sampling, when it is free to count — in BigQuery or - MySQL, "0 of N" is a different claim from "none in my sample". Against a metered API, sample, say - what the sample is, and put any larger enumeration on the task list with its call count. -- **Repeat the observation N times** before calling anything non-deterministic, then vary one input - at a time. -- **Read the actual limit** — quota, timeout, payload cap, row cap — instead of assuming which one - binds. Adjacent limits on the same resource are easy to confuse. -- **Check the history of the constant or config** you are relying on; a value that used to be true is - the classic stale-assumption trap. -- **Dump the attribute across the entire population** when you suspect truncation or a per-record - limit. -- **Go to the other side's telemetry.** A vendor's or upstream's own status data will show what your - logs structurally cannot. -- **Measure the same thing in a different plane.** Each plane here is blind in a particular way, and - the pairs are what break structurally-invisible cases: Sentry sees only what was raised, while - CloudWatch Logs Insights sees every operation including the ones that failed quietly; the BigQuery - log sink sees requests the logging API will not surface; the vendor's stats see the responses your - own pipeline discarded before logging them; client telemetry sees what users got when every backend - table looks healthy. -- **Compare against sibling strata** at the same layer — the healthy peers localise the fault faster - than reading code does. -- **Design a canary that would falsify** the hypothesis rather than a test that would confirm it. -- **Refute it yourself, in writing.** Before a conclusion leaves FINDINGS.md, draft the strongest - case against it under "Hypotheses considered and dropped" and answer that case with a measurement. - A second agent handed the evidence and told to break the conclusion is cheaper than waiting for a - human reviewer. - -## Structural gaps to report rather than infer past - -These make certain questions unanswerable from stored data alone. When one blocks you, say so. - -- No crawl attempt/success/error record, so a zero row count is ambiguous between "found nothing" and - "every request failed". -- No lifecycle metadata or tombstones on the crawl target list, so an intentional retirement is - indistinguishable from an outage. -- No persisted ML filter funnel, so per-stage drop-off cannot be reconstructed afterwards. -- No run history for the daily publish jobs beyond a log line and the dated objects they leave, so - whether today's run happened is inferred from its output. -- No queryable deployment history. The running revision is available per service — Merino's - `/__version__`, the lambdas' `GIT_SHA` — but there is no log of past deploys to line a timeline up - against, so correlate against the current revision and the repo history instead. -- No status code or extraction-probability stored alongside cached hydration results, so a stored row - does not imply a successful extraction. - -## Invariants to check - -Checks worth asserting against the data when they bear on your hypotheses: a broken one narrows the -search, and a holding one eliminates a line cheaply. Read the two groups differently. The first group -is a hard equality or a structural fact, so a violation means something is wrong. The second group is -relative to the stratum's own history, so a violation means *look here*, not *this is the fault* — -healthy production violates a naive absolute version of every one of them. - -Hard: - -| Invariant | Class it points to | -|---|---| -| Canonical domain ∈ approved list | Data-model or identity defect | -| Every bound parameter in shared SQL supplied by every caller | Analytics correctness defect | -| A derived dimension agrees with the entity it describes | Analytics correctness defect | -| The alarm itself emits a value at zero rather than nothing | Observability defect | -| Consumption against documented quota | Capacity near-miss | -| A once-a-day artifact's own date == today (UTC) wherever it is served | Publication or promotion failure | -| The version a serving copy advertises == the version its producer last published | Publication or promotion failure | - -Relative to the stratum's own trailing behaviour: - -| Check | Class it points to | -|---|---| -| Articles per **day** > 0 per surface per source, plus the hourly count inside that stratum's own trailing range. An hour at zero is normal for low-volume strata and for a newly launched surface | Upstream unavailability; resource exhaustion or capacity cliff | -| Items served per section against that section's trailing count. There is no single configured size — the served count varies by layout and by request, so do not assert an equality | Silent validation rejection; ML model or routing failure | -| `items_written` against the job's own trailing ratio, not against `items_received`. Pipelines here dedupe by design, so a healthy ratio is far below one | Silent validation rejection | -| Input-validation reject count against its trailing floor. A steady non-zero floor is normal for third-party content; a step change is not | Silent validation rejection | -| Newest active item age per section against that surface's assembly interval | ML model or routing failure; resource exhaustion or capacity cliff | -| Vendor non-2xx share per domain below a **ceiling** | Upstream unavailability; vendor extraction-quality defect | -| Content-shape ratio per domain within bounds (empty-body share, title length) | Vendor extraction-quality defect | -| Per-domain funnel positive at every stage: discovered → hydrated → approved | Upstream unavailability; filter or threshold misconfiguration | -| Row-count delta on key tables within bounds | Unscoped bulk database operation | -| Queue depth, oldest-message age and dead-letter depth against their trailing values | Resource exhaustion or capacity cliff; silent validation rejection | - -For the relative group, name the plane you measured in: Cloud Logging or the BigQuery log sink for the -crawl and the GCP jobs, CloudWatch Logs Insights for the AWS lambdas, and client telemetry for anything -expressed in users. - -Stratify anything you assert. A rule that holds globally can be false for one surface, because the -surfaces do not all draw content the same way. From 4111b70a33b34f1d1901c76f741e5b7cbed4ae28 Mon Sep 17 00:00:00 2001 From: Matt Miermans Date: Mon, 10 Aug 2026 13:29:52 -0700 Subject: [PATCH 32/51] newtab/hnt-backend-investigation: point diagnosis at the fix, drop 'populations' --- .../skills/hnt-backend-investigation/SKILL.md | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/plugins/newtab/skills/hnt-backend-investigation/SKILL.md b/plugins/newtab/skills/hnt-backend-investigation/SKILL.md index 500f754..2124d73 100644 --- a/plugins/newtab/skills/hnt-backend-investigation/SKILL.md +++ b/plugins/newtab/skills/hnt-backend-investigation/SKILL.md @@ -13,10 +13,13 @@ probing. A New Tab feature with no notes in this skill is still yours to diagnos way, from the Merino provider and config that serve it. Not for in-tree `browser/extensions/newtab` frontend bugs, and not for Merino's non-New-Tab consumers such as Firefox Suggest. -Your product is a **diagnosis**: a root cause, the evidence for it, and the impact quantified. Two -populations can be affected, and every finding should say which: **Firefox New Tab clients** -(everyone the affected feature reaches) and, where the feature has one, the **editors/curators** -working through curation-admin-tools and admin-api. +What you owe first is a **diagnosis**: a root cause, the evidence for it, and the impact quantified. +Mitigating or fixing the problem usually follows straight from that, so carry the diagnosis far enough +to name the change that would resolve it and what would show the recovery. + +Two groups feel these failures, and every finding should say which one it reached: **Firefox New Tab +clients**, meaning everyone the affected feature serves, and, where the feature has them, the +**editors and curators** working through curation-admin-tools and admin-api. Access and traps, per system: [references/data-sources.md](references/data-sources.md) @@ -151,7 +154,7 @@ monitor measuring something other than what it claims. Classify as `not-incident second question has an answer; unconfirmed is not refuted. A report relayed from an editor is ambiguous between what editors see in curation-admin-tools and -what clients see on the surface. Probe both in parallel and state which population you confirmed. +what clients see on the surface. Probe both in parallel and state which group you confirmed. The moment you confirm client-visible impact that is still happening, state it in one line — what, how big, since when — and keep investigating. If that deserves a heads-up in `#hnt-dev-be-alerts`, From 5378c69cfd07a40cfc74c0266899e03f8b35405d Mon Sep 17 00:00:00 2001 From: Matt Miermans Date: Mon, 10 Aug 2026 13:32:09 -0700 Subject: [PATCH 33/51] newtab/hnt-backend-investigation: user memory loads, it is not read --- .../skills/hnt-backend-investigation/SKILL.md | 18 ++++++++++-------- 1 file changed, 10 insertions(+), 8 deletions(-) diff --git a/plugins/newtab/skills/hnt-backend-investigation/SKILL.md b/plugins/newtab/skills/hnt-backend-investigation/SKILL.md index 2124d73..d5a5448 100644 --- a/plugins/newtab/skills/hnt-backend-investigation/SKILL.md +++ b/plugins/newtab/skills/hnt-backend-investigation/SKILL.md @@ -79,17 +79,19 @@ that was never made. Each investigation gets its own subdirectory inside one root directory that the developer keeps their investigations in. Do not assume where that root is. Establish it in this order: -1. **Look in user memory.** `~/.claude/CLAUDE.md` is where the root is recorded, because it is the - only memory scope that applies across every repository you might be launched from. If it names a - root, use it and do not ask. +1. **Check the instructions you were already given.** A root recorded on an earlier run is already in + front of you: user-scope instructions load automatically at the start of every session. If they + name a root, use it and do not ask. 2. **Otherwise ask, once, before your first write.** Where the developer keeps investigations cannot be derived, and guessing puts files somewhere they did not choose. Ask it as a single question at the start, and start pinning down the report while you wait rather than idling. -3. **Then record it.** Append one line to `~/.claude/CLAUDE.md`, creating that file if it does not - exist: `Investigations live in (one subdirectory per investigation).` Tell the developer - you saved it, so they know they will not be asked again. Auto memory is the wrong home for this: - it is keyed to the repository you were launched from, and investigations get started from - whichever service repo is to hand. +3. **Then record it**, so the question is asked once per machine rather than once per investigation. + Add one line to `~/.claude/CLAUDE.md`, creating that file if it does not exist: + `Investigations live in (one subdirectory per investigation).` That is the user scope, the + one that applies whichever repository you are launched from, and it will be loaded for you from the + next session onward. Say that you saved it. Do not put this in auto memory, which is keyed to the + repository you were launched from, while investigations get started from whichever service repo is + to hand. Then create this investigation's subdirectory inside that root. If the root already holds investigations, follow the naming they use; otherwise `
-`, e.g. From 30c13ba6e2b724c59fc7b89338de70678645796c Mon Sep 17 00:00:00 2001 From: Matt Miermans Date: Mon, 10 Aug 2026 13:35:33 -0700 Subject: [PATCH 34/51] newtab/hnt-backend-investigation: drop cost framing, keep the practices --- .../skills/hnt-backend-investigation/SKILL.md | 18 ++++++------- .../references/data-sources.md | 25 +++++++++---------- 2 files changed, 20 insertions(+), 23 deletions(-) diff --git a/plugins/newtab/skills/hnt-backend-investigation/SKILL.md b/plugins/newtab/skills/hnt-backend-investigation/SKILL.md index d5a5448..2e43a25 100644 --- a/plugins/newtab/skills/hnt-backend-investigation/SKILL.md +++ b/plugins/newtab/skills/hnt-backend-investigation/SKILL.md @@ -36,10 +36,9 @@ to discriminate look similar and are not. The characteristic failure in this dom query paired with a wrong inference, stated confidently. **Work several lines at once.** When more than one explanation is plausible, probe them in parallel -rather than following the most attractive one to its end. Breadth is cheap in the unmetered planes — -Sentry, repo reads, CloudWatch alarm history, one live Merino request — and tunnel vision is the -expensive failure. It is not free in BigQuery, Zyte, or CloudWatch Logs Insights: dry-run, narrow the -columns, and narrow the window before you fan out. Keep a list of every line you considered and record which you dropped and why. +rather than following the most attractive one to its end. Tunnel vision is the failure that wastes the +most time, and a probe that comes back negative has still moved you forward. Keep a list of every line +you considered and record which you dropped and why. **Prefer measuring to reasoning.** Reproduce the request. Run the model on real inputs. Count the actual rows. Read the code path instead of assuming its behaviour. A plausible mechanism becomes a @@ -104,8 +103,8 @@ FINDINGS.md is about the same symptom. root, run the first probes anyway and hold their output until you have somewhere to put it. Rewrite it in place, not as `FINDINGS-v2.md`. - `queries/` and `results/` — every query as a file, its output alongside under the same basename. -- `api_responses/` — raw JSON from live calls. Save them even when they look boring; metered APIs - cost money to re-hit and the data may be gone tomorrow. +- `api_responses/` — raw JSON from live calls. Save them even when they look boring; the same request + may not return the same thing tomorrow. - Aggregate before saving. Counts, rates, and shapes answer nearly every question these investigations ask; this directory sits outside any repo and gets linked into tickets, so prefer a distribution over a dump of rows carrying editor identities or user data. @@ -199,8 +198,8 @@ same probe budget as one you can. Work the live ones in parallel rather than serially. Before firing, write beside each hypothesis the result that would kill it. Then issue the probes as independent tool calls in a single batch, backgrounding anything slow and handing a line that needs several dependent steps to a subagent so -the batch still returns together. Cap the first wave at four or five cheap, independent probes, and -hold metered calls and large scans for the second wave. Name each `queries/` and `results/` pair +the batch still returns together. Keep the first wave to four or five quick, independent probes, and +hold anything slow or wide for the second. Name each `queries/` and `results/` pair after its hypothesis, so a result cannot be attributed to the wrong line. Then look at what came back and do it again. Each round should either kill a line or sharpen the next @@ -220,8 +219,7 @@ An explicit `timestamp (UTC) | observation | source` table. - Normalise every timestamp to UTC and label it. Timezone mismatch is a common source of phantom gaps, and a subtraction error can manufacture an outage that never happened. - Extend the window to weeks, not hours — these failures are frequently much older than the report. - Widen freely on the cheap planes; on a billed plane, widen only after a narrow window has shown - you something. + Start with a window you can read quickly, and widen once it shows you where to look. - Distinguish *first occurrence* from *first noticed*, and state both. - Line the window up against the deploys and config changes the prior turned up. - Treat retention limits as limits: a "first seen" date can be the edge of a retention window rather diff --git a/plugins/newtab/skills/hnt-backend-investigation/references/data-sources.md b/plugins/newtab/skills/hnt-backend-investigation/references/data-sources.md index b69a7f2..b5f6d1e 100644 --- a/plugins/newtab/skills/hnt-backend-investigation/references/data-sources.md +++ b/plugins/newtab/skills/hnt-backend-investigation/references/data-sources.md @@ -10,7 +10,7 @@ investigating, rather than stopping or silently substituting a weaker source. **Keep bulk out of your context.** Several of these sources answer a small question with a very large payload, and a context spent on raw JSON is a context not spent on the investigation. Where the bulk arrives through a CLI or `curl`, redirect it to a file in the investigation directory and read back only -the slice you need; a `bq query` or `gcloud logging read` written to disk costs you nothing. Where it +the slice you need; a `bq query` or `gcloud logging read` written to disk stays out of context. Where it arrives through an MCP tool there is nothing to redirect, so hand that retrieval to a general-purpose subagent and keep only what it reports back. The ones that catch people out: @@ -195,7 +195,7 @@ region by `get_recommendation_surface_id` in `merino/curated_recommendations/uti branches on experiment enrolment — it is not a reformatting of the locale string. The `SurfaceId` enum itself lives in `merino/curated_recommendations/corpus_backends/protocol.py`. Stratifying by locale and stratifying by surface are therefore not the same slice, and some surfaces are reachable only through -enrolment. The cheapest resolution is a live response: it echoes the surface it resolved to. +enrolment. The quickest resolution is a live response: it echoes the surface it resolved to. Merino rewrites item URLs with `utm_source=firefox-newtab-` (`get_utm_source` and `update_url_utm_source` in `curated_recommendations/corpus_backends/utils.py`), so a URL from a @@ -263,11 +263,11 @@ Traps that will silently give you a wrong answer: column being introduced, not the crawl changing. - **`crawled_date` and `published_date` are STRING; `crawled_at`, `published_at` and `loaded_at` are TIMESTAMP.** Use the timestamps for any time arithmetic. -- **Cost does not work the way you expect.** `rss_feed_items` (tens of GB) and `zyte_cache` (over a - hundred GB) are **unpartitioned and unclustered**, so a date predicate reduces nothing — only column - selection does. Never `SELECT *` on them and always `--dry_run` first. The - `snowflake_migration_derived` tables are day-partitioned on `happened_at` but do not require a - partition filter, so supply one yourself. +- **A date filter does not narrow the big tables.** `rss_feed_items` (tens of GB) and `zyte_cache` + (over a hundred GB) are **unpartitioned and unclustered**, so a date predicate scans the whole thing + anyway; only naming fewer columns reduces it. Never `SELECT *` on them, and `--dry_run` first to see + what a query will actually touch. The `snowflake_migration_derived` tables are day-partitioned on + `happened_at` but do not require a partition filter, so supply one yourself. - **The `*_v1` event tables are event logs, not current state.** Rows accumulate per change, so a plain `COUNT(*)` over-counts. Where you must reduce the log yourself, take the latest row per id and check the inflation ratio rather than assuming it. @@ -345,9 +345,8 @@ queue-only. CloudWatch Logs Insights is frequently the source that cracks a case for these lambdas — it will give you total operation counts and per-error-type breakdowns that Sentry structurally cannot, because -Sentry only sees what was raised. It bills per GB scanned per query, so pass a reasonably small -`--start-time`/`--end-time` that could answer the question and `stats`-aggregate rather than dumping -`fields`. Also useful: alarm history (transition timestamps and state-reason margins), and pulling an +Sentry only sees what was raised. Pass a reasonably small `--start-time`/`--end-time` that could answer +the question, and `stats`-aggregate rather than dumping `fields`. Also useful: alarm history (transition timestamps and state-reason margins), and pulling an anomaly band itself as a metric-math series to compare its predicted centre against reality. The crawl runs in GCP, not AWS, so its equivalent plane is Cloud Logging or the BigQuery log sink. @@ -375,8 +374,8 @@ this file. ### Zyte -Two separate APIs. The extraction API is **metered — every call costs money**, so save all responses and -never bulk-crawl to satisfy curiosity. Check for a key with `[ -n "$ZYTE_API_KEY" ] && echo present`. Reference keys by +Two separate APIs, both talking to a third party. Save every response, and reproduce the one or two +URLs the question turns on rather than sweeping a domain. Check for a key with `[ -n "$ZYTE_API_KEY" ] && echo present`. Reference keys by variable name in anything you save, so no value lands in a query file or the transcript. **Extraction API** (`ZYTE_API_KEY`, created at https://app.zyte.com/o/612928/zyte-api/api-access) @@ -458,7 +457,7 @@ session back up where it left off Then carry on. The items stay pending and visible while you work; close them when they land, and delete any whose line stopped mattering. -Roughly ordered by how often an investigation needs them, cheapest first. +Roughly ordered by how often an investigation needs them, and how quickly they are resolved. | Blocked source | Ask them to | |---|---| From 08c8a722608e5dcce4a6020b18a0942f64bda7dc Mon Sep 17 00:00:00 2001 From: Matt Miermans Date: Mon, 10 Aug 2026 13:40:34 -0700 Subject: [PATCH 35/51] newtab/hnt-backend-investigation: name the memory scope, not the mechanics --- .../skills/hnt-backend-investigation/SKILL.md | 14 +++++--------- 1 file changed, 5 insertions(+), 9 deletions(-) diff --git a/plugins/newtab/skills/hnt-backend-investigation/SKILL.md b/plugins/newtab/skills/hnt-backend-investigation/SKILL.md index 2e43a25..8027d93 100644 --- a/plugins/newtab/skills/hnt-backend-investigation/SKILL.md +++ b/plugins/newtab/skills/hnt-backend-investigation/SKILL.md @@ -79,18 +79,14 @@ Each investigation gets its own subdirectory inside one root directory that the investigations in. Do not assume where that root is. Establish it in this order: 1. **Check the instructions you were already given.** A root recorded on an earlier run is already in - front of you: user-scope instructions load automatically at the start of every session. If they - name a root, use it and do not ask. + front of you, in your loaded instructions. If they name one, use it and do not ask. 2. **Otherwise ask, once, before your first write.** Where the developer keeps investigations cannot be derived, and guessing puts files somewhere they did not choose. Ask it as a single question at the start, and start pinning down the report while you wait rather than idling. -3. **Then record it**, so the question is asked once per machine rather than once per investigation. - Add one line to `~/.claude/CLAUDE.md`, creating that file if it does not exist: - `Investigations live in (one subdirectory per investigation).` That is the user scope, the - one that applies whichever repository you are launched from, and it will be loaded for you from the - next session onward. Say that you saved it. Do not put this in auto memory, which is keyed to the - repository you were launched from, while investigations get started from whichever service repo is - to hand. +3. **Then remember it**, so the question is asked once per machine rather than once per investigation. + Record the root and the one-subdirectory-per-investigation convention in **user-scope** memory, + `~/.claude/CLAUDE.md`, and say that you saved it. Not auto memory: that is keyed to the repository + you were launched from, while investigations get started from whichever service repo is to hand. Then create this investigation's subdirectory inside that root. If the root already holds investigations, follow the naming they use; otherwise `
-`, e.g. From a0906312f23a1be5bff23b4f611001c1c352aab4 Mon Sep 17 00:00:00 2001 From: Matt Miermans Date: Mon, 10 Aug 2026 13:41:47 -0700 Subject: [PATCH 36/51] newtab/hnt-backend-investigation: state memory scope as a requirement, not a path --- plugins/newtab/skills/hnt-backend-investigation/SKILL.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/plugins/newtab/skills/hnt-backend-investigation/SKILL.md b/plugins/newtab/skills/hnt-backend-investigation/SKILL.md index 8027d93..5f141a1 100644 --- a/plugins/newtab/skills/hnt-backend-investigation/SKILL.md +++ b/plugins/newtab/skills/hnt-backend-investigation/SKILL.md @@ -84,9 +84,9 @@ investigations in. Do not assume where that root is. Establish it in this order: be derived, and guessing puts files somewhere they did not choose. Ask it as a single question at the start, and start pinning down the report while you wait rather than idling. 3. **Then remember it**, so the question is asked once per machine rather than once per investigation. - Record the root and the one-subdirectory-per-investigation convention in **user-scope** memory, - `~/.claude/CLAUDE.md`, and say that you saved it. Not auto memory: that is keyed to the repository - you were launched from, while investigations get started from whichever service repo is to hand. + Remember the root and the one-subdirectory-per-investigation convention, and say that you saved it. + It has to persist for **every** repository, not only the one you happen to be in: investigations get + started from whichever service repo is to hand. Then create this investigation's subdirectory inside that root. If the root already holds investigations, follow the naming they use; otherwise `
-`, e.g. From 8143d22a8fb6881af9231a010634dbbfeb25fbf6 Mon Sep 17 00:00:00 2001 From: Matt Miermans Date: Mon, 10 Aug 2026 13:44:25 -0700 Subject: [PATCH 37/51] newtab: drop the tool list from the README entry --- plugins/newtab/README.md | 15 ++------------- 1 file changed, 2 insertions(+), 13 deletions(-) diff --git a/plugins/newtab/README.md b/plugins/newtab/README.md index 5c215c7..b413626 100644 --- a/plugins/newtab/README.md +++ b/plugins/newtab/README.md @@ -12,17 +12,6 @@ Use this skill when adding cleanup comments to Nova-related changes. See [`skill ### `hnt-backend-investigation` -Diagnostic workflow for Home New Tab **backend** incidents, feature by feature: content recommendations (Merino, the curated corpus, admin-api, the article crawler, the ML section pipeline, the New Tab data pipelines), Picture of the Day, the daily crossword, and other New Tab features served the same way. Its output is an evidence-backed `FINDINGS.md` carrying a root cause and quantified impact. +Diagnostic workflow for Home New Tab **backend** incidents, covering content recommendations, Picture of the Day, the daily crossword, and other New Tab features served the same way. It produces an evidence-backed `FINDINGS.md` with a root cause, quantified impact, and the change that would resolve it. -Use this skill when a Sentry alert fires, an editor reports something broken, or a New Tab feature looks empty, wrong, or stale. See [`skills/hnt-backend-investigation/SKILL.md`](skills/hnt-backend-investigation/SKILL.md) for details. - -**Optional tools.** Anything unreachable is raised as a task and reported under "Could not measure": - -- `gh` -- `gcloud` and `bq` -- the `aws` CLI, with a read-only SSO profile -- a MySQL client, plus the AWS Client VPN -- `uv` or `python3` -- the Sentry MCP server -- the Slack MCP server, for `#hnt-dev-be-alerts` -- Zyte API keys, for reproducing an extraction or reading the vendor's stats +Use this skill when a Sentry alert fires, an editor reports something broken, or a New Tab feature looks empty, wrong, or stale. It works with whatever access you already have, and where it needs a tool or a credential it does not have, it tells you what to install or request and how. See [`skills/hnt-backend-investigation/SKILL.md`](skills/hnt-backend-investigation/SKILL.md) for details. From 98017b5cd42222662353d92ccad5ef9bef085850 Mon Sep 17 00:00:00 2001 From: Matt Miermans Date: Mon, 10 Aug 2026 13:49:14 -0700 Subject: [PATCH 38/51] newtab/hnt-backend-investigation: keep per-source detail in data-sources --- .../skills/hnt-backend-investigation/SKILL.md | 17 ++++++----------- 1 file changed, 6 insertions(+), 11 deletions(-) diff --git a/plugins/newtab/skills/hnt-backend-investigation/SKILL.md b/plugins/newtab/skills/hnt-backend-investigation/SKILL.md index 5f141a1..1fc52f8 100644 --- a/plugins/newtab/skills/hnt-backend-investigation/SKILL.md +++ b/plugins/newtab/skills/hnt-backend-investigation/SKILL.md @@ -134,15 +134,10 @@ keeps, live signal does not. **An error-rate signal is not yet a symptom.** Measure the erroring stage's *output* to fix the blast radius — items per section and per-surface freshness for assembly, the newest object under a dated prefix for a publish job, a live Merino request for anything client-facing — so you can say -whether Firefox clients are affected at all. An -error floor that never reaches the output is a not-incident; errors flat with output at zero is -worse than the alert says. When the report is a Sentry alert, resolve it to a concrete error and a -volume, and decompose the issue by error message before trusting its title or its trend. An alert -can be mis-centred, watching the wrong layer, or counting per open period rather than per day, so -confirm the signal itself before chasing the system behind it. - -Sentry is the plane most of this reads from, so if the `mcp__sentry__` tools are absent, say so and -raise it like any other blocked source, then carry on with the planes you do have. +whether Firefox clients are affected at all. An error floor that never reaches the output is a +not-incident; errors flat with output at zero is worse than the alert says. And when the report is an +alert rather than something someone saw, confirm the signal itself before chasing the system behind +it: an alert can be mis-centred, watching the wrong layer, or counting over the wrong period. **If you cannot reproduce it, that is a result, not a blocker.** Record the exact attempt — surface, locale, time, request, what you saw instead — then switch the question to why the reporter saw it @@ -218,8 +213,8 @@ An explicit `timestamp (UTC) | observation | source` table. Start with a window you can read quickly, and widen once it shows you where to look. - Distinguish *first occurrence* from *first noticed*, and state both. - Line the window up against the deploys and config changes the prior turned up. -- Treat retention limits as limits: a "first seen" date can be the edge of a retention window rather - than onset. See `references/data-sources.md`. +- A source with a retention window cannot establish onset: its earliest record may simply be its + oldest. - **The scheduling layer is not in UTC.** Scheduled dates and the assembly crons run in each surface's own timezone, so a UTC comparison invents a one-day gap for part of every day on any surface offset from UTC — worst for the Americas, and `en-US` is the largest surface. Convert per surface, and say From de61fd6f631ec43afff2db22cd930190045d17bb Mon Sep 17 00:00:00 2001 From: Matt Miermans Date: Mon, 10 Aug 2026 13:50:08 -0700 Subject: [PATCH 39/51] newtab/hnt-backend-investigation: focus the description on when to use it --- plugins/newtab/skills/hnt-backend-investigation/SKILL.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/plugins/newtab/skills/hnt-backend-investigation/SKILL.md b/plugins/newtab/skills/hnt-backend-investigation/SKILL.md index 1fc52f8..194233f 100644 --- a/plugins/newtab/skills/hnt-backend-investigation/SKILL.md +++ b/plugins/newtab/skills/hnt-backend-investigation/SKILL.md @@ -1,6 +1,6 @@ --- name: hnt-backend-investigation -description: Investigates Home New Tab backend errors, outages, and data-quality problems in whichever New Tab feature the symptom lands in, whether content recommendations (Merino, the curated corpus, admin-api, the article crawler, the ML section pipeline, the New Tab data pipelines), Picture of the Day, the daily crossword, or a feature not named here. Confirms the symptom independently, probes competing hypotheses in parallel, stratifies metrics, tries to break its own conclusion, then writes an evidence-backed FINDINGS doc with a root cause and quantified impact. Use when a Sentry alert fires, an editor reports something broken, or a New Tab feature looks empty, wrong, or stale — recommendations, the picture of the day, the puzzle. For backend investigation, not in-tree browser/extensions/newtab frontend work. +description: Diagnoses backend problems behind Firefox Home New Tab, across Merino, the curated corpus, admin-api, the article crawler, Picture of the Day and the daily crossword. Use when a Sentry alert fires on one of those services, an editor reports something broken, or a New Tab feature looks empty, wrong, or stale. Backend only, not in-tree browser/extensions/newtab frontend work. --- # New Tab (HNT) Backend Investigation From 78d43ffb3456e82d3e726462d696de9358db5fff Mon Sep 17 00:00:00 2001 From: Matt Miermans Date: Mon, 10 Aug 2026 13:58:33 -0700 Subject: [PATCH 40/51] newtab/hnt-backend-investigation: trim to preferences, create dirs on demand --- .../skills/hnt-backend-investigation/SKILL.md | 267 +++++++----------- 1 file changed, 106 insertions(+), 161 deletions(-) diff --git a/plugins/newtab/skills/hnt-backend-investigation/SKILL.md b/plugins/newtab/skills/hnt-backend-investigation/SKILL.md index 194233f..9a9b9bb 100644 --- a/plugins/newtab/skills/hnt-backend-investigation/SKILL.md +++ b/plugins/newtab/skills/hnt-backend-investigation/SKILL.md @@ -23,29 +23,17 @@ clients**, meaning everyone the affected feature serves, and, where the feature Access and traps, per system: [references/data-sources.md](references/data-sources.md) -## How to think about this +## How we like this done -**Confirm the symptom before explaining it.** Independently reproduce or measure the reported -problem first, whatever its source. An alert can be miscalibrated, mis-scoped, or watching the -wrong layer, and a description passed through two people can drift. Until you have seen the -symptom yourself in data you pulled, you do not know what you are diagnosing. +You know how to investigate. These are the preferences that make an investigation here land well: -**Form the hypothesis before you run the check.** Write down what you expect to see, and what you -would expect to see if the hypothesis were false. A query designed to confirm and a query designed -to discriminate look similar and are not. The characteristic failure in this domain is a correct -query paired with a wrong inference, stated confidently. - -**Work several lines at once.** When more than one explanation is plausible, probe them in parallel -rather than following the most attractive one to its end. Tunnel vision is the failure that wastes the -most time, and a probe that comes back negative has still moved you forward. Keep a list of every line -you considered and record which you dropped and why. - -**Prefer measuring to reasoning.** Reproduce the request. Run the model on real inputs. Count the -actual rows. Read the code path instead of assuming its behaviour. A plausible mechanism becomes a -number, and only numbers survive review. - -Write like the evidence is asymmetric, because it usually is. Avoid "always", "never", "in every -case"; say what each count is a count of, and whether it is a floor, a ceiling, or a sample. +- **Confirm the symptom yourself before explaining it**, whatever the source. An alert can be + mis-centred or watching the wrong layer, and a description passed through two people drifts. +- **Let the data generate the hypotheses**, one round at a time, rather than enumerating everything up + front. Where several are live at once, probe them in parallel instead of serially. +- **Measure rather than reason.** Reproduce the request, count the rows, read the code path. The + characteristic failure here is a correct query paired with a wrong inference, stated confidently. +- **Keep going while you wait.** Blocked access and unanswered questions are not stopping points. ## Don't stall on the developer @@ -65,13 +53,13 @@ name the view, the filter, and the single number or shape you need, and ask them Restate it exactly once more — whichever comes first: the unblocked lines run out, or you are three probes into a line you had already judged weaker than the blocked one. That is the second and last -ask, and it does not apply if they have already answered. If there is still no response, write up and close out with `Status: blocked`, -name the one unblock under "Could not measure", and leave the access task pending. +ask, and it does not apply if they have already answered. If there is still no response, write up and +close out with `Status: blocked`, name the one unblock under "Could not measure", and leave the access +task pending. Stopping outright is a last resort: only when the blocked source is the only thing that can settle the question **and** it cannot be reduced to a question they can answer for you — a dashboard, -console, or explore is almost never a real stop. A request that quietly disappears is worse than one -that was never made. +console, or explore is almost never a real stop. ## Workspace @@ -93,30 +81,26 @@ investigations, follow the naming they use; otherwise `
-`, e.g. `jul29-empty-de-sections`. Say which directory you created. Reuse an existing one only when its FINDINGS.md is about the same symptom. -- `FINDINGS.md` — one living document. Create it from the skeleton in the write-up step the moment the directory - exists, and in any case before you write a conclusion down; every step that follows writes into it as - it goes, so the write-up step is a final pass over a document that already exists. If you are still waiting on the - root, run the first probes anyway and hold their output until you have somewhere to put it. Rewrite - it in place, not as `FINDINGS-v2.md`. -- `queries/` and `results/` — every query as a file, its output alongside under the same basename. -- `api_responses/` — raw JSON from live calls. Save them even when they look boring; the same request - may not return the same thing tomorrow. -- Aggregate before saving. Counts, rates, and shapes answer nearly every question these - investigations ask; this directory sits outside any repo and gets linked into tickets, so prefer a - distribution over a dump of rows carrying editor identities or user data. -- Rewrite only the FINDINGS.md you wrote and delete only files this session created — anything else - in the directory belongs to another investigation. -- Python work goes in an isolated venv inside this directory: `uv venv` if `uv` is installed, - otherwise `python3 -m venv`. +- `FINDINGS.md` — one living document, from the skeleton in the write-up step. Create it once the + directory exists and in any case before you write a conclusion down; the steps that follow write into + it as they go, so the write-up step is a final pass over a document that already exists. Rewrite it in + place, not as `FINDINGS-v2.md`. +- Add subdirectories only when you have something to put in them, not up front: `queries/` and + `results/` for each query and its output under a matching basename, `api_responses/` for raw JSON + from live calls. Save responses even when they look boring; the same request may not return the same + thing tomorrow. +- Aggregate before saving. This directory sits outside any repo and gets linked into tickets, so prefer + a distribution over a dump of rows carrying editor identities or user data. +- Rewrite only the FINDINGS.md you wrote and delete only files this session created — anything else in + the directory belongs to another investigation. +- Python work goes in an isolated venv inside this directory. ## Step 1 — pin down the report Before your first probe, read [references/data-sources.md](references/data-sources.md). It carries the traps that silently return wrong answers, and several of them look like an outage. -Establish these five facts from the data. The alert or issue carries the symptom and its volume, -stratifying gives you the surface and locale, and a live request tells you whether it is still -happening. +Establish these five facts from the data. | Fact | Why it matters | |---|---| @@ -126,143 +110,106 @@ happening. | The stratum — surface, locale, section, client version, or for a daily artifact the date | The stratum is very often the diagnosis | | Still happening right now? | Live reproduction vs. historical forensics | -**If it is still happening, capture the perishable evidence first** — a live request for the feature, -current logs, and whatever counts or artifact it publishes — and save the raw responses under -`api_responses/`. Establish what normal is while those probes are in flight; history -keeps, live signal does not. - -**An error-rate signal is not yet a symptom.** Measure the erroring stage's *output* to fix the -blast radius — items per section and per-surface freshness for assembly, the newest object under a -dated prefix for a publish job, a live Merino request for anything client-facing — so you can say -whether Firefox clients are affected at all. An error floor that never reaches the output is a -not-incident; errors flat with output at zero is worse than the alert says. And when the report is an -alert rather than something someone saw, confirm the signal itself before chasing the system behind -it: an alert can be mis-centred, watching the wrong layer, or counting over the wrong period. - -**If you cannot reproduce it, that is a result, not a blocker.** Record the exact attempt — surface, -locale, time, request, what you saw instead — then switch the question to why the reporter saw it -and you do not: a stratum you did not hit, a window that has closed, a cache, a client version, or a -monitor measuring something other than what it claims. Classify as `not-incident` only after that -second question has an answer; unconfirmed is not refuted. +If it is still happening, capture the perishable evidence first — a live request for the feature, +current logs, whatever counts or artifact it publishes — and establish what normal is while those +probes are in flight. History keeps; live signal does not. -A report relayed from an editor is ambiguous between what editors see in curation-admin-tools and -what clients see on the surface. Probe both in parallel and state which group you confirmed. +**An error rate is not yet a symptom.** Measure the erroring stage's *output* to fix the blast radius: +items per section and per-surface freshness for assembly, the newest object under a dated prefix for a +publish job, a live Merino request for anything client-facing. An error floor that never reaches the +output is a not-incident; errors flat with output at zero is worse than the alert says. + +**Failing to reproduce is a result, not a blocker.** Record the exact attempt, then switch the question +to why the reporter saw it and you do not: a stratum you did not hit, a closed window, a cache, a client +version, or a monitor measuring something other than what it claims. `not-incident` needs an answer to +that second question; unconfirmed is not refuted. -The moment you confirm client-visible impact that is still happening, state it in one line — what, -how big, since when — and keep investigating. If that deserves a heads-up in `#hnt-dev-be-alerts`, -follow the posting rule under "Posting to `#hnt-dev-be-alerts`". +A report relayed from an editor is ambiguous between what editors see in curation-admin-tools and +what clients see on the surface. Probe both and state which group you confirmed. Once client-visible +impact is confirmed and still happening, say so in one line — what, how big, since when — and keep +going. ## Step 2 — establish what "normal" is -You do not know what is currently normal in this system, so you have no basis on which to reject -your own inference. Build that prior yourself before asking for it — most of it is in reach: +You have no idea what is currently normal here, so you have no basis on which to reject your own +inference. Derive the prior rather than asking for it: - **What the serving path returns now** — a live request for the feature; for recommendations, the surfaces present and not disabled in the section data. - **What the producer last wrote** — per-surface, per-source crawl volume over the last day for recommendations, the newest object under the dated bucket prefix for a publish job. Producer and serving sets differ, and a crawled surface with no sections is not automatically a bug. -- **What shipped recently** — recent commits in the service repos, releases in Sentry, timestamps on - model or config artifacts. +- **What shipped recently** — commits in the service repos, releases in Sentry, timestamps on model or + config artifacts. - **Whether this is normally noisy or seasonal** — a trailing profile of the same metric, by day of week. Do not ask; compute it. -- **Whether it has happened before** — search Sentry for the same signature and the service repos' - GitHub issues. Where one of those is unreachable, record it as unchecked rather than empty; "nothing - similar has fired" is a claim about a source you actually read. +- **Whether it has happened before** — Sentry for the same signature, and the service repos' GitHub + issues. Where one is unreachable, record it as unchecked rather than empty. -What is left is genuinely in the developer's head: anything in flight that a repo or a dashboard -would not show, and whether a postmortem or incident register already covers this area. Raise that -as one short note and carry on without waiting; an existing write-up can answer in a paragraph what -costs hours to reconstruct. +What is left is genuinely in the developer's head: anything in flight that a repo or dashboard would +not show, and whether a postmortem or incident register already covers this area. Raise that as one +short note and carry on without waiting. -Record what you derive and what they tell you in FINDINGS.md, keeping the two apart. When a later -query contradicts the prior, treat your own inference as the suspect first, and say so. +Keep what you derived and what they told you apart in FINDINGS.md. When a later query contradicts the +prior, treat your own inference as the suspect first, and say so. ## Step 3 — form hypotheses from the data, then probe them in parallel -Hypotheses come from what you have already measured, not from a catalogue. Start where pinning down -the report put you: the stage whose output is wrong, the stratum that is affected, the moment it changed. Ask what -could produce exactly that, follow the data one hop upstream, and let each result generate the next -question. A hypothesis you cannot tie to something you have observed is a guess competing for the -same probe budget as one you can. - -Work the live ones in parallel rather than serially. Before firing, write beside each hypothesis the -result that would kill it. Then issue the probes as independent tool calls in a single batch, -backgrounding anything slow and handing a line that needs several dependent steps to a subagent so -the batch still returns together. Keep the first wave to four or five quick, independent probes, and -hold anything slow or wide for the second. Name each `queries/` and `results/` pair -after its hypothesis, so a result cannot be attributed to the wrong line. - -Then look at what came back and do it again. Each round should either kill a line or sharpen the next -question; when a round kills everything, the data has moved you upstream rather than left you stuck. -Being down to a single surviving hypothesis is a prompt, not an answer: ask what else could produce -exactly what you measured before you commit to it. - -Two moves consistently break cases open: **leave the tool you started in** — when a source stops -yielding, measure the same thing somewhere else, since the failure is often invisible in the plane -you began with — and **compare against sibling strata**, which localises a fault faster than reading -code. +Start where pinning down the report left you: the stage whose output is wrong, the affected stratum, +the moment it changed. Ask what could produce exactly that, follow the data one hop upstream, and let +each result raise the next question. + +Work the live hypotheses in parallel. Write beside each one the result that would kill it, then issue +the probes as independent calls in a single batch, handing any line that needs several dependent steps +to a subagent. Keep the first wave small and quick, and hold anything slow or wide for the second. +Name each query file after its hypothesis so a result cannot be attributed to the wrong line. + +Then look at what came back and do it again. Being down to one surviving hypothesis is a prompt, not an +answer: ask what else could produce what you measured before committing to it. And when a source stops +yielding, measure the same thing in another plane — the failure is often invisible in the one you +started in. ## Step 4 — build the timeline -An explicit `timestamp (UTC) | observation | source` table. - -- Normalise every timestamp to UTC and label it. Timezone mismatch is a common source of phantom - gaps, and a subtraction error can manufacture an outage that never happened. -- Extend the window to weeks, not hours — these failures are frequently much older than the report. - Start with a window you can read quickly, and widen once it shows you where to look. -- Distinguish *first occurrence* from *first noticed*, and state both. -- Line the window up against the deploys and config changes the prior turned up. -- A source with a retention window cannot establish onset: its earliest record may simply be its - oldest. -- **The scheduling layer is not in UTC.** Scheduled dates and the assembly crons run in each surface's - own timezone, so a UTC comparison invents a one-day gap for part of every day on any surface offset - from UTC — worst for the Americas, and `en-US` is the largest surface. Convert per surface, and say - which timezone you used. +An explicit `timestamp (UTC) | observation | source` table. Normalise and label every timestamp, since +a subtraction error can manufacture an outage that never happened. Extend the window to weeks rather +than hours; these failures are frequently older than the report. Distinguish first occurrence from +first noticed, line the window up against the deploys the prior turned up, and remember that a source +with a retention window cannot establish onset — its earliest record may simply be its oldest. -## Step 5 — stratify before concluding anything +**The scheduling layer is not in UTC.** Scheduled dates and the assembly crons run in each surface's +own timezone, so a UTC comparison invents a one-day gap for part of every day on any surface offset +from UTC — worst for the Americas, and `en-US` is the largest surface. Convert per surface, and say +which timezone you used. -An aggregate that looks fine is the normal way these problems hide. Slice every metric by the -dimensions the feature actually has — for recommendations **domain, locale, region, surface, section, -experiment branch, and client/addon version**, for a once-a-day global artifact barely more than the -date — and look for a single stratum at zero or down sharply against its own trailing median rather -than against yesterday. When the report says "some users" with no stratum attached, treat experiment branch, -region, and rollout state as the first cuts — some surfaces are reachable only through experiment -enrolment, so an enrolled/unenrolled split is invisible to a locale slice. +## Step 5 — stratify before concluding anything -Strata are not interchangeable, and the volumes are strongly seasonal; both traps, and the -per-surface source mix, are detailed in `references/data-sources.md`. Compare each stratum against -its own history. +An aggregate that looks fine is the normal way these problems hide. Slice by the dimensions the feature +actually has — for recommendations **domain, locale, region, surface, section, experiment branch, and +client/addon version**, for a once-a-day global artifact barely more than the date — and look for a +single stratum at zero or down sharply against its own trailing median rather than against yesterday. +When the report says "some users" with no stratum attached, try experiment branch, region and rollout +state first: some surfaces are reachable only through enrolment, which a locale slice cannot see. -**Liveness is not health.** "The job ran successfully" is compatible with total data loss in every -one of these pipelines. Count what came out, compare it against what went in, and never accept a -green run as evidence. +**Liveness is not health.** "The job ran successfully" is compatible with total data loss in every one +of these pipelines. Count what came out and compare it against what went in. -If the measurements exonerate the backend, stop at the service boundary. Read far enough to name the -owning team and the contract that is being broken, then hand it over rather than continuing into a -system this skill does not cover. A second, unrelated anomaly you trip over on the way is a one-line -note plus a task, not a second investigation. +If the measurements exonerate the backend, stop at the service boundary: name the owning team and the +contract being broken, and hand it over. A second, unrelated anomaly you trip over on the way is a +one-line note plus a task, not a second investigation. ## Step 6 — try to break your own conclusion -Before writing anything down as fact, attack it: - -- Check the boring explanations first: timezone; a filter in your own query; a `LIMIT` silently - truncating; an unrepresentative code path; sampling; a column that does not mean what its name - says; a partially-launched feature. -- Then attack the mechanism itself, with whichever of these would break yours: read the code path - rather than assuming it; enumerate exactly instead of sampling, since "0 of N" is a different claim - from "none in my sample"; repeat a suspicious observation before calling anything - non-deterministic; read the real limit rather than guessing which one binds; and check the history - of the constant or config you are relying on. -- **Measure the effect, not just the mechanism.** A guard that provably runs is not evidence that - the outcome is correct; verify the outcome separately. -- Cross-check against a second, independent source. One source is a hypothesis. -- Re-read the prior you established. If your conclusion implies that something it says is working is - broken, re-check your own measurement first — a filter in your query, a stratum mismatch, or a - wrong surface identifier is the likelier explanation. If the measurement survives that re-check, - the measurement wins: state the contradiction explicitly in FINDINGS.md, note it to the developer - in one line, and keep going. +Before writing anything down as fact, check the boring explanations: a filter in your own query, a +silent truncation, an unrepresentative code path, a column that does not mean what its name says, a +partially-launched feature. Then verify the outcome and not just the mechanism — a guard that provably +runs is not evidence that its effect is correct — and cross-check against a second, independent source, +because one source is a hypothesis. + +Re-read the prior. If your conclusion implies that something the prior says is working is broken, +re-check your own measurement first; a stratum mismatch or a wrong surface identifier is the likelier +explanation. If the measurement survives that, the measurement wins: state the contradiction in +FINDINGS.md, note it to the developer in one line, and keep going. Mark every claim **verified**, **inferred**, or **refuted**. Keep the refuted ones in the document; they stop the next person re-running them. @@ -284,8 +231,7 @@ Three sentences: what broke, since when, what the user-visible effect is. | Time | Observation | Source | ## Prior -**Derived** — the feature's own volumes or published artifacts, recent deploys and model artifacts, -trailing profile, prior occurrences. Each with the query or link. +**Derived** — what you measured, each with the query or link. **Reported** — verbatim answers from the developer, or `asked