diff --git a/plugins/newtab/README.md b/plugins/newtab/README.md index 407457f..cf755c8 100644 --- a/plugins/newtab/README.md +++ b/plugins/newtab/README.md @@ -9,3 +9,9 @@ Skills and tools for Firefox newtab development. Adds `@nova-cleanup` comments to Firefox newtab code for Project Nova parallel implementation tracking. Use this skill when adding cleanup comments to Nova-related changes. See [`skills/nova-cleanup-comments/SKILL.md`](skills/nova-cleanup-comments/SKILL.md) for details. + +### `backend-investigation` + +Diagnostic workflow for Home New Tab **backend** incidents, covering content recommendations, Picture of the Day, the daily crossword, and other New Tab features served the same way. It produces an evidence-backed `FINDINGS.md` with a root cause, quantified impact, and the change that would resolve it. + +Use this skill when a Sentry alert fires, an editor reports something broken, or a New Tab feature looks empty, wrong, or stale. It works with whatever access you already have, and where it needs a tool or a credential it does not have, it tells you what to install or request and how. See [`skills/backend-investigation/SKILL.md`](skills/backend-investigation/SKILL.md) for details. diff --git a/plugins/newtab/skills/backend-investigation/SKILL.md b/plugins/newtab/skills/backend-investigation/SKILL.md new file mode 100644 index 0000000..d99fb7a --- /dev/null +++ b/plugins/newtab/skills/backend-investigation/SKILL.md @@ -0,0 +1,298 @@ +--- +name: backend-investigation +description: Diagnoses backend problems behind Firefox Home New Tab (HNT), across Merino, the curated corpus, admin-api, the article crawler, Picture of the Day and the daily crossword. Use when a Sentry alert fires on one of those services, an editor reports something broken, or a New Tab feature looks empty, wrong, or stale. Backend only, not in-tree browser/extensions/newtab frontend work. +--- + +# New Tab (HNT) Backend Investigation + +Diagnose a problem in the backend services behind Firefox Home New Tab. The page is assembled from +features that are served separately and fail separately — content recommendations (Merino, the +curated corpus, admin-api, the article crawler, the ML section pipeline, the data pipelines), +Picture of the Day, the daily crossword — so establish which one the symptom belongs to before +probing. A New Tab feature with no notes in this skill is still yours to diagnose: work it the same +way, from the Merino provider and config that serve it. Not for in-tree `browser/extensions/newtab` +frontend bugs, and not for Merino's non-New-Tab consumers such as Firefox Suggest. + +What you owe first is a **diagnosis**: a root cause, the evidence for it, and the impact quantified. +Mitigating or fixing the problem usually follows straight from that, so carry the diagnosis far enough +to name the change that would resolve it and what would show the recovery. + +Two groups feel these failures, and every finding should say which one it reached: **Firefox New Tab +clients**, meaning everyone the affected feature serves, and, where the feature has them, the +**editors and curators** working through curation-admin-tools and admin-api. + +Access and traps, per system: [references/data-sources.md](references/data-sources.md) + +## How we like this done + +You know how to investigate. These are the preferences that make an investigation here land well: + +- **Confirm the symptom yourself before explaining it**, whatever the source reported. +- **Let the data generate the hypotheses**, one round at a time, rather than enumerating everything up + front. Where several are live at once, probe them in parallel instead of serially. +- **Measure rather than reason.** Reproduce the request, count the rows, read the code path. The + characteristic failure here is a correct query paired with a wrong inference, stated confidently. + +## Don't stall on the developer + +Every pause costs wall-clock, because the developer may not be watching. Default to proceeding: +pick the sensible option, say which one you picked, and keep going. Ask a blocking question only +when proceeding on any assumption would waste the whole investigation. + +When a source you cannot reach looks promising, raise it **once** as a pending task carrying the +whole errand in its own text, prefixed `User action:` so it reads as theirs rather than yours — e.g. +`User action: connect to VPN so corpus MySQL is reachable — settles whether the DE items exist at +all`. Give the exact command, not a description of the problem; the "Access requests" section of +`references/data-sources.md` has the command for each gated source and how to word the ask. If the +answer lives in a dashboard or console you have no credential for, do not ask for access at all — +name the view, the filter, and the single number or shape you need, and ask them to read it back. + +**Then keep investigating the other hypotheses immediately. Do not wait.** + +Restate it exactly once more — whichever comes first: the unblocked lines run out, or you are three +probes into a line you had already judged weaker than the blocked one. That is the second and last +ask, and it does not apply if they have already answered. If there is still no response, write up and +close out with `Status: blocked`, name the one unblock under "Could not measure", and leave the access +task pending. + +Stopping outright is a last resort: only when the blocked source is the only thing that can settle +the question **and** it cannot be reduced to a question they can answer for you — a dashboard, +console, or explore is almost never a real stop. + +## Workspace + +Each investigation gets its own subdirectory inside one root directory that the developer keeps their +investigations in. Do not assume where that root is. Establish it in this order: + +1. **Check the instructions you were already given.** A root recorded on an earlier run is already in + front of you, in your loaded instructions. If they name one, use it and do not ask. +2. **Otherwise ask, once, before your first write.** Where the developer keeps investigations cannot + be derived, and guessing puts files somewhere they did not choose. Ask it as a single question at + the start, and start pinning down the report while you wait rather than idling. +3. **Then remember it**, so the question is asked once per machine rather than once per investigation. + Remember the root and the one-subdirectory-per-investigation convention, and say that you saved it. + It has to persist for **every** repository, not only the one you happen to be in: investigations get + started from whichever service repo is to hand. + +Then create this investigation's subdirectory inside that root. If the root already holds +investigations, follow the naming they use; otherwise `
-`, e.g. +`jul29-empty-de-sections`. Say which directory you created. Reuse an existing one only when its +FINDINGS.md is about the same symptom. + +- `FINDINGS.md` — one living document, from the skeleton in the write-up step. Create it once the + directory exists and in any case before you write a conclusion down; the steps that follow write into + it as they go, so the write-up step is a final pass over a document that already exists. Rewrite it in + place, not as `FINDINGS-v2.md`. +- `HYPOTHESES.md` — the working list, live and dead, each with what would kill it. Hypotheses go here + rather than in FINDINGS.md, which carries the conclusion they led to. +- Add subdirectories only when you have something to put in them, not up front: `queries/` and + `results/` for each query and its output under a matching basename, `api_responses/` for raw JSON + from live calls. Save responses even when they look boring; the same request may not return the same + thing tomorrow. +- Aggregate before saving. This directory sits outside any repo and gets linked into tickets, so prefer + a distribution over a dump of rows carrying editor identities or user data. +- Rewrite only the FINDINGS.md you wrote and delete only files this session created — anything else in + the directory belongs to another investigation. +- Python work goes in an isolated venv inside this directory. + +## Keep a timeline throughout + +Maintain a `timestamp (UTC) | observation | source` table as you go rather than assembling one at the +end. Where the onset is cheap to establish, get it early: it constrains every other line of enquiry. +Where it is not, leave it open and say so. An assumed onset is worse than an unknown one, because +everything downstream quietly inherits it. + +Normalise and label every timestamp, since a subtraction error can manufacture an outage that never +happened, and line the window up against the deploys and config changes you turned up. + +**The scheduling layer is not in UTC.** Scheduled dates and the assembly crons run in each surface's +own timezone, so a UTC comparison invents a one-day gap for part of every day on any surface offset +from UTC — worst for the Americas, and `en-US` is the largest surface. Convert per surface, and say +which timezone you used. + +## Step 1 — pin down the report + +Before your first probe, read [references/data-sources.md](references/data-sources.md). It carries the +traps that silently return wrong answers, and several of them look like an outage. + +Establish these five facts from the data. + +| Fact | Why it matters | +|---|---| +| Exact symptom, verbatim, and which feature | "Empty section" and "wrong items in section" have disjoint causes | +| Who noticed, and how | Alert / editor report / spotted by hand — sets what evidence exists | +| First and last seen, **with timezone** | Anchors the timeline; reports usually arrive late | +| The stratum — surface, locale, section, client version, or for a daily artifact the date | The stratum is very often the diagnosis | +| Still happening right now? | Live reproduction vs. historical forensics | + +If it is still happening, capture the perishable evidence first — a live request for the feature, +current logs, whatever counts or artifact it publishes — and establish what normal is while those +probes are in flight. History keeps; live signal does not. + +**An error rate is not yet a symptom.** Measure the erroring stage's *output* to fix the blast radius: +items per section and per-surface freshness for assembly, the newest object under a dated prefix for a +publish job, a live Merino request for anything client-facing. An error floor that never reaches the +output is a not-incident; errors flat with output at zero is worse than the alert says. + +**Failing to reproduce is a result, not a blocker.** Record the exact attempt, then switch the question +to why the reporter saw it and you do not: a stratum you did not hit, a closed window, a cache, a client +version, or a monitor measuring something other than what it claims. `not-incident` needs an answer to +that second question; unconfirmed is not refuted. + +A report relayed from an editor is ambiguous between what editors see in curation-admin-tools and +what clients see on the surface. Probe both and state which group you confirmed. Once client-visible +impact is confirmed and still happening, say so in one line — what, how big, since when — and keep +going. + +## Step 2 — establish what "normal" is + +You have no idea what is currently normal here, so you have no basis on which to reject your own +inference. Derive the prior rather than asking for it: + +- **What the serving path returns now** — a live request for the feature; for recommendations, the + surfaces present and not disabled in the section data. +- **What the producer last wrote** — per-surface, per-source crawl volume over the last day for + recommendations, the newest object under the dated bucket prefix for a publish job. Producer and + serving sets differ, and a crawled surface with no sections is not automatically a bug. +- **What shipped recently** — commits in the service repos, releases in Sentry, timestamps on model or + config artifacts. +- **Whether this is normally noisy or seasonal** — a trailing profile of the same metric, by day of + week. Do not ask; compute it. +- **Whether it has happened before** — Sentry for the same signature, and the service repos' GitHub + issues. Where one is unreachable, record it as unchecked rather than empty. + +What is left is genuinely in the developer's head: anything in flight that a repo or dashboard would +not show, and whether a postmortem or incident register already covers this area. Raise that as one +short note and carry on without waiting. + +Keep what you derived and what they told you apart in FINDINGS.md. When a later query contradicts the +prior, treat your own inference as the suspect first, and say so. + +## Step 3 — form hypotheses from the data, then probe them in parallel + +Start where pinning down the report left you: the stage whose output is wrong, the affected stratum, +the moment it changed. Ask what could produce exactly that, follow the data one hop upstream, and let +each result raise the next question. + +Keep them in `HYPOTHESES.md`, a row each, and write the result that would kill one before you go +looking: + +```markdown +| # | Hypothesis | What would kill it | Probe | Result | Status | +``` + +Status is live, killed, parked, or confirmed. Killed rows stay; they are what stops the next person +re-running them. + +Work the live ones in parallel: issue the probes as parallel calls in a single batch, handing any line +that needs several dependent steps to a subagent. Keep the first wave small and quick, and hold +anything slow or wide for the second. Name each query file after its hypothesis so a result cannot be +attributed to the wrong line. + +Then look at what came back, update the rows, and do it again. Being down to one surviving hypothesis +is a prompt, not an answer: ask what else could produce what you measured before committing to it. And when a source stops +yielding, measure the same thing in another plane — the failure is often invisible in the one you +started in. + +## Step 4 — stratify before concluding anything + +An aggregate that looks fine is the normal way these problems hide. Slice by the dimensions the feature +actually has — for recommendations **domain, locale, region, surface, section, experiment branch, and +client/addon version**, for a once-a-day global artifact barely more than the date — and look for a +single stratum at zero or down sharply against its own trailing median. +When the report says "some users" with no stratum attached, try experiment branch, region and rollout +state first: some surfaces are reachable only through enrolment, which a locale slice cannot see. + +**Liveness is not health.** "The job ran successfully" is compatible with total data loss in every one +of these pipelines. Count what came out and compare it against what went in. + +If the measurements exonerate the backend, stop at the service boundary: name the owning team and the +contract being broken, and hand it over. A second, unrelated anomaly you trip over on the way is a +one-line note plus a task, not a second investigation. + +## Step 5 — try to break your own conclusion + +Before writing anything down as fact, check the boring explanations: a filter in your own query, a +silent truncation, an unrepresentative code path, a column that does not mean what its name says, a +partially-launched feature. Then verify the outcome and not just the mechanism: a guard that provably +runs is not evidence that its effect is correct. + +**Corroborate as widely as is reasonable.** Every source that can speak to the question is worth +reading, not merely a second one: one source agreeing with you is a hypothesis, and each further source +that could have failed differently and did not makes the claim harder to overturn. Do this even when the +first answer looked clear, because that is where a wrong inference survives. Stop when what is left +would only re-read the same plane, and record in FINDINGS.md which sources each claim rests on, so one +standing on a single source is visible as such. + +**Contradictory results are the most valuable thing you can find, so do not smooth them away.** Weigh a +result the same whether or not it helps the hypothesis you favour: one that cuts against a hypothesis +already carrying evidence matters more, not less. Record it, say plainly what contradicts what, and make +the conflict the next question rather than something to be explained past. + +The same holds when a measurement disagrees with the prior, or with what the developer told you is +working. Re-check the measurement first, since a stratum mismatch or a wrong surface identifier is +likelier than the system having changed under you. If it survives that, keep both on the record: say +which one you trust and why, tell the developer in one line, and carry on. + +Mark every claim **verified**, **inferred**, or **refuted**, and keep the refuted ones. + +## Step 6 — write FINDINGS.md + +```markdown +# : — investigation + +**Status:** investigating | root cause identified | blocked +**Incident type:** outage | degradation | data-quality defect | user-facing bug | near-miss | not-incident +**Severity:** critical | high | medium | low — anchored to the magnitude below, not chosen by feel +**Affected:** Firefox clients | editors | both — with a magnitude, not an adjective + +## Summary +Three sentences: what broke, since when, what the user-visible effect is. + +## Timeline (UTC) +| Time | Observation | Source | + +## Prior +**Derived** — what you measured, each with the query or link. +**Reported** — verbatim answers from the developer, or `asked