Single-cell cell-type annotation case and benchmark scenario 3

CHANGELOG.md

@@ -2,6 +2,16 @@
 
 ## Unreleased
 
+- Add the single-cell cell-type annotation case (`examples/singlecell_celltype`): marker tables derived
+  from six CELLxGENE datasets annotated by their authors, with the Cell Ontology, HGNC and ASCT+B pinned.
+  Benchmark scenario 3 runs six models on it, alone and in the feedback loop. In the pilot, 35 of 138 first
+  answers gave a Cell Ontology ID that belongs to another term; none was admitted. Wrong or invalid
+  annotations fell from 43% of answers to 18% of those admitted behind the gate, at the cost of half
+  the annotations going to a person. The ASCT+B cross-check proved to be noise as a conflict source.
+  Protocol 2 (contradicting markers only for mixed clusters, no ASCT+B), frozen before the 46 held-out
+  clusters were run: identifier errors in 62 of 276 answers, none admitted; in the feedback loop,
+  answers compatible with the authors' term rose from 176 to 205, wrong or invalid fell from 36% of
+  answers to 20% of those admitted, and 8% of annotations went to a person.
 - Add generic grounders that check what a record cites against pinned reference releases, with
   messages that say what to fix: `OntologyGrounder` (ontology terms: existence, obsoletion, label,
   kind), `GeneGrounder` (HGNC symbols, previous symbols and aliases), `VariantGrounder` (HGVS form,

evaluation/llm_benchmark/README.md

@@ -235,6 +235,116 @@ What this shows:
 - One run, 18 tasks and a changed prompt: model-level numbers move between runs (GPT-5.6-Luna's first
   submissions went from 0 to 3 invalid citations), so read these as directions, not rates.
 
+## Scenario 3: single-cell cell-type annotation
+
+The same loop outside the literature: each model annotates a cluster of a real single-cell dataset from
+its top 20 marker genes, giving a Cell Ontology term and the markers that support it or argue against it
+([case](../../examples/singlecell_celltype/README.md), [celltype_loop.py](celltype_loop.py),
+[results](results/celltype-pilot/summary.md)). The reference is each study's own author annotation.
+Answers are compared with it through the ontology: exact, coarser (an ancestor), finer (a descendant) or
+wrong; an identifier that is not a current cell-type term is invalid.
+
+Bioevidence checks each answer with generic grounders:
+- each cited marker must be one of the cluster's markers in the pinned table;
+- the Cell Ontology term must exist, be current, be a cell type and match its label;
+- gene symbols must be approved HGNC symbols;
+- ASCT+B markers are cross-checked against the Cell Ontology's disjoint lineages.
+
+Fixable findings go back to the model, at most three answers. A record with the model's own contradicting
+markers (BEV004) or an ASCT+B conflict (BEV025) goes to a person. Pilot, 24 clusters from six datasets, six
+models:
+
+| Configuration | Annotated | Exact | Coarser | Finer | Wrong or invalid | Answers with an identifier or marker error | Routed to a person |
+|---|---:|---:|---:|---:|---:|---:|---:|
+| Model alone | 138/144 | 41 | 21 | 17 | 59 | 38/138 | 0 |
+| Model + bioevidence gate | 56/144 | 32 | 13 | 1 | 10 | 0/56 | 82 |
+| Model + bioevidence feedback loop | 67/144 | 35 | 13 | 4 | 15 | 0/67 | 71 |
+
+What this shows:
+
+- **Identifiers.** Models write Cell Ontology identifiers that belong to other terms. In 35 of 138 first
+  answers the ID and the label disagreed, e.g.:
+  - "Paneth cell" under CL:0000147 (pigment cell);
+  - "hepatic sinusoidal endothelial cell" under CL:0000639 (a basophil of the pituitary);
+  - "intestinal crypt stem cell" under CL:0002089 (a mouse ILC term).
+
+  By model: Claude Haiku 4.5 and GPT-5.6-Luna each 12 of 23 answers with an identifier or marker error,
+  Gemini 3.8 Flash 7, Gemini 3.1 Pro 4, GPT-6-Astra 2, Claude Opus 5.5 1. A knowledge base stores the ID,
+  so these errors would enter it silently.
+
+  None was admitted. The fix does not depend on the model: the finding names the term the label belongs
+  to. In the loop 11 were fixed and admitted; 26 went to a person, most because the revised answer still
+  carried its own contradicting markers.
+- **Cell types.** Bioevidence cannot tell whether a cluster is an ILC2 or a Th2 cell. What its routing did
+  was concentrate wrong answers: 59 of 138 first answers were wrong or invalid (43%), against 10 of 56
+  admitted behind the gate (18%) and 15 of 67 in the loop (22%).
+- **Model-declared conflict is the strongest signal.** Answers in which the model cited a marker against
+  its own answer were wrong 63% of the time (39 of 62), against 26% (20 of 76) without. One example:
+  - Claude Haiku 4.5 called a pericyte cluster smooth muscle;
+  - it added that CCL2 and IGFBP7 suggest pericytes;
+  - that record went to a person.
+- **The ASCT+B cross-check was mostly noise.** It fired on 23 first answers, 13 of them wrong, but for
+  reasons that do not hold. For example, CD44 cited for pancreatic ductal cells was flagged because
+  ASCT+B happens to list CD44 only for a lung T cell. FCER1G for NK cells was flagged because it is listed
+  for basophils and neutrophils. ASCT+B biomarker lists say where a gene is a marker, not that it is
+  specific. Disjointness removed the within-lineage alarms, but a partial reference still cannot support
+  "only". This protocol should not use it as a conflict source; the generic grounder stays, for
+  references that state specificity or exclusion.
+- **The cost is the review load:** half of the annotations went to a person. Models cited markers against
+  their own answer in 64 of 144 first answers, because the prompt asked for "any that argue against it".
+  That is honest, and it carries the signal above, but it is more than a curation team can review.
+
+### Held-out test split (protocol 2)
+
+Two lessons from the pilot became protocol 2, committed before the 46 held-out clusters were run:
+
+- contradicting markers are asked for only when some clearly point to another cell type (a mixed cluster,
+  doublets);
+- the ASCT+B cross-check is left out.
+
+Nothing was changed after the results were seen ([results](results/celltype-test/summary.md)). Six models,
+46 clusters:
+
+| Configuration | Annotated | Exact | Coarser | Finer | Wrong or invalid | Answers with an identifier or marker error | Routed to a person |
+|---|---:|---:|---:|---:|---:|---:|---:|
+| Model alone | 276/276 | 117 | 33 | 26 | 100 | 63/276 | 0 |
+| Model + bioevidence gate | 198/276 | 113 | 27 | 21 | 37 | 0/198 | 78 |
+| Model + bioevidence feedback loop | 255/276 | 138 | 31 | 36 | 50 | 0/255 | 21 |
+
+What this shows:
+
+- **The loop makes the annotations better, not only safer.**
+  - Answers compatible with the authors' term (exact, coarser or finer) rose from 176 to 205.
+  - Exact matches rose from 117 to 138.
+  - Wrong or invalid fell from 100 (36% of answers) to 50 admitted (20% of what was admitted).
+  - Only 21 of 276 (8%) went to a person, against 71 of 144 (49%) in the pilot.
+- **Most of the gain is identifiers.**
+  - 62 first answers gave a Cell Ontology ID that does not match its label, or is not a current cell type.
+  - By model: Claude Haiku 4.5 had 23 of 46 answers with an identifier or marker error, GPT-5.6-Luna 16,
+    Gemini 3.1 Pro 10, Gemini 3.8 Flash 9, Claude Opus 5.5 4, GPT-6-Astra 1.
+  - The finding names the term the label belongs to, so the model can correct the ID to what it meant.
+  - Outcome: 56 of the 62 were fixed and admitted (25 exact, 14 finer, 4 coarser, 13 wrong); 3 went to a
+    person and 3 were still rejected after three answers.
+  - None of these identifier errors depends on which model made it, and none was admitted.
+- **The model's own conflict is now a sharp signal.**
+  - Only 17 first answers cited markers against their own answer (protocol 1: 64 of 144).
+  - 14 of those 17 were wrong (82%), against 33% of the rest.
+  - In the loop, 13 of the 18 records sent to an expert were wrong.
+- **What remains is semantic, and much of it is the reference.** 50 admitted answers disagree with the
+  authors' term with valid identifiers and real markers.
+  - 7 called the duodenal cluster the authors label "B cell" a plasma cell. Its top markers are JCHAIN and
+    MZB1, so the models are probably right, but in the Cell Ontology a plasma cell is not a B cell.
+  - 11 called a "mesenchymal cell" a fibroblast or pancreatic stellate cell.
+  - Others are real fine-grained confusions: NK T cells called gamma-delta T cells (5), memory CD4 T cells
+    called naive (5), an ILC3 called a Th17 cell.
+
+  Bioevidence cannot see these; expert labels and an audit of admitted records (#23, #24) can.
+
+Limits:
+- Author labels are the reference. Some are coarse ("stem cell", "precursor cell") and some debatable, so
+  "wrong" is partly label noise; exact plus coarser plus finer is the fairer measure of agreement.
+- One run per split. The test split has 46 clusters from the same six datasets as the pilot.
+
 ## Semantic checks (#31)
 
 Grounding cannot tell whether a verbatim quote supports the claim, so two semantic layers were

evaluation/llm_benchmark/agent_loop.py

@@ -94,26 +94,26 @@ def wsl_path(path: Path) -> str:
     return f"/mnt/{text[0].lower()}{text[2:]}" if re.match(r"^[A-Za-z]:/", text) else text
 
 
-def call_model(key: str, prompt: str, timeout: int = 600) -> tuple[dict, list[str]]:
-    """One fresh CLI call answering STEP_SCHEMA, through the WSL installs of Claude Code, Codex and Antigravity."""
+def call_model(key: str, prompt: str, timeout: int = 600, schema: dict | None = None) -> tuple[dict, list[str]]:
+    """One fresh CLI call answering `schema` (default STEP_SCHEMA), through the WSL installs of Claude Code, Codex and Antigravity."""
     spec = run_models.MODELS[key]
     with tempfile.TemporaryDirectory(prefix="agent-", dir=Path("C:/t/agent-work")) as directory:
         work = Path(directory)
-        (work / "schema.json").write_text(json.dumps(STEP_SCHEMA), encoding="utf-8")
+        (work / "schema.json").write_text(json.dumps(schema or STEP_SCHEMA), encoding="utf-8")
         (work / "prompt.txt").write_text(prompt, encoding="utf-8")
-        schema, folder = wsl_path(work / "schema.json"), wsl_path(work)
+        schema_file, folder = wsl_path(work / "schema.json"), wsl_path(work)
         if spec["cli"] == "claude":  # the WSL install; the Windows CLI's login can expire mid-run
             command = ("PATH=$(ls -d ~/.nvm/versions/node/*/bin | tail -1):$PATH claude -p --model "
                        f"{spec['model']} --tools '' --strict-mcp-config --no-session-persistence --output-format json "
-                       f"--json-schema \"$(cat {schema})\" < prompt.txt")
+                       f"--json-schema \"$(cat {schema_file})\" < prompt.txt")
             parse = run_models.claude_parse
         elif spec["cli"] == "codex":
             disable = " ".join(f"--disable {f}" for f in run_models.CODEX_DISABLED)
             command = (f"codex exec -m {spec['model']} --ignore-user-config {disable} --sandbox read-only "
-                       f"--skip-git-repo-check --ephemeral --json --output-schema {schema} -o last.json - < prompt.txt")
+                       f"--skip-git-repo-check --ephemeral --json --output-schema {schema_file} -o last.json - < prompt.txt")
             parse = run_models.codex_parse
         else:
-            command = (f"agy --model {spec['model']} --output-format stream-json --json-schema {schema} --sandbox "
+            command = (f"agy --model {spec['model']} --output-format stream-json --json-schema {schema_file} --sandbox "
                        f"--print-timeout {timeout}s --print \"$(cat prompt.txt)\"")
             parse = run_models.gemini_parse
         done = subprocess.run(["wsl.exe", "-e", "bash", "-lc", f"export PATH=$HOME/.local/bin:$PATH; cd {folder} && {command}"],