Soaked: 2026-08-22 · Re-soaked: 2026-08-23 (fix batch, then #5) ·
Package: dash_improve_my_llms-2.7.0-py3-none-any.whl
sha256:05180075dd43ddb083af555e3cc7bf5345fe84c6554e5b8f17fb9bc807ade1a0
— verified against the file BEFORE installing; only the recorded sha admits · Host:
llms-2plot-dev (fork of dash-documentation-boilerplate 1.6.7, the future
llms.2plot.dev) · Backends: Flask and FastAPI, every finding reproduced on
both.
This file gates the v2.7.0 tag push.
sha256:05180075dd43ddb083af555e3cc7bf5345fe84c6554e5b8f17fb9bc807ade1a0
All five findings are fixed and verified from the application side. Every strict xfail this soak ever raised has flipped and been converted to a positive assertion; none remain.
| # | Sev | Finding | Status |
|---|---|---|---|
| 1 | HIGH | unhashable denylist entry 500s every request | FIXED — empty denylist, nobody blocked, nothing 500s |
| 2 | HIGH | traditional per-vendor block enforced, not published | FIXED — User-agent: Googlebot / Disallow: / emitted, and the 403 still served |
| 3 | docs | GEO.md contradicted the malformed-entry behaviour | FIXED in docs — code was right, doc moved |
| 4 | docs | GEO.md contradicted the resolver failure shapes | FIXED in docs — both shapes now stated distinctly |
| 5 | MED | H1 dedup missed the crawler document | FIXED at 93a02d6 — the guard is mirrored on both lanes |
Finding #5, closed. html_generator.py now carries the identical
leading-h1 guard prerender.py has. Measured across all 11 pages on both
backends: exactly one <h1> on the browser lane AND one on the crawler
lane, zero drift. All three prose shapes pinned at the generator — prose
that opens with its own h1 (header contributes only the description), prose
that starts mid-thought, and a page with no prose at all (the header's h1 is
the only one and stays). A dedup that simply deleted the header h1 would have
passed the end-to-end test and left doc-less pages with no heading; that is
why the generator-level shapes are pinned separately.
| Flask | FastAPI | |
|---|---|---|
| Full suite | 597 passed, 1 skipped, 0 xfailed | 594 passed, 4 skipped, 0 xfailed |
audit_links: 0 broken internal links, 0 broken anchors. The 4 external
flags are inherited network peers unreachable from this sandbox
(ai-agent.buzz, piratesbargain.com); lib/network_directory.py is
untouched since the fork.
The crawler-document bytes changed and nothing else moved — the full inherited suite is green on both backends.
The 2026-08-23 interim verdict (TAG HELD on #5), kept for the record
#1, #2, #3 and #4 are all FIXED and verified. The eight strict xfails that pinned them flipped to XPASS on the rebuilt wheel — the signal fired — and every one has been converted to a positive assertion. The two SEO-batch fixes are pinned as well.
One new finding holds the tag: #5. It is not a regression — 2.6.1 carries it identically — but it is the same defect the batch's own H1 dedup set out to fix, left unfixed on the other of the package's two document paths, and that path is the one Googlebot actually receives. Per the re-soak charter, anything not green is a new entry and the tag stays held; the call on whether to ship anyway is the owner's, and #5 carries what is needed to make it.
| # | Was | Now |
|---|---|---|
| 1 | unhashable denylist entry 500s every request | fixed — empty denylist, nobody blocked, nothing 500s |
| 2 | traditional per-vendor block enforced, not published | fixed — User-agent: Googlebot / Disallow: / is emitted |
| 3 | GEO.md contradicted the malformed-entry behaviour | fixed in docs — code was right, doc moved |
| 4 | GEO.md contradicted the resolver failure shapes | fixed in docs — both shapes now stated distinctly |
| 5 | — | NEW — crawler-document H1 duplication, every page |
Re-soak totals: 585 passed / 1 skipped / 1 xfailed (Flask) · 582 / 4 / 1 (FastAPI). The single xfail is #5.
The package's own 600-test suite proves 2.7.0 against itself. This soak proves the thing that suite structurally cannot: the callable seam driven by a real control board on a real boilerplate app, across both backends, with the whole inherited site suite asserting nothing regressed.
Method: fork the boilerplate, install its pinned dash-improve-my-llms 2.6.1
from the existing fleet environment, run the inherited suite GREEN as a
baseline, then force-install the local 2.7.0 wheel over it and re-run.
Afterwards, wire the 2.7.0 surface the way a production satellite actually
would — configure_geo(deny_countries=<callable>), panel=True,
RobotsConfig(vendor_policy=<callable>) — and attack it.
| # | Item | Verdict |
|---|---|---|
| 1 | Byte-identical when unset | PASS — see below |
| 2 | Geo: every surface class, exemptions, unknown, the seam | PASS, 1 defect (#1) |
| 3 | Panel: gate, headers, anti-drift, resolution line | PASS, no defects |
| 4 | W2 vendor policy: published vs served | FAIL on one class (#2) |
| 5 | Idempotency hardening | PASS — the fix works |
| 6 | W4 / W5 / W6 | PASS, no defects |
| 7 | The site's own guards on 2.7.0 | PASS |
| Stage | Flask | FastAPI |
|---|---|---|
| Inherited suite on 2.6.1 (baseline) | 322 passed | 319 passed, 3 skipped |
| Same suite on 2.7.0, zero knobs configured | 322 passed | 319 passed, 3 skipped |
| Same suite on 2.7.0 with the seam live | 322 passed | 319 passed, 3 skipped |
The third row is the one worth having. This host calls configure_geo()
unconditionally with a callable denylist, registers the panel, and attaches a
callable vendor_policy — against an empty store. 322 inherited assertions,
including the generic-UA prerender test and the llms.txt/tier surfaces, are
unmoved. An empty denylist really is a strict no-op.
Final suite after the soak's own tests were added: 486 passed / 1 skipped /
8 xfailed (Flask), 483 passed / 4 skipped / 8 xfailed (FastAPI). The
eight xfails are #1 and #2 below, pinned strict=True so they fail loudly the
day the package fixes them and the markers have to come out.
#1 — HIGH FIXED 2026-08-23 — a deny_countries callable returning an unhashable entry 500s every request
Contradicts a documented guarantee. docs/GEO.md: "a raising callable or
a malformed entry is logged once and treated as an empty denylist
(fail-open); it can never take down the request path." It can.
dash_improve_my_llms/geo.py:221, in _deny_set():
def _deny_set() -> Tuple[str, ...]:
if _config.deny_callable is None:
return _config.deny_static
try:
raw = tuple(_config.deny_callable())
except Exception:
_warn_once("deny_countries callable raised; treating denylist as empty (fail-open)")
logger.debug("geo denylist callable failure", exc_info=True)
return ()
cached = _callable_cache.get(raw) # <-- line 221, OUTSIDE the try
...tuple(...) succeeds for a list containing a dict, so the except never
fires. _callable_cache.get(raw) then hashes the tuple, raises
TypeError: unhashable type: 'dict', and that escapes _deny_set() →
gate() → handle_bot_request() → the adapter's before_request hook.
from dash_improve_my_llms import configure_geo
# Any nested object in the returned sequence. A dict, a list, or a set.
configure_geo(deny_countries=lambda: [{"code": "RU"}])Then fetch anything at all:
GET / -> 500
GET /llms.txt -> 500
GET /robots.txt -> 500
GET /assets/main.css -> 500
GET /healthz -> 200 (exempt paths return before _deny_set)
Expected: every one of those answers exactly as if the denylist were
empty — the callable is malformed, so nobody is blocked, and one warning is
logged.
Actual: TypeError: unhashable type: 'dict' out of the middleware; the
whole site 500s for every visitor in every country, not just for the denied
one. Confirmed identically on Flask and FastAPI.
Shapes that trigger it: [{"code": "RU"}], [["RU"]], [{"RU"}], and
["RU", {"x": 1}] — note the last: a valid entry plus one nested object is
still a total outage.
The store behind this seam is JSON on a mounted disk, written by a control board. Three ordinary paths produce a nested object in that list:
- a hand-edit during an incident (
{"geo_deny": [{"code": "RU"}]}is the shape someone reaches for when they want to add a note); - a schema change in a site's own store;
- 2.8's bot × country matrix. The addendum records
deny_matrixarriving through this same seam. A newer worker writing a matrix-shaped value that an older worker'sgeo_deny()returns verbatim is exactly this bug, and it would land during a rolling deploy.
This satellite's lib/policy_store.py sanitizes before persisting, so the
site itself is not exposed — but the package's contract is that it does not
have to.
Move the cache lookup inside the try, or normalize before caching. The
minimal change:
try:
raw = tuple(_config.deny_callable())
cached = _callable_cache.get(raw)
if cached is None:
cached = _normalize_codes(raw, strict=False)
_callable_cache.clear()
_callable_cache[raw] = cached
return cached
except Exception:
_warn_once("deny_countries callable raised; treating denylist as empty (fail-open)")
logger.debug("geo denylist callable failure", exc_info=True)
return ()_normalize_codes already does str(entry), so it copes with nested objects
once it is reached; only the hash on the way to it is fatal.
tests/test_geo_guardrail.py::test_an_unhashable_entry_fails_open
(4 parametrized cases, xfail(strict=True)).
#2 — HIGH FIXED 2026-08-23 — a per-vendor block on a traditional crawler is enforced but never published
Contradicts the W2 contract. CHANGELOG [2.7.0]: "One fold
(vendors.effective_policies) drives robots.txt AND the middleware" — so
what a site says and what it does "holds by construction". For the
traditional class, under the default config, it does not.
dash_improve_my_llms/robots_generator.py:
150: blocked = [v for v in VENDORS if policies[v.key] == "block" and v.cls != "traditional"]
...
196: if config.allow_traditional:
# comment-only block: "# Googlebot, Bingbot, etc. - covered by *"
else:
# emits real Disallow groups, correctly consulting `policies`Line 150 excludes traditional vendors from the AI-blocked section. Line 196
renders the traditional class only in the allow_traditional=False
branch. So with the default allow_traditional=True — this app's config, and
the fleet's — a per-vendor override on a traditional vendor is never rendered
at all, and User-agent: * Allow: / continues to govern it.
app._robots_config = RobotsConfig(vendor_policy={"googlebot": "block"})| Check | Result |
|---|---|
effective_policies(config)["googlebot"] |
"block" |
GET / as Googlebot |
403 |
/robots.txt group for Googlebot |
none emitted |
What /robots.txt therefore tells Googlebot |
Allow: / (via *) |
Affects all four traditional vendors: googlebot, bingbot, slurp,
duckduckbot. Reproduced on Flask and FastAPI, and directly against
generate_robots_txt() with no app at all.
The coarse flag is fine, which is what isolates this:
generate_robots_txt(..., config=RobotsConfig(allow_traditional=False))
# -> emits Disallow groups for Googlebot, Bingbot, Slurp, DuckDuckBot (P4 works)
generate_robots_txt(..., config=RobotsConfig(vendor_policy={"googlebot": "block"}))
# -> emits no Googlebot group at allThe else branch even honours overrides correctly: with
allow_traditional=False, vendor_policy={"googlebot": "allow"} Googlebot
gets no Disallow while the other three do. Only the if branch ignores
policies.
Per-vendor overrides are precisely what a control board writes — this
satellite's B7 board offers exactly this control. An operator blocking a
misbehaving traditional crawler gets a host that 403s Googlebot while its own
robots.txt says Allow: /. Googlebot keeps crawling because it was invited,
collects 403s on every page, and Search Console fills with errors — with the
site's published promise insisting nothing is wrong. The failure is
SEO-severe, silent, and points the operator at the wrong layer.
It is also the one drift the panel cannot reveal: the panel reads the fold, so it agrees with the middleware and disagrees with the served bytes.
Have the allow_traditional=True branch consult policies the way the
else branch already does — emit Disallow: / for any traditional vendor
whose effective policy is block, and leave allow/meter covered by *.
Equivalently, drop the v.cls != "traditional" filter at line 150 and let
the blocked section carry them.
A robots.txt comparison that iterates User-agent: groups cannot see this
bug: the vendor has no group, so it is skipped. The check has to resolve
each vendor's verdict through the * fallback — a vendor with no group of
its own is not unregulated, it is governed by *. That is how this soak
found it, after a first version of the same test missed it.
tests/test_vendor_policy.py::test_a_traditional_vendor_block_is_published_as_well_as_enforced
(4 parametrized cases, xfail(strict=True)), with
test_the_traditional_block_really_is_enforced and
test_the_coarse_flag_path_publishes_traditional_blocks holding the two
halves of the diagnosis in place.
docs/GEO.md, "the reloadable seam":
Callable failures degrade the safe way: a raising callable or a malformed entry is logged once and treated as an empty denylist (fail-open)
The implementation treats those two cases differently, and only the first
matches the sentence. _normalize_codes(raw, strict=False) (geo.py:85) skips
the bad entry with a warn-once and keeps the valid ones:
configure_geo(deny_countries=lambda: ["RU", "XX", "nonsense"])
# GEO.md predicts: empty denylist, nobody blocked
# actual: RU is blocked; XX and "nonsense" are skipped with a warningThe code's behaviour is the better one — voiding a whole compliance denylist because one entry is stale would be a worse failure than honouring the valid part. So the doc is the half that should move, not the code. Worth fixing before the tag because it is a compliance surface: an operator reading GEO.md and finding a stale entry in the store will predict that nobody is blocked, and be wrong.
Pinned by tests/test_geo_guardrail.py::test_a_malformed_entry_does_not_void_the_whole_list
(asserting the code's behaviour, with the disagreement in the docstring).
#4 — LOW (docs) FIXED 2026-08-23 in GEO.md — a raising resolver= falls back to headers, it does not go "unknown"
docs/GEO.md, "Resolution order":
- Your
resolver(headers), if configured (exceptions → unknown, warned once).
resolve_country (geo.py:~250) sets code = None on the exception and then
falls through to the header loop — its own warning says so: "resolver raised;
falling back to header resolution". So with a broken resolver and a real
CF-IPCountry present, the country still resolves and a denied country is
still blocked.
There is also an asymmetry worth a sentence in the docs: a resolver that
raises falls back to headers, while a resolver that returns an invalid
value returns None immediately and does not fall back.
Both behaviours are defensible; neither is what the doc says. Pinned by
test_a_raising_resolver_falls_back_to_headers and
test_a_resolver_returning_garbage_does_not_fall_back.
#5 — MEDIUM FIXED 2026-08-23 at 93a02d6 — the H1 dedup missed the crawler document, so every page ships two identical <h1>s to Googlebot
Found on the re-soak, 2026-08-23. Not a regression — 2.6.1 carries it identically — but it is the same defect the batch's own fix targets, on the half that matters more for search.
The package serves two different documents, and the fix landed on one:
| Requester | Document | Built by | H1s per page |
|---|---|---|---|
| browser-like UA | app shell + universal prerender | prerender.py |
1 ✅ |
| declared crawler / any non-browser UA | static crawler document | html_generator.generate_static_page_html |
2 ❌ |
prerender.py:204 carries the fix, and its own comment states the finding:
# SEO-audit finding (2026-08-23, confirmed on every host): the header's
# h1 duplicated the doc body's own opening markdown H1 — two identical
# h1s on every prerendered page.
prose_opens_with_h1 = prose.lstrip().lower().startswith("<h1")
if prose_opens_with_h1:
header = f"<header><p>{description}</p></header>"
else:
header = f"<header><h1>{name}</h1><p>{description}</p></header>"html_generator.py:328 has no such guard:
<header>
<h1>{title}</h1> # <-- unconditional
<p>{description}</p>
</header>
...
<main>
{body_html} # <-- opens with the prose's own <h1>Fetch any page with a crawler User-Agent and count:
curl -s -A 'Googlebot/2.1' https://host/getting-started | grep -c '<h1'
# 2Measured on this app, all 11 pages, both backends, and the pair is byte-identical every time:
/ h1=2 ['Dash Improve My LLMs — …', 'Dash Improve My LLMs — …']
/getting-started h1=2 ['Getting Started', 'Getting Started']
/reference/configuration h1=2 ['Configuration', 'Configuration']
…11/11
Expected: one <h1> per document, as the prerender path now produces.
Actual: two identical ones, in <header> and again at the top of
<main>.
The prerender path serves browsers. The crawler-document path serves Googlebot, ClaudeBot, GPTBot and every AI fetcher — the audience the whole package exists for. Fixing the h1 signal for humans and leaving it broken for search engines inverts the priority the fix was written with.
It is also cheap: the same three-line guard, moved.
Apply prerender.py's test to html_generator.py:
body_opens_with_h1 = body_html.lstrip().lower().startswith("<h1")
header_h1 = "" if body_opens_with_h1 else f"<h1>{title}</h1>"tests/test_prerender_seo.py — the marker is gone and the behaviour is
asserted positively:
test_the_crawler_document_has_exactly_one_h1— end to end, every page;test_exactly_one_h1_on_both_lanes_for_every_page— the control that would have caught this originally: browser and crawler counts compared per page, in one place, so a future fix cannot cover half the surface without failing here;- three generator-level shape tests (prose with its own h1, prose with none, no prose at all);
test_both_lanes_carry_the_identical_guard— both modules must carry the guard, so the lanes cannot drift apart again.
templates/index.html shipped an <h1> in its <noscript> block, giving
the browser document a second, site-wide h1 that competed with the page's
own — a crawler runs no JS and parses noscript. Demoted to <h2> here. Same
defect class, one layer out, and worth a line in the boilerplate.
Recorded so the next session does not re-derive it.
Geo — every surface class 451s for a denied country, with
Cache-Control: no-store: /, a docs page, /assets/*, /llms.txt,
/llms-small.txt, /llms-full.txt, per-page llms.txt, /robots.txt,
/sitemap.xml, /favicon.ico, POST /_dash-update-component (the SPA
navigation route), /llms-policy, /admin/control-board, /_dash-layout,
/_dash-dependencies, and unknown paths. Humans and bots alike.
No bypass found. Exempt paths match exactly and case-sensitively:
/healthz passes; /healthz/, /healthz-evil, /healthz/anything,
/HEALTHZ, /Healthz, //healthz, /./healthz, /a/../healthz and
/healthz%20 all 451. /healthz?x=1 passes and serves only the health JSON —
the query string is not part of the matched path, which is correct and leaks
no content.
Geo resolution: all five documented edge headers resolve; CF-IPCountry
wins the priority order; matching is case-insensitive on the value;
XX, T1, empty, ZZZ, de-DE and ?? are all "unknown"; the default
allows unknown and unknown="deny" blocks it while exempt paths still answer.
The seam works. A store write is picked up by the next request in the
same process with no restart, un-toggling recovers, and configuring before
the store is populated still works — so the denylist is genuinely read per
request rather than snapshotted at configure_geo() time. A raising callable
fails open and warns once across five requests. A corrupt store fails open
rather than locking the site to its last known denylist.
Panel: unset token → 404 for everyone; wrong token → 404 whose body names
neither the panel nor the package; empty token → 404; the token is genuinely
read per request (rotation kills the old value on the next fetch, deletion
revokes); X-Robots-Tag: noindex, nofollow and Cache-Control: private, no-store on success; absent from robots.txt, sitemap.xml, the llms index and
page_registry; POST refused; a render mutates nothing. Geo runs before the
token gate, so a denied country gets 451 rather than a 404 that would leak the
ordering. The vendor table agrees with this host's served robots.txt in both
directions on the default config. "this request resolved to: DE (via cf-ipcountry)" renders correctly end to end — the per-host check GEO.md
mandates works.
Prerender idempotency (the fix): planting data-dimll-prerender inside a
comment in this repo's real templates/index.html, booting run.py in a
fresh subprocess and fetching / still produces the prerender block — the
exact trap that silently disabled email and flows. Also verified with the
marker in a <meta> tag and a <script> body. A genuine second injection is
still a no-op, so the probe was not loosened into uselessness. The plant was
removed and a test asserts the file came back clean.
W2 default posture: ClaudeBot and GPTBot 403 on pages while /llms.txt,
/llms-small.txt, /llms-full.txt and per-page llms.txt all answer 200
with non-empty bodies — the flagged behaviour change, with its docs-open half
confirmed from the app side. Policy surfaces open to everyone; assets never
vendor-gated; AI-search and traditional crawlers untouched; meter renders
Allow and behaves as allow.
W4: unset never limits; over-ceiling bots get 429 with a numeric
Retry-After and Cache-Control: no-store; the body names the conduct rule;
humans and policy routes are never limited; buckets key on the edge-header
client IP so one noisy agent cannot lock out another; 0 and None both mean
off; and a limiter monkeypatched to raise fails open for ten consecutive
corpus fetches.
W5: genuinely dark. metering_enabled() is False, a priced verdict
resolves to gated, no surface answers 402. Guarded against passing for the
wrong reason: PRICED is absent from _VERDICTS (if it were added, the
degrade at access.py:179 would become dead code and the seam would go live
silently), and turning metering on does make PRICED survive.
W6: the hub can tighten a vendor but cannot loosen one — not past a coarse
flag, not past a local per-vendor override — and can lower the rate ceiling or
impose one where none existed but cannot raise it. A bulletin that raises
changes nothing. Five address-shaped payloads (pay_to, payto, pay-to,
wallet, recipient, including one nested three levels deep) are each
refused whole, with a control proving ordinary bulletins carrying prices as
strings are still accepted.
One design shape confirmed rather than reported as drift: Omgili is one
vendor publishing two robots tokens (Omgilibot, Omgili) matched by the
single UA substring omgili. A naive one-row-per-User-agent comparison
reads that as panel/robots drift. get_bot_vendor() classifies every token
robots.txt publishes, so it is correct — but it is worth knowing before
someone "fixes" it.
Superseded by the FINAL VERDICT at the top of this file: TAG READY.
All five findings are fixed and verified. No strict xfails remain. Push
v2.7.0 at
sha256:05180075dd43ddb083af555e3cc7bf5345fe84c6554e5b8f17fb9bc807ade1a0,
publish, and this repo's floor moves to >=2.7.0 (see PHASE1-REPORT.md for
the collapse-the-capability-block step that rides it).
The original 2026-08-22 recommendation, kept for the record
- Fix #1 — it is a total-outage path that contradicts an explicit guarantee, the change is four lines, and 2.8's matrix makes it more likely to be hit, not less.
- Fix #2 — the per-vendor path is what control boards write, and the
failure is silent and SEO-severe. If it slips, it must be documented in
docs/as a known limitation ofvendor_policyon the traditional class, because the current CHANGELOG language promises the opposite. - Fix #3 and #4 in
docs/GEO.md— one sentence each, and both are on the compliance surface where operators reason from the doc rather than the code. - Re-soak: rebuild the wheel,
pip install --no-deps --force-reinstallit here, and runpytestplusDASH_BACKEND=fastapi pytest. The eight strict xfails become XPASS-failures the moment #1 and #2 are fixed, which is the signal the fixes landed; then delete the markers and keep the tests.