Skip to content

fix(i18n): rescue unparseable translated MDX before attribute repair - #153

Open
hxy91819 wants to merge 14 commits into
openclaw:mainfrom
hxy91819:fix/i18n-mdx-syntax-repair
Open

fix(i18n): rescue unparseable translated MDX before attribute repair#153
hxy91819 wants to merge 14 commits into
openclaw:mainfrom
hxy91819:fix/i18n-mdx-syntax-repair

Conversation

@hxy91819

@hxy91819 hxy91819 commented Sep 1, 2026

Copy link
Copy Markdown
Member

Problem

Recent Translate Full runs kept failing at shard packaging even after the translation itself succeeded (e.g. run 33169524293: Docs MDX check passed (759 files) then mdx protected attribute repair failed). Root cause: the packaging-time repair chain parses every page with strict format: "mdx", while check-docs-mdx compiles pages by file extension (.md as plain Markdown). Translated pages with JSX-looking damage — fabricated <id> elements, stray closers, malformed attribute characters — therefore pass the workflow gate and only explode inside the repair chain, discarding otherwise complete (and expensive) translation shards.

Fix

Add a parser-guided syntax repair stage (.github/scripts/i18n/repair_mdx_syntax.mjs) that runs before the existing protected-attribute repair in package_artifact.py:

  • Acceptance equals the downstream chain's own tolerant parser (parseMdx), and diagnosis reuses the same tolerant masking with an offset map (parseMdxForOffsets, now exported), so valid Markdown constructs — HTML comments, prose less-than — are never diagnosed or rewritten.
  • Deterministic bounded loop (64 patches) keyed to the parser's own diagnostics: fabricated elements absent from the source are removed with prose kept (the diagnosed opener is located via a backward, comment/code-aware search — never the first same-name token); missing or stray closing tags are resynced; void elements are self-closed; unquoted attribute values are quoted (stopping before a /> delimiter); unterminated single-line comments are closed. Multi-line unterminated comments fail closed instead of guessing what stays hidden.
  • Both repair writers now validate --locale as a single safe path segment before any write (the Python-side manifest validation previously ran only after Node had already written).
  • Comment masking preserves UTF-16 code-unit length so astral characters cannot shift diagnostic offsets.
  • Artifacts record mdx_syntax_repair_outcome; failed shards keep the existing empty-artifact contract for the weekly reconciliation. Docs updated in docs/.i18n/translation-workflow.md.

Testing

  • 14-case parser-guided rescue matrix (fabricated/stray/void/attribute/comment/astral/mixed-prose scenarios) plus fail-closed cases, wiring test asserting syntax repair runs before protected-attribute repair, and locale-traversal rejection for both repair writers: python3 -m pytest .github/scripts/i18n/tests/test_i18n_scripts.py — 106 passed; 2 pre-existing failures unrelated to this change (test_translation_workflows_pin_latest_codex_and_tier_effort expects codex@0.146.0 while the workflow pins 0.146.1; a canary git-rebase test that also fails on pristine main in sandboxed environments), verified failing on unmodified HEAD.
  • Structured review (autoreview, Codex gpt-5.6-terra xhigh): clean, no accepted/actionable findings after resolving review findings (locale traversal guard, flow-level fabricated element handling, adjacent stray-closer offsets, astral-safe masking, prose-aware diagnosis, />-aware value quoting).

@hxy91819
fix(i18n): rescue unparseable translated MDX before attribute repair
4d37f02 
Full-translation shards keep failing at packaging when translated pages
contain JSX-looking damage (fabricated <id> elements, stray closers,
bad attribute characters) that the strict MDX parsers in the repair
chain reject while check-docs-mdx passes .md pages as plain Markdown.

Add a parser-guided syntax repair stage that runs before the existing
protected-attribute repair:

- acceptance equals the downstream chain's tolerant parser, so valid
  Markdown constructs (HTML comments, prose less-than) pass untouched
- fabricated elements absent from the source are removed with prose
  kept; missing or stray closing tags are resynced; void elements are
  self-closed; unquoted attribute values are quoted; unterminated
  comments are closed
- deterministic bounded loop (64 patches) keyed to the parser's own
  diagnostics; unresolvable damage still fails the shard fail-closed

Also validate --locale as a single safe path segment in both repair
writers (the Python-side manifest validation runs only after the Node
process has already written), and preserve UTF-16 length when masking
comments so astral characters cannot shift diagnostic offsets.

Artifacts record mdx_syntax_repair_outcome; failed shards keep the
existing empty-artifact contract for the weekly reconciliation.
@hxy91819
hxy91819 requested a review from a team as a code owner September 1, 2026 05:28
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 1, 2026

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

@hxy91819
STORY-01: 建立真实故障基线
26902ef 
- 冻结真实失败 fixture×2（run 27629404260 产物树：anthropic-vertex HTML comment 29:2、taxonomy Accordion/div 错配 1075:5），provenance 至翻译提交 fe5cb01
- strict-mdx-oracle.mjs（@mdx-js/mdx@3.1.1）回放可重复：worker 两次与 validator 独立重放字节级一致；historical-scan 696 页 6 失败
- experiment-schema.json 四类实验记录契约 + 无辅助基线（双 fixture final_failure exit 1）
- 计划状态：STORY-01 done（validator=pi 结论 CONTINUE，策略 SHA-256 fddfca19…）；解锁 STORY-02
- 未决留档：逐 shard 映射不可证、超时/重试阈值留给 STORY-03/03.1
@hxy91819
STORY-02: 冻结兜底模块契约
b18331f 
- contract.md：唯一 Codex action 启动/修复/checker/同会话反馈/日志/可注入参数契约；AUXILIARY_MODE 仅 none/prettier/pr153（临时副本、hash 幂等、失败回同一 action）
- fixture-map.json：2 真实 fixture 五入口映射 + 4 类 fail-closed 边界 + 3 类运行时故障
- validator=pi 逐条核实 4 条验收通过；结论 PATCH_PROMPT 处置为 STORY-03 补充要求（日志 repair_mode+顶层 error 三元组双写、checker 两条行为断言），冻结稿不回改
- 计划状态：STORY-02 done；解锁 STORY-03
@hxy91819
STORY-03: 实现本地修复闭环 + 计划 v2 迁移
d5c9a0c 
- tools/mdx-fallback-lab/：六阶段本地实验闭环（npm test 7/7；checker 行为断言；辅助臂 fail-closed；参数全注入缺省 fail-closed）
- 真实 opt-in（CODEX_HOME=/root/.codex，gpt-5.6-sol/high）：plugin-html-comment enhanced=success（2 轮/210s，strict oracle 独立复核 compile_success）——AC-03 达成；taxonomy final_failure 诚实留档（1416:339 既有二次错误 → STORY-03.1 输入）
- 参数演化 120s→300s 实测授权留档；三轮环境诊断归档；experiment.ndjson 含 repair_mode+顶层 error 三元组双写
- validator=delegate/economy 结论 CONTINUE（独立复核 npm test+oracle+ndjson+边界零越界）
- 计划：非破坏迁移 v1→v2（plan-v2/，v1 保留审计）；STORY-03 done，六项 AC passed=true
- 依赖路径调整：CODEX_HOME personal 账号不支持 sol 且配额受限 → 默认账号；worker 沙箱禁网 → 宿主原生 subagent（用户授权 full access）
@hxy91819
STORY-03 增补+STORY-03.1: 多轮接力协议（D-09）与用户决策收口
9763537 
- D-09（owner=user）：方案 A 仅增强现有 Codex action；多轮接力；300s 单轮实测；MAX_ATTEMPTS 注入默认 4；辅助不接入
- 接力实测：双 fixture enhanced=success（plugin 1 轮 70.7s；taxonomy 1 轮 166.6s 修复 1061/1075+1417 既有双错误，strict oracle 独立复核 exit 0，内容保留 3789/3790 行）
- 实验记录追加至 8 条+三轮归档链；parameter-evolution 补 300s×4 条目；npm test 9/9
- 决策包 story03.1-decision-package-2026-09-04/（validator PATCH 修正后复核通过）
- STORY-03.1 done：AC-01..05 passed；STORY-04 解锁（授权条件=D-09）
@hxy91819
STORY-04: 增强现有 Codex action 接入翻译流水线（D-09/D-10）
b2b8297 
- translate-locale-reusable.yml：单入口 ≤4 轮有界接力（Repair→Enforce scope→Recheck，三重门控），env 注入 MDX_REPAIR_MAX_ATTEMPTS=4 / HARD_TIMEOUT_MS=600000（D-10）/ AUXILIARY_MODE=none（非 none fail-closed）
- 新增 mdx_repair_relay.py decide/report：契约 §1 启动条件、预算与辅助 fail-closed、内容快照、无阈值内容丢失守卫、repair_mode/rounds/顶层 error 三元组、changed paths sha256
- package_artifact 逐页 partial-success（不可救页排除+标记）；apply_artifacts finalizer 解释 failed_paths；docs-mdx-repair.md 更新接力措辞（D-09 授权，上游同步遗留已记录）
- pytest 119 passed（11 新增+3 修改，含 2 个基线即红诚实修复）；validator=delegate CONTINUE
- STORY-04 done：AC-01..05 passed
@hxy91819
STORY-05: 独立 CI 验证子流水线
7096114 
- mdx-repair-validation.yml：workflow_dispatch+workflow_call；offline job 零 secret（工具链/全量测试/lab/oracle gate/单入口断言）；real-codex-relay job（opt-in 门控、逐字节复刻生产接力、三态分类 success/agent_failure/environment_failure、artifact 先传后置红）
- mdx_repair_validation.py：oracle-gate/single-entry/classify；preflight 复用生产分类
- 权限 contents:read、persist-credentials:false、无 push/发布
- pytest 132 passed（13 新增）；validator=delegate CONTINUE
- STORY-05 done；runner 实测留待交付推送后触发
@hxy91819
STORY-06: 阶段化发布机制——canary 开关、RELEASE gate 与回退演练
04c1913 
- translate-locale-reusable.yml：mdx_repair_enabled（默认 false=原失败路径逐步等价）/canary_locales/canary_paths/canary_gate_failure_policy；mdx-repair-gate 前置作业复用 mdx-repair-validation（real_codex, secrets inherit）；15 relay 步骤条件化；发布前 Consume gate（abort 先于发布）；发布后 r2-smoke（28273967200 R2 stale 教训）+ release summary
- mdx_repair_canary.py：decide/gate/summary/r2-smoke（fail-closed；证据缺失或不一致即拒）
- 操作文档 docs/.i18n/mdx-repair-canary-operations.md
- pytest 158 passed；validator=delegate CONTINUE（等价性独立复核）
- 真实 canary 证据待推送后演练收集（AC-01/02/04 线上项）
@hxy91819
fix: 移除 codex-args --full-auto（CLI 0.146.1 不兼容，runner 实测 agent 无法启动）
d200a4e 
- 首轮真实 canary 演练（run 33932528630）实证：--full-auto 被 0.146.1 拒绝，agent 未启动
- 移除 8 处（生产 relay 4 轮 + validation 4 轮）；沙箱语义由 sandbox: workspace-write + safety-strategy: drop-sudo 表达
- 重要发现：生产原 Repair 步同写法 + continue-on-error → 上游升 pin 后生产修复链疑似已静默失效（STORY-07 复盘输入）
- 版本探针 exports 兼容修复（@mdx-js/mdx、tsx）；测试改为负向断言 + sandbox/drop-sudo 计数
- pytest 158 passed
@hxy91819
fix(integration): setup-go 1.25→1.26 对齐源仓库 go.mod 要求
2f3b9a2 
- 源仓库 openclaw/openclaw go.mod 已要求 go>=1.26.0；dispatch 模式演练实测
  docs-i18n 因 go 1.25.14+GOTOOLCHAIN=local 失败（5 次尝试全灭）
- 预存版本漂移，与 canary 改动无关；生产 translate-full 因 workflow-ref
  artifact 自带工具链而未暴露
@hxy91819
fix(integration): translate Check 步补装 tsx@4.20.3（dispatch 模式无 workflo…
1207b87 
…w-ref 工具链）

- check-docs-mdx 经 tsx shim 执行；生产 translate-full 由 workflow-ref artifact
  携带 node_modules，dispatch 模式缺 tsx → MODULE_NOT_FOUND
- 与 mdx-repair-gate 作业的显式 pin 同模式
@hxy91819
STORY-06 done: 真实 canary 演练全绿（run 33935656061）
5a6345a 
- 分类 success（frozen_fixtures_pass_strict_recheck，600s×4/none），三 job success
- 证据 story06-live-drill-2026-09-05/：三轮演练迭代（--full-auto 发现、sha 竞态、go/tsx 工具链）+ 分类/逐轮诊断/单入口/oracle gate
- AC-01..04 passed；遗留进 STORY-07（live summary、relay 2-4 轮观察、上游 --full-auto 同步）
@hxy91819
STORY-07: 黄金验收与线上复盘（final_story，Epic 收口）
f7ab89b 
- GC-01..06 逐条结论：01/02/04/05 pass；03 pass（补字面整篇删除断言，npm test 10/10）；06 pass-with-conditions（持续观察窗按 D-08 挂账至 PR 合并后首轮真实发布）
- postmortem：--full-auto 静默失效窗取证（08-05 pin 升级→08-27 修复，22 天；生产原 Repair 步同写法+continue-on-error）；辅助均不纳入（D-09）；上游反哺清单
- 验收 commit=5a6345abb8，全部离线判据 validator 独立复跑通过（oracle sha256 与 runner 修复快照逐字节对齐）
- STORY-07 done；8/8 stories、6/6 golden cases
@hxy91819

hxy91819 commented Sep 5, 2026 •
edited
Loading

Copy link
Copy Markdown
Member Author

Update: MDX repair fallback chain (Epic I18N-MDX-CODEX-FALLBACK) delivered on this branch

This branch gained 11 commits since 4d37f029f0 (HEAD a1607335e9), building on the original PR #153 parser-guided repair to deliver the full translation MDX repair fallback chain, verified on production openclaw/docs runners.

TL;DR

The single codex-action entry is now an enhanced bounded relay (≤4 rounds, 600s/round, env-tunable); a read-only CI validation workflow and a canary/rollback mechanism were added (both default-off = byte-equivalent to current behavior); the real-failure fixture repairs pass locally AND on production runners. Bonus finding: the production repair chain had likely been silently failing for 22 days (--full-auto incompatible with CLI 0.146.1) — fixed here and flagged for upstream.

What was added (in commit order)

  1. Real failure baseline: 2 frozen real-failure fixtures (HTML comment / Accordion mismatch, provenance to the translation commit) + strict @mdx-js/mdx@3.1.1 oracle + 696-page scan (6 failures)
  2. Frozen repair contract: single-entry six stages (full-page diagnosis → sole action → lightweight checker → bounded same-session feedback → strict re-verify)
  3. Local experiment harness tools/mdx-fallback-lab/: multi-round relay protocol (each round fixes fed-back diagnostics until strict compile passes or budget cap); real-codex repairs succeeded on both fixtures (npm test 10/10, incl. whole-file-deletion interception)
  4. Production integration: translate-locale-reusable.yml relay chain (HARD_TIMEOUT_MS=600000/round, MAX_ATTEMPTS=4, AUXILIARY_MODE=none — all env-injected, non-numeric fails closed); per-page partial-success in package_artifact/apply_artifacts (failed pages no longer drop the whole shard)
  5. Standalone CI validation workflow mdx-repair-validation.yml: zero-secret offline job (full test suite + oracle gate + single-entry assertions) + real_codex opt-in job with three-state classification (success/agent_failure/environment_failure); top-level permissions: contents: read, no push/publish
  6. Canary/rollback: mdx_repair_enabled (default false = expression-evaluator-verified equivalence to the pre-canary failure path), RELEASE gate (only success proceeds; abort fires before any publish action), R2 smoke (lesson from #28273967200 R2 staleness), release summary + evidence artifacts; operations manual at docs/.i18n/mdx-repair-canary-operations.md
  7. Golden acceptance GC-01..06 + postmortem (see Evidence)

Key findings (postmortem highlights)

  • Production repair chain likely silently broken for 22 days: CLI 0.146.1 rejects --full-auto (proven by run 33932528630 — agent never started), while the original Repair step carried --full-auto + continue-on-error: true; window opened by upstream chore(deps): refresh docs tooling #123's pin bump (Aug 5). This branch removes the flag (sandbox semantics already expressed by sandbox: workspace-write + safety-strategy: drop-sudo) — recommend upstream audit for the same pattern.
  • Source repo go.mod now requires go≥1.26 (workflow setup-go pinned 1.25 — aligned); dispatch mode needs explicit tsx@4.20.3 (added).
  • Live runner validation: run 33935656061 — three jobs successful, classification=success (frozen_fixtures_pass_strict_recheck, 600000ms×4/none). Earlier failing runs (agent_failure / toolchain gaps) were classified honestly and archived — no masked failures.

Notes for reviewers / maintainer

  • Branch is 26+ commits behind upstream/main: rebase + re-verify before merge. Expected conflict surfaces: translate-locale-reusable.yml (relay rounds vs upstream changes) and .openclaw-sync/docs-mdx-repair.md (relay wording).
  • .openclaw-sync/docs-mdx-repair.md is a source-sync mirror: the relay wording change must be fed back to openclaw/openclaw, or the next sync will overwrite it.
  • Post-merge observation window (plan D-08): first commit_locale=true real publish to collect canary-release-summary + verified R2 smoke; first field evidence of relay rounds 2-4 / budget exhaustion.
  • Canary enable/observe/rollback manual: docs/.i18n/mdx-repair-canary-operations.md (everything default-off; no behavior change unless explicitly enabled).

Evidence

Planning/evidence artifacts live in a local orchestration workspace (untracked as of a1607335e9); key data is inlined above. Live runs: 33932528630 (agent_failure, honestly classified) → 33935656061 (success). Full reports (golden acceptance, postmortem, contract, experiment records) available on request.

@hxy91819
chore: untrack local large-task planning workspace
a160733 
- plans/ (epic/story plan, evidence, orchestration state) is a local
  working artifact, not part of the published docs mirror
- removed from tracking; ignored locally via .git/info/exclude
- files remain on disk for continued local development
@hxy91819

hxy91819 commented Sep 5, 2026

Copy link
Copy Markdown
Member Author

Handoff: landing this PR, remaining work, and watch items

Handoff notes for whoever lands this PR (or the next working session). The full golden acceptance lives in the comment above; this comment covers how to land, what remains, and what to watch.

Landing procedure

  1. Rebase onto openclaw/docs:main (branch is 26+ commits behind). Expected conflict surfaces:
    • translate-locale-reusable.yml — relay rounds (4×) + canary/gate steps vs upstream changes. Do not reintroduce --full-auto (see cautions).
    • .openclaw-sync/docs-mdx-repair.md — relay wording (semantic merge; must stay in sync with openclaw/openclaw afterwards).
    • .github/scripts/i18n/tests/test_i18n_scripts.py — 39 tests were added/modified here.
  2. Local gates (all must pass): 
    python -m pytest .github/scripts/i18n/tests/ -q        # 158 passed
cd tools/mdx-fallback-lab && npm test                   # 10/10
node plans/i18n-codex-mdx-fallback/agent/evidence/story01-real-fixtures-2026-09-01/strict-mdx-oracle.mjs <fixture1> <fixture2>   # exit 1 = expected
python3 .github/scripts/i18n/mdx_repair_validation.py single-entry --help
    Full command list: plans/i18n-codex-mdx-fallback/agent/evidence/story07-golden-acceptance-2026-09-05/verification-commands.md (local workspace; see evidence note below).
  3. Push both remotes (fork = PR head; openclaw/docs = where Actions run with real secrets), then trigger verification:
    • gh workflow run -R openclaw/docs translate-locale-reusable.yml --ref fix/i18n-mdx-syntax-repair with the required inputs: locale=zh-CN locale_slug=zh-cn mode=incremental shard_index=0 shard_total=1 worker_parallel=1 thinking_effort=high pending_limit=1 publish_ref=<pinned main commit sha>, canary: mdx_repair_enabled=true canary_locales=zh-CN canary_paths=channels/line.md canary_source_path=channels/line.md canary_gate_failure_policy=fallback. First drill: keep commit_locale=false (read-only).
    • Standalone mdx-repair-validation.yml is dispatchable once it exists on the default branch; until then its content is exercised via the gate job's workflow_call above.
  4. Merge is the maintainer's call. Post-merge, start the observation window (plan D-08).

Source-sha drift gotcha (cost us 2 failed drills)

The workflow validates source_sha against .openclaw-sync/source.json inside the publish_ref checkout — and main is a moving target (auto-sync lands frequently). Pin publish_ref to a specific main commit sha and read source.json from that same sha before dispatch. Chasing main by name fails with "publish ref source … does not match requested …".

Post-merge observation checklist (plan D-08)

  • First real publish with commit_locale=true: collect canary-release-summary artifact + verified R2 smoke (the only open GC-06 live item)
  • First field evidence of relay rounds 2-4 / budget exhaustion (currently covered by offline mocks only)
  • Observation-window data (repair rate / duration / checker false positives) written back to the plan and docs/.i18n/mdx-repair-canary-operations.md
  • Feed back to openclaw/openclaw: ① relay wording of .openclaw-sync/docs-mdx-repair.md (or the next source sync overwrites it) ② the --full-auto removal ③ audit the Aug 5 → Aug 27 silent-failure window against upstream logs

Cautions

  • Never reintroduce --full-auto: CLI 0.146.1 rejects it (run 33932528630 — agent never started). Sandbox semantics are already expressed by sandbox: workspace-write + safety-strategy: drop-sudo. The original Repair step carried this flag with continue-on-error: true, which is why the production repair chain likely failed silently for ~22 days (Aug 5 → Aug 27).
  • Dispatch-mode toolchain trio (no workflow-ref artifact in dispatch): go 1.26 (setup-go), @mdx-js/mdx@3.1.1, tsx@4.20.3. Don't drop these installs.
  • Everything canary is default-off and byte-equivalent to the pre-canary failure path (equivalence verified by an expression evaluator over real workflow conditions, independently re-checked).
  • docs/** generated pages and locale files must never be hand-edited.
  • Evidence artifacts (golden acceptance, contract, experiment records) live in a local orchestration workspace, untracked since a1607335e9 — available on request; key data is inlined in the comment above.

Evidence index

  • Golden acceptance + postmortem: plans/i18n-codex-mdx-fallback/agent/evidence/story07-golden-acceptance-2026-09-05/ (local workspace)
  • Live runs: 33932528630 (failure, honestly classified) → 33935656061 (success)
  • Decisions D-01..D-10 and story records: plans/i18n-codex-mdx-fallback/plan-v2/agent/ (local workspace)
  • Delivery HEAD at time of writing: f7ab89bd58 → a1607335e9 (plans untracked)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@hxy91819