Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
87 changes: 87 additions & 0 deletions devlog/_plan/260905_windows_suite_stabilization/000_plan.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,87 @@
# 000 — Plan: stabilize the Windows suite

Unit: get the Windows test suite to zero failures on the runtime this repository
pins, and keep it there. Base `dev` at `00834d710`, 2026-09-05.

Runner: the user's own Windows box `desktop-c795oh4` (Windows 10.0.26200.9168,
16 cores, Git-bash), checkout at `C:\ocxwin\repo`, reached over SSH. Single
machine, so suite runs are **strictly serial** under `/c/ocxwin/.suite.lock` and
never overlapped.

**Always pin the runtime explicitly:**

```bash
cd /c/ocxwin/repo && B=./node_modules/bun/bin/bun.exe && "$B" --version # 1.4.0
```

A bare `bun` on that box is 1.3.14 and produces a fictional failure list. That
mistake was made once, cost ~70 minutes, and is recorded in `001`.

## Baseline

| shard | pass | skip | fail | wall | note |
|---|---|---|---|---|---|
| 1/4 | 4459 | 39 | 2 | 971s | |
| 2/4 | 4606 | 16 | 22 | 1147s | **contaminated** — 22 → 0 on a clean tree, see `007` |
| 3/4 | 4305 | 11 | 1 | 1274s | |
| 4/4 | 4413 | 12 | 0 | 888s | |

**Three real failures, two defects**, both in test-harness code. No product
defect identified.

Shard 2's 22 were contamination I created: a `kill -9` on the wedged 1.3.14
shard left a Windows handle on `tests/.tmp-oauth-store-multi-test`, so every
later teardown in that fixture hit EPERM. Clean, the file is 22 pass in 1.4s.
`007_acl_defect_retracted.md` has the falsification probe and the diagnosis it
destroyed. That count was measured after the kill, so the confirmation run
re-measures it.

**Before any measurement a conclusion depends on**, clear what a killed run
leaves behind:

```bash
cd /c/ocxwin/repo && ls -d tests/.tmp-* 2>/dev/null; ps | grep bun
```

## Work phases

Two, **independent** — disjoint write sets, no shared API.

| phase | doc | defect | failures | write set |
|---|---|---|---|---|
| wp-argv | `020_defect_launcher_argv.md` | a test reads the `cmd.exe` launcher's argument grammar as its mock API | 2 | `tests/multi-agent-keep-native-v1.test.ts` |
| wp-cwd | `030_defect_unlinked_cwd.md` | the test needs a POSIX unlinked cwd, which Windows cannot produce | 1 | `tests/update-notify.test.ts` |

`010_defect_acl_seam.md` and `040_acl_stub_hygiene.md` are **RETRACTED** (`007`).
Between them they would have added a test helper and rewritten 18 test files to
prevent a defect that does not exist.

## Research

`001`-`006` are analysis and are not implemented from:

| doc | what it is |
|---|---|
| `001_runtime_fault.md` | the 1.3.14-vs-1.4.0 A/B, and the method correction |
| `002_v140_baseline.md` | the corrected baseline and root-cause roll-up |
| `003_void_preload_analysis.md` | VOID — a 1.3.14-only mechanism; records a latent hazard at `tests/preload.ts:41` |
| `004_void_singles_analysis.md` | VOID — four of six "singles" do not exist on 1.4.0 |
| `005_wedge_resolution.md` | RESOLVED — the shard-3 wedge was the runtime; no code target |
| `006_void_inventory_1314.md` | VOID — the first inventory, kept as the record of the mistake |

## Acceptance for the unit

1. Four shards, pinned runtime, **0 fail, twice consecutively**, with logs.
2. Every fix is a root-cause change: no assertion weakened, no timeout inflated
without naming the intrinsic operation it covers.
3. macOS unchanged for every touched file, verified by running it.
4. `bun run typecheck` clean.
5. Published as pull requests against `dev`, each filling the template.
6. Any Windows landmine not already in the `fuck-powershell` corpus is added
there and passes `lint-cases` + `validate-graph`.

## Out of scope

Product changes (none are indicated), release promotion, npm publish, and the
repository-wide local suite on macOS — the user prohibited the last one; focused
files and `typecheck` only.
Original file line number Diff line number Diff line change
@@ -0,0 +1,88 @@
# 001 — The first baseline used the wrong Bun. Everything it concluded is void.

Research doc. Written after the plan audit at `A` returned FAIL and its first
blocker turned out to be correct.

## What happened

The 2026-09-05 baseline in `000` was run with the Bun on the Windows box's PATH,
`~/.bun/bin/bun` = **1.3.14**. The repository pins **1.4.0**
(`package.json:68`, `dependencies.bun`), and `.github/actions/setup-project-bun`
installs exactly that version, keeping "the runtime SOT in one place". The
checkout already carried it at `node_modules/bun/bin/bun.exe`.

So the baseline measured a runtime that neither CI nor a correct local run uses.

## The controlled comparison

Same box, same checkout, same two files, same flags — only the binary differs:

```
$ ./node_modules/bun/bin/bun.exe test --isolate --timeout 60000 \
tests/abort-idle-deadline.test.ts tests/codex-reset-credit-operation-ledger.test.ts
50 pass · 0 fail · 211 expect() calls · [8.63s]

$ ~/.bun/bin/bun test --isolate --timeout 60000 \
tests/abort-idle-deadline.test.ts tests/codex-reset-credit-operation-ledger.test.ts
6 pass · 44 fail · 203 expect() calls · [8.75s]
```

The 44-failure guard defect exists only on 1.3.14.

The wedge behaves the same way:

```
$ bun 1.3.14 test --isolate tests/client-hub-relay.test.ts tests/cline-pass-reasoning-efforts.test.ts
→ killed at the 240s deadline; the second file never printed a line (exit 124)

$ bun 1.4.0 test --isolate (identical command)
12 pass · 0 fail · [1.60s]
```

## What this invalidates

- `000` — every shard count. The four-shard baseline must be re-measured.
- `010` — the preload run-id provenance analysis. The mechanism it describes is
real in the source (the auditor verified (a)-(e) line by line, correcting one
citation: the non-win32 early return is `scripts/test-run-lock.ts:164`, not
`:162`). What is NOT established is that this mechanism fires on the runtime
the project actually uses. On 1.4.0 the guard arms and the same files pass.
- `020` — S1 and S6 were argued as ambient-`CODEX_HOME` defects. The auditor
showed the preload assigns a per-file `CODEX_HOME` when it completes, so both
may simply be downstream of the guard fault and disappear with the runtime.
- `030` — the wedge, and with it the attribution to
`tests/cline-pass-reasoning-efforts.test.ts`.

## What survives

The ordered-pair experiment the auditor asked for was run, and it settles the
wedge boundary that adjacency alone could not:

| run | 1.3.14 | 1.4.0 |
|---|---|---|
| `cline-pass` alone | 6 pass, 0.95s | — |
| `client-hub-relay` → `cline-pass` | **wedged, exit 124** | 12 pass, 1.6s |
| `cline-pass` → `client-hub-relay` | 12 pass | — |
| pair without `--isolate` | 12 pass | — |

Order-dependent, `--isolate`-dependent, and runtime-dependent. That is an isolate
realm-transition fault in 1.3.14, not a defect in either test file — which is why
no code change was made against it.

## Method correction

Pin the runtime explicitly in every command against the box:

```bash
cd /c/ocxwin/repo && B=./node_modules/bun/bin/bun.exe && "$B" --version
```

A bare `bun` on that machine is 1.3.14 and must not be used for any measurement
that a conclusion depends on. `.github/workflows/ci.yml` never had this problem:
it calls `./.github/actions/setup-project-bun` before every test step.

## Cost of the mistake

Roughly 70 minutes of shard time and four documents' worth of analysis, caught by
the `A` gate before a single line of product code was changed. That is the gate
working. The re-measured baseline replaces `000` in `002`.
112 changes: 112 additions & 0 deletions devlog/_plan/260905_windows_suite_stabilization/002_v140_baseline.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,112 @@
# 002 — Corrected baseline on the pinned runtime (`bun 1.4.0`)

Same box, same checkout, same serial lock. The only change from `000` is the
binary: `./node_modules/bun/bin/bun.exe` (1.4.0, the version `package.json:68`
pins) instead of the 1.3.14 on `PATH`.

## Shard 1/4

```
4459 pass · 39 skip · 2 fail · 128135 expect() calls · [971.45s]
```

**52 → 2.** The 50 that disappeared were the 1.3.14 isolate fault (`001`), not
defects in this repository. Both survivors are in one file:

```
(fail) ocx v2 keep-native-v1 > enabling the native-v1 pin disables the global V2 override before catalog sync
(fail) ocx v2 keep-native-v1 > mode v2 honors a pre-existing native-v1 pin instead of enabling the global override
```

Shards 2-4 are running and land in the table below as they finish.

| shard | pass | skip | fail | wall |
|---|---|---|---|---|
| 1/4 | 4459 | 39 | **2** | 971s |
| 2/4 | 4606 | 16 | **22** | 1147s |
| 3/4 | 4305 | 11 | **1** | 1274s — **past the 1.3.14 wedge** |
| 4/4 | 4413 | 12 | **0** | 888s |
| **total** | **17783** | **78** | **25** | 4280s |

## What the corrected baseline says

| | first attempt (1.3.14) | corrected (1.4.0) |
|---|---|---|
| shard 1 | 52 | 2 |
| shard 2 | 122 | 22 |
| shard 3 | no verdict (wedged) | 1 |
| shard 4 | 5 | 0 |
| **defects** | unknowable | **3** |

25 failures, three root causes, and one of them is a single file. Shard 4 —
which `000` reported as five failures including two that looked like a
containment breach at `tests/service.test.ts:1283` — is **completely green**.
That alleged breach was the 1.3.14 guard fault, not a real hole in the armed-test
refusal.

### The three defects

| # | failures | file | mechanism | doc section |
|---|---|---|---|---|
| 1 | 2 | `tests/multi-agent-keep-native-v1.test.ts` | `.cmd` shim argv used as a mock API | "The one real defect so far" |
| 2 | 22 | `tests/oauth-store-multi.test.ts` | async `icacls` seam left unstubbed (+8 exposed siblings) | "The second real defect" |
| 3 | 1 | `tests/update-notify.test.ts` | POSIX unlinked-cwd is unreachable on Windows | "The third real defect" |

All three are **test-harness defects**. **No product defect was identified**, and
that phrasing is deliberate: `src/lib/win-exec.ts` is verifiably correct and is
what defect 1 trips over, `src/lib/windows-secret-acl.ts` offers the async seam
defect 2 forgot to use, and defect 3 asks the filesystem for something Windows
does not provide. What the evidence supports is "no product defect identified;
the failures point at harness teardown" — not the stronger claim that none can
exist. `010` carries the red/green A/B that would upgrade or refute that for
defect 2.

So the unit is "three fixtures encode POSIX assumptions", not "Windows is
broken".

### Sequencing: the three phases are INDEPENDENT

An earlier draft called this a dependency chain (2 → 1 → 3). It is not, and
describing risk ordering as dependency was wrong: defect 1 and 3 consume nothing
from the ACL helper, and defect 2 touches neither `src/cli/v2.ts` nor
`tests/update-notify.test.ts`. Disjoint write sets, no shared API.

They may be built and reviewed in parallel. If they are published as a stack it
is for review convenience only, and the order is then by size — `010` (22
failures), `020` (2), `030` (1) — which is a presentation choice, not a
constraint.

Shard 2's 22 are one file, `tests/oauth-store-multi.test.ts`, and one mechanism.
The 122 failures `000` recorded for this shard are gone: the 68 recovery and 49
fabric guard failures do not exist on the pinned runtime.

### Where the implementation plans live

This document is research: baseline and root-cause analysis only. One diff-level
document per surviving phase, each independently landable:

| doc | defect | failures | files touched |
|---|---|---|---|
| `010_defect_acl_seam.md` | half-installed ACL stub seam | 22 | `tests/helpers/windows-secret-acl-stubs.ts` (new), `tests/oauth-store-multi.test.ts` |
| `020_defect_launcher_argv.md` | launcher argv used as a mock API | 2 | `tests/multi-agent-keep-native-v1.test.ts` |
| `030_defect_unlinked_cwd.md` | POSIX unlinked cwd unreachable | 1 | `tests/update-notify.test.ts` |

No product source file appears in that table.

## Evidence

Shard logs are retained at `.tmp/win/v140-{1,2,3,4}.log` (gitignored; 603KB,
661KB, 597KB, 646KB). `grep -c '^(fail)'` over them gives 4, 44, 2, 0 — twice
the reported per-shard counts for 1-3 because Bun prints each failure once
inline and once in the trailing summary, and 0 for shard 4 either way.

**Shard 3 clears the wedge.** `030` predicted this from the pair experiment;
the full shard confirms it in situ:

```
1227:tests\client-hub-relay.test.ts:
1235:tests\cline-pass-reasoning-efforts.test.ts:
```

Eight log lines apart. On 1.3.14 that boundary consumed 14 minutes and never
produced a second file. No code changed in between — only the runtime.
Loading
Loading