Skip to content

Add portable IDE field trials for Cursor VS Code and Codex - #10

Open
LeiterConsulting wants to merge 1 commit into
mainfrom
codex/portable-ide-field-tests
Open

LeiterConsulting wants to merge 1 commit into
mainfrom
codex/portable-ide-field-tests

Conversation

@LeiterConsulting

@LeiterConsulting LeiterConsulting commented Oct 15, 2026 •

Copy link
Copy Markdown
Owner

Provide a portable field-test kit so Cursor, VS Code and Codex can be evaluated on separate systems without access from the authoring machine. Each host gets an operator checklist and the same eight small cases with pinned task-plus-link prompts.

The standard-library runner creates fresh disposable Git workspaces, retains existing dirty/untracked fixtures, keeps the evaluator outside agent context, and collects operator-scored results into a scoped ZIP. An external behavioral oracle checks denied access, missing telemetry and neighboring behavior; fixture success never scores IDE adherence automatically. Preparation refuses existing destinations, target code execution requires an explicit flag, and collection rejects stale source/prompt/Git identities and missing pass evidence. Returned bundles include structured reports and only explicitly referenced operator-redacted evidence.

The branch is ready for distributed field runs before merge. The default method revision pins 19b3e1b; the kit records its own digest separately. Framework version and activation instructions remain unchanged.

Validation at 1e1054b:

  • Repository validation: 10 modules and 328 supported local links; whitespace checks passed.
  • Local suite: 132 discovered, 129 passed, three host symlink-capability skips.
  • Push CI and PR CI: all 132 tests passed without skips in every Windows/Linux Python 3.10/3.13 job.
  • Eighteen new tests cover host-profile preparation, operation without kit Git metadata, an actual failing-before-repair tenant regression, opt-in execution, preserved inputs, stale evidence, malformed declarations and scoped ZIP contents.
  • The public commit source ZIP was downloaded. All 14 kit files match canonical Git bytes. The extracted kit prepared eight cases, rejected expected initial defects, retained pending observations and produced a portable partial-results ZIP containing only its four structured reports.
  • Actual IDE interaction remains not run by these automated checks; all generated host scores start not_run.

Run guide · Operator evaluator · Pinned source ZIP

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant