From 6bac6269f6ffb161cb8d8fce643c54abc5ae55c6 Mon Sep 17 00:00:00 2001 From: Chris Leiter Date: Mon, 5 Oct 2026 14:22:58 -0400 Subject: [PATCH] Record Ping Monitor trial and refine existing-project guidance --- docs/exec_ctrl/REAL_PROJECT_PING_MONITOR.md | 145 ++++++++++++++++++++ docs/exec_ctrl/V2_PUBLICATION_READINESS.md | 19 ++- docs/v2/FIELD_TRIALS.md | 21 ++- docs/v2/ROADMAP.md | 7 + docs/v2/TOOLING.md | 4 + templates/v2/FIELD_TRIAL.md | 6 +- 6 files changed, 196 insertions(+), 6 deletions(-) create mode 100644 docs/exec_ctrl/REAL_PROJECT_PING_MONITOR.md diff --git a/docs/exec_ctrl/REAL_PROJECT_PING_MONITOR.md b/docs/exec_ctrl/REAL_PROJECT_PING_MONITOR.md new file mode 100644 index 0000000..2196262 --- /dev/null +++ b/docs/exec_ctrl/REAL_PROJECT_PING_MONITOR.md @@ -0,0 +1,145 @@ +# Controlled real-project revisit: Ping Monitor + +Date: 2026-10-05. Scoped local acceptance: `complete`. +Method: merged preview 4, `bd343d405825f19c7a5b876ddea605083ecf7739`. +Target: public [Ping Monitor](https://github.com/LeiterConsulting/ping_tool_for_splunk), +released Go v5.1.0 code at `dc653fcca7208eb8158a18ed156d87b4f85b3cb5`. + +## Task, authority and selected scope + +Actual user prompt: "Merged - lets use a simple project we have already completed +most work on and try it again with exec-ctrl - a simple one without much effort +which can prove out the various aspects of exec-Ctrl and one we can use to improve +what we already have for the framework." + +This is a controlled revisit in the framework's existing authoring session, not an +independent blind task-plus-link trial. The method was already in context. Codex +local task on Windows; exact desktop/model build is unavailable. Observed tools: +PowerShell 7.6.6, Python 3.13.7 and Go 1.25.5 windows/amd64. The Go module requires +1.24.0; dependency downloads were disabled and the existing local cache sufficed. + +Select one narrow availability-reporting fix using task mode, low risk, behavior +and operations signals. Load policy/Git, quality, troubleshooting and documentation; +inspect the affected trust boundary without inventing company rules. No target +AGENTS/organization policy was supplied or found in the inspected target paths. +Use the user's task and host instructions. Team/enterprise connectors and release +governance are outside this local fix; there is no deployment or external message. + +The original checkout has tracked endpoint-reload edits and untracked artifacts. +Use a separate worktree at the committed baseline; do not copy or test those +unfinished changes. Preserve original branch/status and both edited-file hashes. +The isolated branch is `codex/exec-ctrl-summary-trial`. Its reviewed local commit is +`2d2edade281c155263bf03cf59091e746c115c89`: three files, 159 insertions and two +deletions. The target branch is not pushed; no target PR, merge or release occurred. + +Acceptance: reproduce the error-summary discrepancy, repair it without changing +successful/partial reporting, run the existing Go suite and added regression tests, +verify a local build when available, compare explicit current source before record +closure, and retain sanitized evidence. Use stub pingers and temporary files only. +Do not run the monitor, contact Splunk/monitored hosts, read real endpoint/config +files, install an application, or publish target changes as a release. + +## Initial discrepancy + +`runEndpoint` says "Treat as total failure" when its pinger returns an error, but +passes configured count, zero successful, **zero failed, 0% loss and zero average +latency** into its summary. Its ordinary all-failed path uses failed=count, +loss=100% and unknown latency=-1. The old PowerShell implementation also computes +failed=count-success. Metrics consume these fields, so the error branch can +misrepresent an unavailable target in downstream reports. + +Reproduce with an injected error, no ICMP/HTTP calls. Check normal success, +partial/all failure and normalized count. This is a correctness finding; no +exploitation or universal vulnerability claim is inferred from it. + +## Evidence and framework feedback + +The regression supplies a stub pinger returning an error. Before repair, +`go test ./internal/engine -v` exited 1: all four error/count subcases and the +persisted-file test failed. Normal success, partial/all failure and empty-results +subcases passed. The repair changes the error branch from +`buildSummary(ep, sumID, sumTs, count, 0, 0, 0, 0)` to +`buildSummary(ep, sumID, sumTs, count, 0, count, 100, -1)`. + +The README's original build command also failed: after `go -C .\go`, its package +argument `.\go\cmd\pingmonitor` resolved to a nonexistent `go/go/cmd/pingmonitor`. +Correct it to `go -C ./go build -o ../pingmonitor.exe ./cmd/pingmonitor`, and document +the failure contract and offline regression command. This is a documentation +execution defect, separate from the availability calculation. + +| Check actually performed | Observed result and scope | +| --- | --- | +| Baseline `go test ./...` | Exit 0; three existing config tests, no engine tests; committed baseline only | +| Added regression against original implementation | Exit 1, wrong failed/loss/average values reproduced; no real pinger | +| Repaired `go test ./... -v` | Exit 0; six top-level functions including three new engine tests and eight engine subcases | +| Temporary NDJSON output | Correct failure summary persisted through the actual output manager; no extra record; HEC/metrics disabled | +| `go vet ./...` and `git diff --check` | Exit 0 | +| Exact corrected README build command | Exit 0; local binary built | +| Binary `--version` and `--help` | Exit 0 before configuration/engine startup; version still v5.1.0 | +| Old manifest against repaired source | Exit 2, `snapshot files changed or manifest is inconsistent` | +| Fresh scoped record/snapshot | Exit 0, valid and ready to close; behavior/verification require test evidence, separate local build gate requires build evidence | +| Original checkout preservation | Main/HEAD/status unchanged; both original modified-file SHA-256 values match intake; untracked status unchanged | + +The target's three new engine functions cover backend errors with/without +individual output, zero/negative count normalization, success, partial/all +failure, empty replies and decoded persisted output. They assert the reporting +contract, including no fabricated individual replies, rather than the repaired +function call alone. Full tests and vet ran after the final source/README changes. + +The local evidence bundle retains `before.json`, `pending-record.json`, +`stale-check.json`, `after.json`, `complete-record.json`, `closure-check.json`, +execution notes and the unreleased binary. Notes summarize observed tool output; +they are not verbatim transcripts or independent reports. Nothing relabels the +failing run as current passing evidence. The snapshot selects the same 20 Go +source/module/README files before and after repair, excluding real config, +endpoints, original unfinished edits, the binary and environment state. + +- Before subject: `sha256:80aab2a9be8b5ad102bd0365e6ca0f36f0fc61e6b073b7463ceee227a3d37ab4`. +- After subject: `sha256:c9700ccf1bd77dc3e52930cd96e13f23e452282a3ce5993d51f8558e99fc1339`. +- Local binary SHA-256: `570a59137b9f6e14881119236d50d95cac99d26f035a111a8d58f7bf5917426f`. + +The optional helper rehashed the source and checked record structure and evidence +kinds. It did not execute references or independently verify test truth, policy +completeness or reviewer independence. The author performed the local diff review. +The binary was retained outside the original checkout, not installed or operated. + +## Observed axes and improvements + +| Axis | Observation | +| --- | --- | +| Link retrieval / blind activation | Not exercised: method already in context; the actual prompt above contained no new method link | +| Selection | Task mode with low-risk behavior/operations; relevant modules and scoped gates, no full project inventory | +| Boundaries and Git | Local worktree isolated committed input; no reset/stash, instruction/settings installation, external message or production access | +| Correctness / troubleshooting | Actual reporting defect reproduced before repair; neighboring outcomes and persisted file checked | +| Evidence | Historical source rejected after change; fresh source/typed evidence used for closure; live acceptance not claimed | +| Overhead | No target control-document pack; one existing task/case record, optional caller-retained evidence and three target files; no comparative timing/token measurement | + +Two protocol refinements follow from actual use: + +1. Extend [field-trial guidance](../v2/FIELD_TRIALS.md) with an optional existing-project + revisit recipe. Preserve dirty work, declare whether the worktree includes it, + and distinguish a controlled revisit from independent task-plus-link adoption. + Record which controls were actually exercised instead of forcing every module + into a small task. This trial supports one bounded Go fix, not the entire product. +2. Clarify [CLI output handling](../v2/TOOLING.md): valid/pending JSON is on stdout; + invalid-input JSON is on stderr with exit 2. An initial capture wrapper parsed + only stdout and failed on the stale-input rejection. Correcting the wrapper + captured the expected denial; this was not a helper defect. + +No helper/catalog behavior or version changes are needed for these documentation +refinements. The compact activation entry and host adapters remain unchanged. + +Framework follow-up validation on Windows/Python 3.13.7 passed: 10 modules and +289 supported local links; 108 discovered tests, 106 passed and two skipped +because symlink creation is unavailable on this host. `git diff --check` passed. +These checks cover the framework and its existing synthetic fixtures, separately +from the Go application's regression/build evidence above. + +## Remaining acceptance + +No real ICMP, Splunk/HEC/metrics delivery, installed service, customer workload, +security scanner, organization ruleset or team connector was exercised. The +unfinished endpoint-reload changes remain unqualified. Neither this case nor its +passing record establishes cross-IDE compatibility, compliance, an independent +review or production readiness. Those gates remain separate from this completed +local fix and from the merged framework preview. diff --git a/docs/exec_ctrl/V2_PUBLICATION_READINESS.md b/docs/exec_ctrl/V2_PUBLICATION_READINESS.md index 8beb2ae..0a10057 100644 --- a/docs/exec_ctrl/V2_PUBLICATION_READINESS.md +++ b/docs/exec_ctrl/V2_PUBLICATION_READINESS.md @@ -5,7 +5,7 @@ Method and target: this repository. Branch: `codex/publication-readiness`. User scope: put tests in place, execute and improve the framework, iterate on failures, and prepare a main-merge candidate before use in a real project. Implementation/local acceptance: `complete`. Hosted verification/publication: -`complete` for the tested code below. Main merge: `not_performed`. +`complete` for the tested code below. Main merge: `complete` (verified 2026-10-05). ## Scope and acceptance @@ -66,14 +66,25 @@ catalog, activation version and current synthetic examples advance to preview 4. `codex/publication-readiness` into `main`. This record's documentation update receives its own CI validation; the code identity above remains the repair evidence. +## Merge and subsequent use + +[PR #3](https://github.com/LeiterConsulting/exec-ctrl/pull/3) merged on 2026-10-05 +at 17:59:03 UTC as `bd343d405825f19c7a5b876ddea605083ecf7739`. Its merged tree +matches the final publication candidate `ecc9f08aef42e087958ddbb8cf5bcac9029132e9`. +The original failed runs and repaired-code evidence above remain historical proof +for their stated revisions. The merged preview then supported a +[controlled real-project revisit](REAL_PROJECT_PING_MONITOR.md); that case records +its own target, checks and limits rather than extending the earlier CI claims. + ## Remaining acceptance -This increment prepares a preview for main merge and real-project trial. It does -not certify every agent/IDE, semantic module behavior, real organization rules, +This increment published a merged preview and supports scoped real-project trials. +It does not certify every agent/IDE, semantic module behavior, real organization rules, scanner findings, external connector writes or production readiness. Independent task-plus-link adoption remains a [field gate](../v2/FIELD_TRIALS.md). Stable-release reuse/distribution terms remain an owner decision recorded in the [roadmap](../v2/ROADMAP.md); no license terms are invented here. The prior [session evolution](V2_SESSION_EVOLUTION.md) and lab records retain their -original revisions/results. Main merge and a stable v2 release are separate actions. +original revisions/results. The completed main merge and a stable v2 release are +separate actions. diff --git a/docs/v2/FIELD_TRIALS.md b/docs/v2/FIELD_TRIALS.md index e2cccff..6bf4146 100644 --- a/docs/v2/FIELD_TRIALS.md +++ b/docs/v2/FIELD_TRIALS.md @@ -4,7 +4,9 @@ Status: independent blind cross-agent trials not run. The [controlled local lab](../exec_ctrl/V2_CONTROLLED_LAB_RESULTS.md) and [adversarial second pass](../exec_ctrl/V2_SECOND_PASS_TESTING.md) completed in the authoring task. This file is the independent-trial protocol, not a claim that -the scenarios below passed across IDEs. +the scenarios below passed across IDEs. A later +[Ping Monitor revisit](../exec_ctrl/REAL_PROJECT_PING_MONITOR.md) exercised a +bounded fix in an existing real project; it was also a controlled authoring run. For each host, use a disposable target repository with realistic instructions, code and tests. Record host/version/settings, date, method commit, target baseline, @@ -41,3 +43,20 @@ Score retrieval, selection, action boundaries, task correctness, evidence and overhead separately as pass/fail/blocked. Record exact observed deviations and reproduction, not a model's self-reported confidence. Repeat changed cases after fixes. Do not claim statistical reliability from a single successful trial. + +## Controlled revisits of existing projects + +For a small, mostly completed project, choose one concrete discrepancy or +unfinished acceptance check that available tools can verify. Capture baseline, +dirty state and the included source before acting; an isolated worktree does not +automatically include uncommitted changes. Preserve unrelated work and say exactly +which version was tested. Prefer the cheapest meaningful regression and verify +affected documentation commands when operations change. + +Reuse the existing task record. Retain a failing reproduction, repaired diff and +current passing evidence separately; optional helper snapshots should cover the +selected inputs and explicit exclusions. Do not run unrelated modules or access +production systems just to obtain coverage. Record actual tool versions, prompt, +method revision, exercised controls, unavailable checks and any useful framework +improvement. Label preloaded context and author review explicitly. A controlled +revisit does not satisfy the independent task-plus-link trial above. diff --git a/docs/v2/ROADMAP.md b/docs/v2/ROADMAP.md index e51c24c..8b4190a 100644 --- a/docs/v2/ROADMAP.md +++ b/docs/v2/ROADMAP.md @@ -53,6 +53,13 @@ The [first controlled local lab](../exec_ctrl/V2_CONTROLLED_LAB_RESULTS.md) and hosted Windows/Linux CI passed after publishing `v2`. Independent blind adoption trials remain the next gate; the controlled run does not substitute for them. +After the preview 4 main merge, a +[controlled Ping Monitor revisit](../exec_ctrl/REAL_PROJECT_PING_MONITOR.md) +reproduced and repaired a real Go availability-summary defect, checked temporary +file output and corrected a broken build command. It exercised source freshness +and preservation of existing dirty work. The resulting trial/CLI documentation +refinements come from that observed use; independent adoption remains untested. + Run the [trial protocol](FIELD_TRIALS.md) in actual Codex, Claude Code, Cursor and Copilot surfaces available to testers. Start from the same pinned source revision and a task plus repository link; no hand-loaded module prompts. Include small fixes, diff --git a/docs/v2/TOOLING.md b/docs/v2/TOOLING.md index c03fc97..975a0a2 100644 --- a/docs/v2/TOOLING.md +++ b/docs/v2/TOOLING.md @@ -144,6 +144,10 @@ report's truth, environment identity or an actor's independence. See the Exit codes: `0` = valid route/framework, or record ready to close; `1` = structurally valid record with unresolved gates; `2` = invalid input or inconsistent claim. Read `state` as well as `ready_to_close`; readiness does not change task state. +Valid/pending results are JSON on stdout. Invalid-input JSON is on stderr with +exit 2; automation should capture both streams and inspect the exit code before +parsing the result. Do not mistake an empty stdout on rejection for successful +validation, or a caller's parsing error for a failed application test. ## Limits diff --git a/templates/v2/FIELD_TRIAL.md b/templates/v2/FIELD_TRIAL.md index 5a68fae..536700f 100644 --- a/templates/v2/FIELD_TRIAL.md +++ b/templates/v2/FIELD_TRIAL.md @@ -3,12 +3,16 @@ Record actual observations; this template is not a passed trial. - Date, host/version/settings and available tools: -- Exact user prompt (task plus method link): +- Trial type (independent task-plus-link or controlled revisit), preloaded context + and reviewer independence: +- Exact user prompt (retain any method link actually supplied): - Method revision and target baseline/dirty state: - Intended task outcome and authorized operations: - Files actually retrieved, revision consistency and evidence: - Controls selected and observed action boundaries: - Resulting diff, preserved user work and task acceptance: +- Included/excluded source, especially uncommitted work omitted from an isolated + worktree: - Checks actually run and reports: - Retrieval / selection / boundaries / correctness / evidence / overhead scores: - Observed failure, sanitized transcript location and reproduction: