Repository navigation
Codetrail: local learning system on a pinned openwiki fork - #1
Merged
Merged
Conversation
![[CL]](https://github.githubassets.com/images/gravatars/gravatar-user-420.png){kind=avatar}
Add a Node/TypeScript .gitignore covering build output, dependencies, env files, and pipeline/agent artifacts so parallel feature worktrees stay clean. Untrack agent session state that does not belong in the repo. Co-Authored-By: [CL] <noreply@[aN].com>
…provenance Vendor all 586 tracked files of langchain-ai/openwiki at 7bb583d664f2d1c2484fa44b9d591f6103ca8166 byte-identically (git archive, so blob bytes are preserved; only .gitignore stays the fork's own file) and record the import in UPSTREAM_BASE.json: the pin (commit, 0.7.2, MIT), a sha256 for every upstream file and the nine-entry modification allowlist from §6.1. scripts/check-provenance.mjs is the verdict and runs before install (ESM, node: builtins only): the default mode verifies every recorded digest and rejects drift outside the allowlist, --report prints the counts and allowlist, --self-test proves the check can fail by rewriting src/retrieval/wiki.ts in a temp copy, and --license asserts the MIT text, the upstream copyright line and the commit/repo named in docs/provenance.md. scripts/git-askpass.mjs answers GIT_ASKPASS from CODETRAIL_GIT_TOKEN without echoing it, and .node-version pins the runtime to 24.13.0. pnpm install --frozen-lockfile exits 0 with the upstream lockfile unchanged.
Thread an optional writeRepoFiles flag through the three generation entry points (CodeModeRepoSetupOptions, BeginRepositoryRunInput, NativeRepositoryGenerationOptions) so the learning module can analyse a user's checkout without writing .github/workflows/openwiki-update.yml, AGENTS.md or CLAUDE.md into it. Every field is optional and defaults to the upstream CLI behaviour, so existing callers and upstream tests are unchanged. Also add the /learn entry link (data-testid="learning-entry") to the visualizer header with one CSS rule, keeping PAGE a constant string with no new script or style element, and union the .gitignore with the remaining upstream patterns while keeping .artifacts/ and .worktrees/.
… the TS projects
Every JavaScript entry point now runs under an interpreter that satisfies
engines.node (>=22.22.0), even when the ambient `node` is older, and the
toolchain reports and audits that fact instead of assuming it.
- scripts/with-node.mjs resolves CODETRAIL_NODE_BIN, then the .node-version
installs (nvm/fnm/Volta/Homebrew/versioned), then PATH; it propagates the
child's exit code verbatim, maps signals to 128+signum, supports
--print-bin/--require, strips pnpm's `--` separator, and only augments the
child PATH — it never installs, switches or writes global Node config.
- scripts/check-env.mjs prints {node,pnpm,sqliteFts5,upstreamCommit,
nodeVersion,resolvedNode,resolvedVia}; --simulate-node fails before touching
node:sqlite, --pins names any drifting pin (engines.node, packageManager,
.node-version, both tsconfig entries), --audit-global scans scripts/,
package.json and README.md for global-Node mutations and proves the launcher
is read-only.
- The package.json script table routes every JS entry through the launcher and
adds env:check/provenance:check/fixtures/test:upstream/test:learning/
start:learning/smoke/accept/phase:check/backup/restore/gc/licenses.
- tsconfig.json keeps DOM code out of the server project and
tsconfig.client.json compiles src/learning/ui/client.ts against DOM.
- vitest.learning.config.ts isolates test/learning/** with the offline guard
hook (created by the test-harness issue).
- README.md is replaced by the minimal Codetrail README carrying the literal
"Node.js 22.22.0 or newer"; the six Chinese sections land in the docs issue.
tsconfig.json now carries the learning include/exclude split, so it joins
UPSTREAM_BASE.json's allowlist (10 of the 12 allowed paths) with a reason, and
docs/provenance.md plus the provenance test record the new count.
Verdicts: pnpm run env:check, env:check -- --pins, check-env --audit-global and
typecheck all exit 0; --simulate-node 22.21.0 exits 1 with
node_version_unsupported; test:upstream passes 239 files with no .only.
`node scripts/check-env.mjs --audit-global` (the literal AC-ENV-04 command) exited 1 on a machine whose ambient node is below `engines.node`, even though the launcher resolves a usable interpreter: the node verdict ANDed `process.version` with the resolved one. It now depends on the resolved interpreter only, so the report describes what the scripts actually run under. Also route the last two direct `node` invocations (`clean`, `postbuild`) through `scripts/with-node.mjs`, drop the `-e` exemption that let them pass the script-table test, broaden the launcher read-only guard to the mutating fs/fs-promises APIs, and filter the `node:sqlite` ExperimentalWarning at the source instead of removing warning listeners the launcher never added.
Remove the dangling `setupFiles` entry from vitest.learning.config.ts so the learning acceptance suite can run, and commit the QA-added environment tests covering the launcher guard, the pinned node verdict and the pins checks.
Add the 18-module contracts barrel that every later learning component imports instead of redeclaring: branded ids, the closed error/reason tables, source refs, lesson/job/change shapes, the byte-exact key formulas, route DTOs, row records, the fifteen repository interfaces, service ports and the DI roots. The layer is a leaf (contracts/**, zod in lesson.ts, and import type from @langchain/core only) and ships with guards for isolated typechecking, import categories, barrel reachability and single declaration, plus independent recomputation of every digest and the PRD §5 error table.
…ories Add the node:sqlite persistence layer: one WAL writer connection with foreign keys and a 5000 ms busy timeout, an idempotent migration 1 reproducing the normative DDL, and fifteen synchronous repositories that own all SQL for their tables. The partial unique index on jobs(stream_id) is the database-level single-writer guarantee, messages persist answer_phase/unknown_reason/incomplete verbatim, and conversations.generation_ids_json has no update path.
Deliver the learning module's configuration surface so later waves stop
inventing paths, literals or provider wiring:
- paths: one data root (~/.codetrail or CODETRAIL_DATA_DIR), every accessor
derived from it, staging/<jobId>/{tree,knowledge} as the only build area,
and assertInsideDataDir rejecting traversal, outside paths and symlink
escapes with out_of_scope_path.
- env: resolveEnv with the documented defaults (300000 ms watch interval,
port 4319) and applyProcessHardening forcing telemetry, tracing, LFS and
prompt settings without writing any file or global config.
- defaults: the frozen §4.3 constants.
- model: createModelHandle with the fixture/anthropic seam,
assertModelConfigured (model_not_configured) and a stable data-sent manifest.
- credentials: an atomic 0600 credentials.json store with a hidden token and
CODETRAIL_GITHUB_TOKEN fallback, plus redactSecrets.
Adds the shared prerequisites every later learning suite depends on:
- `scripts/make-fixtures.mjs` builds the ten fixture repositories from fixed
bytes with a fixed author/committer/date under TZ=UTC and
core.autocrlf=false, and records their SHAs in `fixtures/manifest.json`, so
a second run leaves every SHA unchanged.
- `test/learning/setup/guard.ts` patches fetch, http(s).request/get,
net.connect/createConnection, tls.connect and dns.lookup to throw for any
non-loopback target while loopback test servers keep working; it fails
closed on unparseable targets and installs a per-test temporary
CODETRAIL_DATA_DIR so nothing touches `~/.codetrail`.
- `test/learning/helpers/fake-github.ts` serves Link paging,
state=all&sort=updated&direction=desc, X-RateLimit-*, Retry-After,
injectable 429/5xx and missing objects, truncates /compare at 300 files and
records every `{method, path}`.
- `scripts/scan-copy.mjs` and `scripts/scan-code.mjs` honour their documented
flags, print `path:line` and exit non-zero on a hit, with explicit (never
silent) exclusions for tests and doc strings.
- `test/learning/architecture/{upstream,dom}-boundary.test.ts` fail on
upstream imports outside the adapter and on DOM globals outside the client.
Exclude the `.tmp` fixture tree from Vitest discovery and ESLint, since the generated repositories carry their own `*.test.ts` files and malformed third-party source; `.gitignore` does not affect either tool. Record the two fork-patched config files in the provenance allowlist. Make the fixture Git helper independent of the developer's config by clearing global/system config, disabling autocrlf and signing, and pinning the fixture identity.
The generated fixture tree is already excluded from Vitest discovery and ESLint, but `eslint .` still flagged every `process`/`Buffer`/`fetch` reference in the repository's `.mjs` scripts, including the fixture and scanner scripts added for this issue. Add a scoped `**/*.mjs` block with Node's runtime globals so those files lint cleanly; no other rules change.
…lt points Provide the test doubles the learning suites depend on: a real BaseChatModel driven by occurrence-indexed JSON fixtures, a Clock that honours CODETRAIL_NOW, crash fault points for publish-boundary tests and a file:// bare-remote helper. Every call count and crash assertion downstream rests on these being deterministic and offline.
…oval Add the physical read-only project layer: canonical local/GitHub identity, shell-free Git execution with a zero-argv credential helper, bare mirrors that preserve unpushed commits, detached read-only worktrees with verified tree hashes, the SnapshotService/SnapshotReader read surface, and soft-delete removal that writes only deleted_at and watch_enabled in one transaction.
…d state Two bindTools() calls on one scripted model overwrote a single offeredTools field, so an earlier bound runnable (and the unbound base model) answered as whichever worker kind was bound last. Tool names now come solely from the per-invocation call options that withConfig carries, keeping every runnable on its own tool set. createClock() validated the entire runtime env through resolveEnv(), so an unrelated malformed variable such as CODETRAIL_PORT broke clock creation; it now validates only CODETRAIL_NOW.
Phase A now runs real verdicts: check-env, check-provenance, contracts-leaf, the 0a-3d suites and a-gate.test.ts, stopping at the first red suite and refusing to pass when a path is missing or a phase is unregistered. The gate test proves the five A properties against the production pipeline: fixed-SHA commit/tree hashes, a byte-identical source tree, baseline reuse vs. a recorded fallback, interrupted-job recovery on the same job id, and failure isolation.
…d drain on signals
…nd the read-only answer surface Register the B stage in the phase-check registry and add the gate test that drives the real server: a Chinese last-mile-mini lesson through the reader route, three follow-up turns bound to one generation with passing checks, verbatim restore after teardown and re-composition over the same data directory, unknown labelling for unsupported claims, source_missing gaps for unavailable external background, and exact equality on the five read-only tools and the answer route surface.
Encode the research-side failure scenarios as fourteen executable tests over the real fixtures, scripted model and production builders/validators: - R01/R02/R07: three-layer last-mile citations, no invented decision, independent format/existence sources, planned-only boundaries stay planned and unsupported claims become unknown with a reason. - R03/R05/R06/R16: PR call-site/rename change records, version-pinned conversations, Chinese/identifier query parity, and distinct before/ after refs at identical paths and line numbers. - R08/R09/R17: injection text stays inert and untrusted, local unpushed commits are analysed while dirty files are excluded and reported, and lessons keep macro/micro layers, separate rationale/analysis labels, explainable empty choices and independent reading positions. Also classify repository-hosted design/decision documents in the label and boundary matrices, so a planned-only design note can no longer back an implemented boundary or a recorded rationale claim.
…preview separation Register the C stage in phase-check (updates suite + c-gate test) and add the gate that drives real watch ticks against the fake GitHub server and the watch-remote fixture. It asserts one event per observed change across commits, merges, force-push gaps and renames; one event/job for a repeated delivery; a failed analysis retaining its event with the published cursor unadvanced and a waiting successor; independent default-branch and PR cursors; and PR preview generations separated from main by head and base context.
The '--through=D exits non-zero while C is red' test spawned a real phase-check run that re-executed the entire A and B chain (~5 min) from inside a unit test, which went red under load and drove the QA timeout. Build a throwaway --root tree (stub scripts and a stub scripts/with-node.mjs bridge) so the CLI's real registry, ordering, simulation and exit-code logic are exercised in milliseconds; the real A->C chain stays the acceptance command `pnpm run phase:check --through=C`. Also fail fast when runPending cannot transition a queued job, and replace the vacuous published-cursor assertions with a conditional check that a published cursor never advances past the failed event.
…sion-layer edge cases
…ttier The smoke, accept, backup/restore, gc and licence commands plus the JSON log writer and their tests now match the repository's Prettier style, so `pnpm run format:check` stays clean for the ops surface.
…ocations A backup taken from a copied or moved data dir archives absolute paths that name the original root, not manifest.sourceRoot. Restore previously used those paths verbatim, so it could delete and re-clone inside the live source data dir. Every rewritten path is now sandbox-checked with a mirror of assertInsideDataDir, stale absolute paths fall back to mirrors/<repoId>.git, work/<repoId>/<sha> and the generation's relative knowledge dir, and no rm/git operation runs on a path outside the restore root. Adds a regression test that seeds dir A, copies it to B, backs up B and restores into a fresh dir while asserting A is byte-for-byte untouched and the restored state is readable over HTTP.
Encode the follow-up-side failure scenarios as executable regressions that drive real ticks and real child processes: - R04: duplicate delivery yields one event/job; squash keeps two records; resuming from an interrupted cursor loses and duplicates nothing. - R10: init and update write no workflow/agent files and keep the analysed tree hash stable; an incompatible baseline falls back with a reason. - R11: multi-node, multi-page and large-diff discovery create exactly the server's change tally; unreadable history becomes history_gap; a moved base re-evaluates impact with a new cache key. - R12: a post-discovery failure stays retryable with parked successors; a SIGKILL leaves an interrupted job (same id) and no duplicate event. - R13: the four-point crash matrix exposes only complete versions whose lesson hashes match their sealed trees. - R14: cancellation and budget exhaustion never publish or succeed; 429s retry within budget and the four failure classes differ by code/copy. - R15: each source kind validates structurally; 404/409/403 ids behave; an out-of-scope source blocks publish while valid content lists the gap.
…endix A traceability map
A simulated red was never actually run, so continuing through the later phases only manufactured a "green while red" artifact and took minutes. Stop at the simulation and name the red phase first in the failure line, which is what the A gate asserts.
main was created from the GitHub UI with its own root commit, so the two branches had no common ancestor and GitHub refused to open a PR. This merge joins them. The only conflict was the copyright line in LICENSE; resolved in favour of main's 'Copyright (c) 2026 Jack11111eee'.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
Implements the single-user local learning system described in
project-learning-system-design.md(实施设计 v0.2): a bounded fork oflangchain-ai/openwiki(pinned at7bb583d6, MIT) that learns from arepository's change history and answers questions grounded in cited code.
Scope
Built by SWE-AF as 35 dependency-ordered issues across 15 levels. Every issue
passed its own QA + code-review gate before merging.
src/learning/test/learning/Verification
pnpm run phase:check --through=Dregressions:17 matrix:10pnpm run phase:check --verify-orderpnpm run acceptd-gate.test.tsProvenance
UPSTREAM_BASE.jsonrecords the sha256 of every one of the 586 upstream filesplus the closed list of 9 allowed modifications;
pnpm run provenance:checkverifies both. See
docs/provenance.md.Known issues
Tracked in
docs/known-issues.md— reproducible entries, not a wish list.Notes for review
main's LICENSE commit into thishistory.
mainwas created from the GitHub UI with its own root commit, sothe two branches shared no ancestor and GitHub refused to open this PR. The
only conflict was the LICENSE copyright line, resolved in favour of
main.test/learning/integration/holds 32 integration-suite files generated bythe pipeline's integration stage. They are not tracked and are not part of
this PR.