M2-13: canonical Bar resampling and interval materialization

[rustyeddy]

What changed

Adds Manager.Build, the canonical-build counterpart to Sync (#80): it executes exactly the ActionNormalizeCanonical and ActionDeriveCanonical entries in a Plan, mirroring Sync's own "only these actions" scope split (Build never runs ActionDownloadRaw/ActionRepairRaw; Sync never runs a canonical build action — each reports the other's actions in its own Skipped).

• normalizeAndPublish (build_normalize.go): raw → same-interval canonical, wiring M2-08 Normalize and validate OANDA records into canonical Bars #76's normalizer to actual publication for the first time. Any Suspicious/Rejected record anywhere in the partition aborts the whole call before anything is published (no partial publish); Incomplete records (OANDA's own complete flag false) are excluded without aborting.
• deriveAndPublish + aggregateBars (resample.go): resamples canonical D1 into canonical W1 — the interval with no raw source. Re-checks D1 completeness per calendar week against Coverage, independently of whatever range originally produced the Action, and leaves an unready week absent rather than aborting the whole month. The aggregation formula (OHLC/ticks/max-spread/tick-weighted avg-spread) is mined from trader-first-try/datamanager/candle_agg.go's aggregateWindow; legacy never actually built W1, so this is that formula applied to a genuinely new target, not transplanted code.
• oanda.FingerprintPartition: a targeted single-partition fingerprint (same sha256:<hex> form Inspect produces), used by normalizeAndPublish for Manifest.RawFingerprint without re-walking the whole raw archive.
• Manager.readAllBars: a Bars-draining convenience wrapper. Needed because Bars requires full coverage of its queried range while Coverage does not — deriveAndPublish checks D1 readiness once for the whole month via Coverage, then loads bars per ready week only via readAllBars.
• build_corpus_test.go (corpus build tag, operator-run, excluded from CI/make check, matching M2-07 Implement OANDA raw-archive inventory and integrity inspection #75's fullarchive precedent): H4/D1 corpus comparison against OANDA-native partitions, and a one-time legacy candle-v2 comparison, both required by the issue's acceptance criteria.

Why

Issue #81 (M2-13): materialize W1 (the one required interval with no raw provider source) via a canonical, DST-safe resampler, and validate derived intervals at corpus scale against OANDA-native data before any future candle-v2 deletion.

How it was tested

• build_test.go/resample_test.go: full unit/integration coverage of Build, both action paths, cache invalidation, cancellation, and edge cases. marketdata package coverage 92.3%, oanda package coverage 92.1%.
• make check (fmt, vet, test -race across the whole repo) is green.
• A real corpus run was performed against the operator's preserved archive during this issue (constants reverted to empty before commit — they default to "" and every corpus test skips itself). Two findings:
  • D1 derivation (this issue's actual production output) agrees with native OANDA D1 — the only disagreements found were at the comparison tool's own single-partition month-boundary limitation (it doesn't span adjacent raw months the way production's readAllBars/Bars does), never mid-month.
  • OANDA's native H4 anchors to the FX daily rollover (02:00/06:00/.../22:00 UTC in winter), not UTC-midnight truncation as ADR-012 currently specifies for H4. This is recorded as a quality finding (see ADR-020), not silently fixed — H4 is not a production derivation target in this issue, and revising ADR-012 is a separate, future architectural decision.

Documentation

• docs/arch/adr-020-historic-data.org: new DONE section covering both build paths, the mined aggregation formula, the same-month-key Parent convention, and the corpus-run findings above.
• marketdata/doc.go: new Manager.Build section.
• marketdata/internal/provider/oanda/doc.go: new FingerprintPartition section.

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This PR introduces a canonical “build” path in marketdata.Manager to materialize canonical datasets from a precomputed Plan, complementing Sync by handling only canonical actions (normalize + derive). It wires raw→canonical publication for the first time and adds D1→W1 resampling to materialize W1 (an interval with no raw provider source), along with documentation and corpus-scale operator tooling.

Changes:

• Add Manager.Build orchestration plus supporting publication/caching plumbing for canonical normalization and derivation.
• Implement D1→W1 resampling (deriveAndPublish, aggregateBars) and supporting helpers (weekIsD1Ready, readAllBars).
• Add raw single-partition fingerprinting and expand tests/docs (unit tests + build-tagged corpus operator tests + ADR/docs updates).

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
marketdata/build.go	Adds Manager.Build, build result types, version constants, and canonical publish helper w/ cache invalidation.
marketdata/build_normalize.go	Implements raw→canonical normalization + publication with abort-on-bad-record semantics.
marketdata/resample.go	Implements D1→W1 derivation + publication and the aggregation logic.
marketdata/query.go	Adds internal readAllBars helper to drain Bars() into a slice for resampling.
marketdata/build_test.go	Adds tests covering Build’s normalize/derive paths, skipping behavior, cancellation, and cache invalidation.
marketdata/resample_test.go	Adds focused unit tests for aggregateBars and weekIsD1Ready.
marketdata/build_corpus_test.go	Adds build-tagged (corpus) operator-run corpus comparison tooling against real archives.
marketdata/internal/provider/oanda/writer.go	Adds FingerprintPartition helper for targeted raw partition hashing.
marketdata/internal/provider/oanda/writer_test.go	Adds tests validating FingerprintPartition output and error behavior.
marketdata/internal/provider/oanda/doc.go	Documents FingerprintPartition and its intended use in canonical build.
marketdata/doc.go	Documents Manager.Build, its scope split vs Sync, and normalize/derive semantics.
docs/arch/adr-020-historic-data.org	Records the resampling/materialization decisions, formulas, and corpus-run findings.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[rustyeddy]

Fixed: aggregateBars now uses num.ParseRate (not MustParseRate) on both the per-bar tick weight and the total-tick inverse, propagating the error instead of risking a panic on a large tick count. Commit 3ea9620.

[rustyeddy]

The overall Build/Sync capability split and normalization policy look good, but I found four correctness/contract issues that should be addressed before merge. I also agree with Copilot's two existing comments (MustParseRate on runtime values and partial results from readAllBars). Latest CI is green.

[rustyeddy]

This unconditional check prevents a W1-only build even though ActionDeriveCanonical reads only canonical D1. It also makes a plan containing only skipped raw actions fail before those actions can be reported in Skipped. That contradicts deriveActionsW1's documented support for a Manager with no RawRoot. Please require RawRoot only when executing ActionNormalizeCanonical (where the dependency is actually used), and add a W1-only/no-raw-root test.

[rustyeddy]

Fixed: the RawRoot check moved out of Build and into normalizeAndPublish (the only path that actually reads raw data), so a W1-only Plan — or a Plan whose only entries are ActionDownloadRaw/ActionRepairRaw, reported in Skipped — now works with no RawRoot configured. Added TestBuild_DeriveWorksWithNoRawRoot and TestBuild_PlanWithOnlySkippedActionsReportsThemWithoutRawRoot. Commit 3ea9620.

[rustyeddy]

The records and their recorded fingerprint come from two separate opens. Sync can atomically replace this raw file after ReadPartitionRecords closes it but before FingerprintPartition reads it, causing canonical bars from revision A to be published with revision B's fingerprint. Please parse and hash the same opened file/snapshot (or otherwise verify a stable snapshot) so Manifest.RawFingerprint necessarily identifies the bytes that produced the bars; add a replacement-between-read-and-hash regression test.

[rustyeddy]

Fixed: added oanda.ReadPartitionSnapshot, a single os.ReadFile that both fingerprints and parses the identical in-memory bytes, replacing the previous ReadPartitionRecords+FingerprintPartition two-open pattern. normalizeAndPublish now uses it exclusively. Added TestReadPartitionSnapshot_ConsistentUnderConcurrentReplace, which races a WritePartition replace against ReadPartitionSnapshot under -race and asserts every observed snapshot's record count and fingerprint always describe the same file revision. Commit 3ea9620.

[rustyeddy]

d1Cov covers only [monthStart, monthEnd), but the last W1 span whose start belongs to this month commonly extends into the next month. weekIsD1Ready therefore cannot see a missing/stale/invalid adjacent D1 partition. If that partition is absent, this week is declared ready and readAllBars aborts the whole build; if it exists but is stale, Bars can supply it and the stale input is silently aggregated. Compute coverage over the full union of week spans being considered (through the final week's end), then test a month-end week with missing and stale next-month D1.

[rustyeddy]

Fixed: added weekSpansForMonth, which computes the full union of a month's week spans up front (including a final week's spillover past monthEnd), and deriveAndPublish now queries D1 coverage over that full union rather than just [monthStart, monthEnd). A missing or invalid next-month D1 partition is now correctly visible to weekIsD1Ready and the boundary week is skipped rather than wrongly declared ready. (A genuinely stale, as opposed to missing/invalid, next-month partition isn't independently reachable through this particular coverage call, since deriveAndPublish's own D1 coverage query passes a nil raw-inventory lookup — the same 'cannot verify staleness' precedent already established elsewhere in this package — so I added Missing and Invalid regression tests, which are the non-Current statuses this query path can actually produce: TestBuild_DeriveSkipsBoundaryWeekWhenNextMonthD1Missing and TestBuild_DeriveSkipsBoundaryWeekWhenNextMonthD1Invalid.) Commit 3ea9620.

[rustyeddy]

A W1 partition is not actually derived from only this same-month D1 revision: its boundary weeks can consume D1 bars from adjacent monthly partitions. Recording only the same-month parent means rebuilding an adjacent D1 partition will not make this W1 partition stale, even though its output may depend on the changed data; propagating only this parent's raw fingerprint has the same provenance gap. The explicit single-parent simplification therefore breaks the acceptance requirement for parent lineage/staleness. Please represent all contributing parent revisions (or define a deterministic composite parent revision/fingerprint) and have isStale compare the complete input set.

[rustyeddy]

Fixed: added combineParentLineage, which folds every D1 manifest that actually contributed a published bar (still exactly one in the common, non-boundary-spanning case — output unchanged from the original single-parent design) into one composite sha256: Revision/RawFingerprint. coverage.go's isStale now recomputes the identical composite from a stored Manifest's own LastBar/Span via the same w1SpansNextMonth rule deriveAndPublish uses to decide what to record, so the two can never disagree about which case applies. Added TestBuild_DeriveCombinesParentLineageAcrossMonthBoundary, which rebuilds only the contributing next-month D1 partition and confirms Coverage now reports the W1 partition Stale. Commit 3ea9620.

[rustyeddy]

Addressed all six review findings (2 from Copilot, 4 from @rustyeddy) in commit 3ea9620 — replied individually on each thread with the specific fix. Summary:

1. aggregateBars: num.ParseRate instead of MustParseRate on runtime tick counts (panic risk fixed).
2. readAllBars: returns nil, not a partial slice, on non-EOF errors.
3. Build's RawRoot check moved into normalizeAndPublish — a W1-only Plan now works with no RawRoot.
4. oanda.ReadPartitionSnapshot: one atomic read producing records + fingerprint together, closing the TOCTOU window against a concurrent Sync replace.
5. deriveAndPublish's D1 coverage query now covers the full union of week spans (weekSpansForMonth), so a missing/invalid next-month D1 partition a boundary week spills into is correctly seen as not-ready.
6. Manifest.Parent's Revision/RawFingerprint are now a composite (combineParentLineage) over every D1 partition that actually contributed a bar, not just the same-month one; isStale recomputes the identical composite so the two never disagree.

New regression tests for all six (including a -race concurrent-file-replacement test). marketdata coverage 92.2%, oanda coverage 91.9%. make check green. ADR-020 updated with a post-review fixes addendum.

[rustyeddy]

One convergence edge case remains after the otherwise solid review fixes:

1. Build a January W1 partition while the final January-starting week spills into February, but February D1 is not yet available. deriveAndPublish correctly skips that week and publishes the earlier weeks.
2. Because the skipped week contributed no bar, the W1 lineage contains only January's D1 revision. The resulting W1 partition is therefore PartitionCoverageCurrent, but its Coverage.Gaps includes the skipped final week.
3. February D1 later arrives. That does not change the recorded January-only parent lineage, so the W1 partition does not become stale.
4. deriveActionsW1 currently immediately skips every PartitionCoverageCurrent partition, without considering its gaps, so no new ActionDeriveCanonical is produced and the missing W1 bar never converges.

Please make W1 planning reconsider a Current partition when it has W1 gaps, and schedule derivation once the complete D1 input for those week spans—including cross-month spillover—is available. A useful end-to-end regression would: build with February D1 absent, verify the boundary W1 gap; publish February D1; call Plan again; verify it emits ActionDeriveCanonical; run Build; and verify the boundary W1 bar is now present.

[rustyeddy]

Fixed in commit 5fc235f. deriveActionsW1 now reconsiders a PartitionCoverageCurrent W1 partition when Coverage's own W1-level Gaps overlap its month, checking each overlapping gap's own D1 readiness via weekIsD1Ready (the same per-week test deriveAndPublish applies at Build time) over a D1 Coverage queried through weekSpansForMonth's coverageEnd, so cross-month spillover is visible to the planner too — not just requiring the whole month's D1 to be gapless, which stays the (deliberately lazy) behavior for the missing/invalid/stale branches.

Added TestPlan_W1ConvergesAfterBoundaryGapFillsIn, exactly the end-to-end sequence you described: build with the boundary week's D1 absent → verify the gap → publish that D1 data → Plan again → verify ActionDeriveCanonical is emitted for the already-Current partition → Build → verify the boundary bar is present and the gap is gone.

make check green, coverage held at 91.9%. ADR-020 updated.