Skip to content

feat(pj_base): attach_source_record runtime-host slot for the M3 source cache (0.28.0) - #189

Open
facontidavide wants to merge 4 commits into
mainfrom
feat/attach-source-record
Open

feat(pj_base): attach_source_record runtime-host slot for the M3 source cache (0.28.0)#189
facontidavide wants to merge 4 commits into
mainfrom
feat/attach-source-record

Conversation

@facontidavide

Copy link
Copy Markdown
Contributor

The SDK half of PJ4's M3 source cache (PJ4 docs/RECORDING_AND_CACHING.md §3.4): one optional runtime-host tail slot so a provider can state, once per download, "this dataset is request X" — and the host does everything else (capture, cache, restore).

The slot

/* tail slot on PJ_data_source_runtime_host_vtable_t ("pj.runtime.v1"), offset 104, size 104 → 112 */
bool (*attach_source_record)(void* ctx, PJ_string_view_t descriptor_json, PJ_error_t* out_error) PJ_NOEXCEPT;

C++: DataSourceRuntimeHostView::attachSourceRecord(std::string_view), forwarded on DatasetIngestHostView so toolboxes use their canonical facade. [stream-thread]; bytes copied during the call; absent slot (older host) reads as "no caching" via PJ_HAS_TAIL_SLOT with an explicit wrapper error.

Contract highlights

  • No identity parameter, no dataset parameter. The host keys its cache on its own digest of the descriptor bytes (an internal keyspace; the layout round-trips the same bytes, so it never needs to agree with any provider identity scheme), and the unspoofable scope is the provider id from the binding. The ctx is already per-download, so the dataset is implicit. This deliberately simplifies the spec's original (dataset, identity, descriptor_json) sketch — a plugin-supplied identity would allow aliasing one request's artifact onto another's, and a host cannot canonicalize under a provider-declared field policy anyway.
  • Refusal is allowlist-semantics: the host bounds and parses the descriptor and refuses anything it cannot fully account for; parse/bounds failure is a contract failure (false + PJ_error_t), never a trust verdict, never affecting ingest.
  • Refill-safe: the host may defer applying the record until its ingest transaction commits (staged on the ingest context), so a committed in-place refill or replacing reload does not lose it.
  • Last attach before the first push_message on this context wins; attaching after ingest begins is the error. Byte-exact re-attach is idempotent (matches PJ4's byte-equal trust rule).
  • D9 note carried in the header: the descriptor is request identity, never parser policy; credential material never belongs in one.

Mechanics

Precedent-exact tail-slot addition (the 0.21 set_host_info pattern): offset/size pinned by the ABI-layout sentinels test, both-direction compatibility (old plugin / old host), baseline.abi untouched per MINOR policy. VERSION 0.27.0 → 0.28.0; CHANGELOG entry with the offset line; docs updated (data-source-guide.md method table, ARCHITECTURE.md tail-slot history).

Tests

85/85 on both legs (RelWithDebInfo and Debug+ASAN); attach_source_record_test 5/5: string fidelity across the ABI, host-error propagation, absent-slot detection, short-struct_size gating, unbound-view refusal.

Review provenance: one adversarial combined review (ABI mechanics verified precedent-exact; all findings were contract-text and are applied in the second commit).

Downstream note: PJ4 and pj-official-plugins each need a one-line designated-initializer addition when they bump to 0.28 (-Werror=missing-field-initializers).

🤖 Generated with Claude Code

facontidavide and others added 4 commits September 3, 2026 22:45
One provider->host call, stated once at download start: the canonical
descriptor JSON of the reproducible request a source answers. The host
copies the bytes and derives the record identity itself (the plugin never
supplies an identity, per the descriptor_import doctrine), enabling the
host-driven transparent source cache. Byte-identical re-attach is
idempotent; different bytes fail; failure is a contract failure, never a
trust verdict, and never affects ingest. Old hosts read as no-caching via
PJ_HAS_TAIL_SLOT; the C++ wrapper reports the absence explicitly.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The record is stored verbatim and keyed on the host's own digest (an
internal keyspace — never required to agree with a provider identity
scheme); the unspoofable datum is the provider id, taken from the binding.
Refusal is allowlist-semantics (unknown fields are an error), not a field
denylist. Last attach before the first push_message wins and the host may
stage the record until its ingest transaction commits, so a replacing
reload or in-place refill never silently loses it. Thread tag matches
push_message ([stream-thread]). DatasetIngestHostView gains the forward;
guide/architecture docs list the slot; tests add the unbound-host and
short-struct_size-with-stale-pointer gates.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant