feat(iceberg): virtual-dataset info APIs + decimal-FK detection + subject-key fallback + secret redaction#1450
Open
aaj3f wants to merge 18 commits into
Open
feat(iceberg): virtual-dataset info APIs + decimal-FK detection + subject-key fallback + secret redaction#1450aaj3f wants to merge 18 commits into
aaj3f wants to merge 18 commits into
Conversation
… tables
Two deterministic Iceberg->R2RML emitter fixes found by the live Snowflake
ENTERPRISE_DEMO.DW audit.
FIX 1 - decimal FK detection (concern B). Snowflake NUMBER(38,0) surrogate keys
arrive as Iceberg decimal(38,0), NOT long, so the integer-only FK candidacy gate
silently dropped every *_KEY column before the name match.
- EmitColumn::is_key_type() gates FK candidacy on integer OR scale-0 decimal
(is_integer() retained for the strict-integer notion).
- key_types_match() lets scale-0 decimals match regardless of precision; integers
keep exact-width matching; integers and decimals never cross-match.
- Range-containment is now a DISAMBIGUATOR, not a hard gate: a single name+type
parent joins on name+type ALONE (Snowflake often supplies no bounds); range only
breaks ties among multiple same-named parents.
- json_to_typed_bound() parses integer-valued decimal-string bounds ("12345"),
since scale-0 decimal stats serialize as strings.
- New NonKeyTypeSkipped diagnostic surfaces a *_KEY/_ID column skipped for a
non-key type instead of dropping it silently.
FIX 2 - subject-key auto-fallback (concern F). Keyless / unverified-key tables
yielded NoSafeSubjectKey and no subject, leaving "select all tables" unsaveable.
- New SubjectStrategy { Auto (default), Identifier (strict = prior behavior) } on
EmitOptions plus a per-table override.
- Auto: a <STEM>_KEY/_ID column that is not provably non-null is used anyway
(SubjectKeyUnverified, downgraded from NoSafeSubjectKey) and kept join-compatible
(indexed as an FK parent); a table with no key-like column gets a deterministic
composite subject over all columns (new SubjectKeySynthesized diag) - never a rownum.
- Per-table override widened from Option<String> to Option<Vec<String>> for
composite subject keys (composite rendering already existed); a synthesized
composite is never indexed as a single-column PK.
- Override and identifier_field_ids paths stay STRICT under both strategies,
preserving FK-parent safety for a nullable identifier.
API/wire additions the solo side can use: GenerateOptions gains subject_strategy
(default auto); the server TableOverrideEntry gains subject_key (composite list) and
subject_strategy, keeping the existing single-string primary_key for backward
compatibility.
The enterprise_default.ttl golden is unchanged (byte-identical): its fixture is
all-integer-keyed with required identifiers and exercises none of the new branches.
New focused tests cover decimal(38,0) FK detection (engine + full preview pipeline),
bounds-free name+type joins, range-based disambiguation, the non-key-type skip
diagnostic, the Auto name-fallback and composite synthesis, the composite override,
and per-table strategy override.
…act config secrets Add a shared, virtual-aware branch to the ledger-info builder so a query-in-place R2RML/Iceberg dataset returns the SAME JSON shape as a native ledger's `info` (classes, properties, per-class counts) derived entirely from metadata — the compiled R2RML mapping plus Iceberg loadTable snapshot row counts (metadata-only, NEVER a Parquet/data scan) — instead of a thin nameservice stub. The db-server `/info` route and `LedgerInfoBuilder::execute` (MCP `get_data_model`, and by extension solo) both route through the one shared builder. Stop leaking the resolved OAuth2 client_secret / bearer token (P0): the virtual ledger_info emits source metadata only (type / catalog-uri / tables / snapshot), never auth; and `redact_graph_source_config` masks secret leaves in any stored graph-source config echoed to a client — the nameservice JSON-LD (`f:graphSourceConfig`), the generic graph-source stub, and the SSE `ns-record` stream. The lossless storage serialization is intentionally unchanged (query-time catalog auth and the config round-trip depend on it), so redaction is applied at the emission boundary. Native committed-ledger info/data-model output is byte-identical.
Multi-table "Generate Mapping" (solo's generate_r2rml over a full Iceberg catalog) fetched each table's Tier-A+B metadata preview SEQUENTIALLY — one OAuth token exchange + REST loadTable + manifest-list/manifest Avro reads per table, awaited in a plain for-loop. Measured live, a 16-table star schema took ~47s (vs ~3s single-table), past solo's synchronous generate invoke/gateway timeouts (30s router poll, 60s CloudFront origin read), so the router returned an opaque UpstreamError even though every per-table preview succeeded. Single-table generate is one round-trip, safely under the ceiling — which is why only the multi-table path failed. The emitter itself is NOT at fault: against live Snowflake decimal data it produces a correct 16-table mapping (FK joins resolved, 31 diagnostics, identical Turtle) whether the tables are generated singly or together. The only thing multi-table changes is wall-clock time. Fetch the previews with bounded concurrency (buffered(8)). `buffered` (not buffer_unordered) yields results in REQUEST ORDER, so the emitted mapping order and the first-table snapshot pin stay byte-for-byte deterministic; each preview still reads its own table's current snapshot. Live: a full 16-table generate drops from ~47s to ~10s (~5x), well under the ceiling, output unchanged. Add a secret-gated live integration test (it_iceberg_generate_live) that skips cleanly without FLUREE_ICEBERG_LIVE=1 + a resolvable PAT, builds the connection exactly as solo's build_iceberg_connection does, reproduces the multi-table path against live Snowflake, asserts a sane star-schema mapping (FK joins resolved, every table present in the Turtle), and guards against a regression back to sequential via a wall-time bound.
…e crawl Querying an R2RML/Iceberg graph source by class worked in SPARQL but not FQL. Three distinct gaps, all in the R2RML query path: 1. Variable @type dropped. `?s rdf:type ?type` (FQL `@type: ?t`, SPARQL `?s a ?t`) was rewritten with the class variable discarded, so `?type` came back null. Add `R2rmlPattern::type_var`; the rewrite records the variable and the operator binds it to each matched subject's declared class IRI — the same class-driven scan a bound `class_filter` performs, class projected instead of filtered. 2. Variable predicate dropped. `?s ?p ?o` bound subject+object but left `?p` null, and `<iri> ?p ?o` (the UI subject inspector) was rejected outright. Add `R2rmlPattern::predicate_var`; the operator binds `?p` to each triple's predicate IRI, which also makes bound-subject wildcards resolvable. 3. Crawl over a graph source returned []. `{"select": {"?s": ["*"]}}` uses native binary-index hydration, but an R2RML source has no flakes, so every subject resolved to null and the array came back empty (Solo "View Instances" broke). Expand a wildcard crawl through the R2RML operator: rewrite it to a flat `?s ?p ?o` + `?s a ?type` scan (bounded by a triple LIMIT so it early- terminates), then regroup the flat rows into per-subject JSON-LD documents. Bound-class @type and SPARQL are unchanged; native-ledger paths are untouched (the crawl expansion is gated on graph_source_id + a wildcard JSON-LD select). Tests: unit parity (FQL @type lowers to the same rdf:type scan as SPARQL a), engine tests for bound/variable @type and wildcard ?p binding over a mock provider, crawl-rewrite unit tests, and a skippable live-Iceberg @type test gated on FLUREE_TEST_ICEBERG_* env vars.
bplatz
reviewed
Jul 7, 2026
| // subjects have no rdf:type triple). | ||
| match pattern.type_var { | ||
| Some(tv) => { | ||
| for class_iri in triples_map.classes() { |
Contributor
There was a problem hiding this comment.
rr:class/rdf:type triples are only emitted on this subject-only branch. The variable-predicate wildcard path below iterates predicate_object_maps only, so <iri> ?p ?o / ?s ?p ?o never returns the rdf:type -> class triple.
bplatz
reviewed
Jul 7, 2026
| row.push((pv, Binding::iri(pred_iri))); | ||
| } | ||
| } | ||
| row.push((obj_var, object_binding)); |
Contributor
There was a problem hiding this comment.
For a templated (non-constant) predicate, ?p is left unbound but ?o is still pushed here, yielding a solution with a bound object and an unbound predicate.
bplatz
reviewed
Jul 7, 2026
| let where_clause = obj.get("where")?; | ||
| let limit = obj | ||
| .get("limit") | ||
| .and_then(JsonValue::as_u64) |
Contributor
There was a problem hiding this comment.
Only limit is extracted here; offset is never read or propagated into the flat query, so a crawl with a non-zero offset silently returns the first page.
bplatz
reviewed
Jul 7, 2026
| // `?s ?__crawl_p ?__crawl_o` — every (predicate, object) of the subject. | ||
| where_patterns.push(json!({ "@id": subject_var, CRAWL_PRED: CRAWL_OBJ })); | ||
| // `?s a ?__crawl_type` — the subject's declared class(es). | ||
| where_patterns.push(json!({ "@id": subject_var, "@type": CRAWL_TYPE })); |
Contributor
There was a problem hiding this comment.
?s a ?type is appended as a required pattern, so subjects whose triples-map declares no rr:class are inner-joined out of the crawl entirely. Multiple classes also produce a predicate x type cartesian that consumes the row budget.
bplatz
reviewed
Jul 7, 2026
| if let (Some((provider, table_provider)), Some(json)) = (r2rml.as_ref(), input.as_jsonld()) | ||
| { | ||
| if view.graph_source_id.is_some() { | ||
| if let Some(expanded) = crate::graph_source::crawl::expand_wildcard_crawl( |
Contributor
There was a problem hiding this comment.
This wildcard-crawl expansion only runs in execute_formatted. execute_tracked has no equivalent, so the same crawl issued with tracking headers falls through to native hydration and returns empty.
bplatz
reviewed
Jul 7, 2026
| col.name | ||
| ), | ||
| )); | ||
| SubjectKey::single(col.name.clone()) |
Contributor
There was a problem hiding this comment.
Worth reconsidering: SubjectKey::single sets index_as_pk = true, so under Auto this not-provably-non-null, uniqueness-unverifiable column becomes a live FK parent. The Identifier path returns none() for the same nullable case precisely to avoid that, and the composite-synthesis path emits a usable subject with index_as_pk = false. FK joins from other tables would then resolve against a column that may be null/non-unique, silently changing join cardinality with no diagnostic. Saveability only needs a subject, not an FK-eligible PK — consider index_as_pk = false here.
bplatz
approved these changes
Jul 7, 2026
bplatz
left a comment
bplatz
left a comment
Contributor
There was a problem hiding this comment.
Some comments noted that should be reviewed, approving to not delay once you get a chance to take a look.
…r-info A virtual dataset's info reported classes as class->count only, with no per-class property membership — the native path derives that from commit history, which a virtual dataset lacks. So the instance view and the data-model / LLM reader (which read stats.classes[c].properties) saw classes with only @id, and the LLM's 'Inspecting Knowledge' overview listed classes with no properties. Derive it deterministically from the compiled R2RML mapping (metadata-only, no data scan): each triples map contributes its class(es) and, to each, the predicates of its predicate-object maps with datatypes. Emit the native class shape — classes[c] = { count, properties: { <pred>: { types: {<datatype>: count}, langs: {}, ref-classes: {} } } } — so native consumers work unchanged. langs is empty (Phase-1 R2RML has no language tags); FK ref-classes (relationship targets) are a noted follow-up. The flat stats.properties block is unchanged. Gates: fmt + clippy --all-features -D warnings clean; fluree-db-api ledger_info tests 17 pass (extended the virtual-info test to assert per-class membership + class-scoping).
Both the native (build_ledger_info_with_options) and virtual
(build_virtual_ledger_info) builders now construct one typed LedgerInfo
(+ Ledger/Stats/ClassInfo/PropertyInfo/PropertyStat/NamedGraph/Index/
Source/Catalog), so the compiler enforces native<->virtual parity on the
stats/classes/properties core that previously drifted silently (a virtual
class emitted only `count` while a native class emitted
`count`+`properties`+`subclass-of`, breaking the instance view + LLM
data-model reader with no compile-time signal).
The nested stat types intentionally do NOT derive Default, so omitting a
shared field (e.g. per-class `properties`) is a compile error rather than a
silent shape divergence. Serde layout preserves exact null-vs-absent
fidelity (a plain Option emits `null`; skip_serializing_if omits the key)
and the kebab/camel wire keys (commit-t, index-t, ref-classes, subclass-of,
named-graphs, table-row-counts, g-id, commitId, indexId, ...).
Public API boundaries keep returning JsonValue via LedgerInfo::into_json,
so the db-server /info routes, the ledger-info cache, and the MCP markdown
formatter are untouched and the on-the-wire bytes are unchanged. LedgerInfo
also derives Deserialize so solo/conformance can consume the wire response
as the shared type.
commit and nameservice remain JsonValue: they are dynamic JSON-LD documents
whose shapes differ per path (native NsRecord vs virtual redacted
GraphSourceRecord; native commit object / {"error":...} / null).
Golden tests pin the virtual output to the pre-refactor json! bytes
(full-shape + null-count fixtures); the native path is covered by the
existing ledger-info integration tests (it_indexing_stats,
it_ledger_info_*).
…-estimate note for virtual datasets Three changes to the virtual (Iceberg/R2RML) dataset `/info` path. 1. Populate per-class `ref-classes` in build_virtual_ledger_info. For each RefObjectMap predicate, resolve rr:parentTriplesMap -> parent map and record its class(es) as the predicate's FK relationship targets (count = child row count; unknown degrades to 0 so the target class survives count loss). This mirrors the emitter's own round-trip FK reconstruction, keyed to classes() instead of table_name(). Extends the classes-and-counts test and both golden fixtures with a ref property. 2. Fix the /info timeout: replace the serial, client-per-table loadTable loop (a redundant OAuth exchange + loadTable per table, no keep-alive) with ONE shared RestCatalogClient reused via R2rmlCache.rest_client under the SAME fingerprint key as the scan path (extracted into rest_client_cache_key), a bounded .buffered(8) fan-out, and a wall-clock budget (FLUREE_ICEBERG_INFO_COUNT_BUDGET_MS, default 10s). On timeout/failure the counts degrade to empty while the mapping-derived structure still renders. 3. Flag virtual-dataset counts as coarse estimates in the MCP data-model markdown (format_data_model_markdown), detected from the emitted `source` block. Native output stays byte-identical.
A graph-source alias carrying a `#txn-meta` fragment (or `:branch#...`) hit
normalize_ledger_id, which splits `:` but not `#`, so the lookup missed and a
commit-history `from:{ds}#txn-meta` query 500'd (NotFound) instead of returning
[]. Route through parse_graph_ref/select_graph like the native db() path: the
default graph tags graph_source_id for provider resolution; a #txn-meta ref
selects the (empty) system graph on the genesis snapshot WITHOUT tagging
graph_source_id, so it returns [] rather than 500-ing or routing txn-meta to a
data provider that has no such graph.
…der Auto A Snowflake-managed Iceberg table can populate identifier_field_ids without marking the column `required`. select_identifier_subject_key rejected such a nullable declared identifier (NoSafeSubjectKey) → empty subject template → the scan later 500'd with "Subject map must have rr:template, rr:column, or rr:constant" (~80x on a wide schema). Thread SubjectStrategy in and mirror the name-fallback path already in this file: under Auto adopt the nullable identifier (downgrade to SubjectKeyUnverified) so the table stays browsable, but do NOT index it as an FK parent (a nullable parent key silently drops child rows at join); under strict Identifier keep the NoSafeSubjectKey rejection. Update the two tests that asserted the old empty-subject behavior under the default Auto strategy and add strict-strategy coverage.
Pull the graph-source subgraph-crawl interception out of GraphQueryBuilder into crawl::maybe_expand_crawl so every formatting terminal (not just the alias path) can route a virtual-dataset crawl through the R2RML operator instead of native binary-index hydration. Adds a FLUREE_R2RML_CRAWL_EXPAND master kill-switch (default on). Behaviour-identical for the alias path. Refs #1450
…ain the wildcard FIX 1 (correctness — the deployed "View Instances shows no instances"): wire crawl interception into all three FromQueryBuilder terminals (execute_formatted / _string / _tracked) via a shared try_expand_crawl helper, so the ledger-scoped/connection query path (POST /v1/fluree/query) routes a subgraph crawl through R2RML instead of native-hydrating the empty virtual index to []. Gated cheapest-first (is_wildcard_crawl), single-source only (parse_dataset_spec), else falls through unchanged. FIX 2 (perf co-requisite): class-constrain the crawl's wildcard in the R2RML rewriter. A new class_groups-emit fusion pass sets class_filter on a standalone wildcard (?s ?p ?o) + its co-located type-var (?s a ?t) when the subject shares a class, pruning the TriplesMap fan-out (16->1 for a per-table Iceberg mapping) while the wildcard's per-(p,o)-row semantics still return null-column subjects correctly. Guarded by wildcard_class_fusion_is_safe, keyed on SUBJECT-TEMPLATE disjointness (not the predicate-keyed class_fusion_is_safe): fuse only when every non-class TriplesMap is provably prefix-disjoint from the class's templates (refusing on column/template-less subjects), so a vertically-partitioned mapping is never under-fetched. Refused when reasoning is active (threaded reasoning_active through PreparedExecution -> ExecutionContext -> rewrite_patterns_for_r2rml) since the class prune is an exact rr:class match. Kill-switch FLUREE_R2RML_CRAWL_CLASS_FUSION (default on), coupled to the crawl master switch so fusion-off also forces expand-off (never the worse-than-today unfused fan-out + 429 storm). FIX 4: generalize detect_wildcard_crawl beyond ["*"] to ["@id"] (id-only, no POM scan) and explicit forward-predicate lists (distinct object var per predicate so they star-collapse into one scan + inherit class fusion via @type). FIX 3: verified unnecessary — the constant-@id subject inspector (<iri> ?p ?o) already lowers to new_bound_subject + subject-template reversal; constant-root crawls fall back to that working flat path. Tests: 8 rewrite-level fusion/guard unit tests (disjoint/vertical-partition/ column-subject/reasoning), 6 in-crate end-to-end crawl tests over a mock provider (subject parity, id-only, scan-count pruning, vertical-partition safety, multi-class @type, exact LIMIT), crawl detection/build unit tests, and a repaired live-Iceberg test (fixed the panicking flat @type query + added a query_from()/from crawl assertion on the deployed path).
Add DEBUG-level tracing spans on fluree_db_* targets so a host running with FLUREE_TRACING=xray/otlp (subscriber filtering `off,fluree=debug`) sees discrete, timed child spans for a virtual-dataset scan instead of the whole Iceberg cost collapsing into the native query_run span. This lets a slow virtual-dataset query be attributed to cold-remote-retrieval vs. caching vs. Parquet decode. Spans (all DEBUG, via the codebase's `.instrument(debug_span!(...))` idiom): - r2rml.scan_table (graph_source/r2rml.rs): the whole scan SETUP (loadTable + planning); closes when the stream is constructed. Body split into scan_table_inner so the trait method wraps it without holding a span guard across an .await. - r2rml.load_table (graph_source/r2rml.rs): the cold REST catalog round-trip — the highest-value span (isolates the ~1-3s cold retrieval / OAuth cost). - iceberg.oauth_token (auth/oauth2.rs): the OAuth token exchange, nested under load_table; no fields recorded (avoids capturing the secret/token). - iceberg.scan_plan (scan/send_planner.rs): manifest-list/manifest reads + file pruning; records files_selected/files_pruned/estimated_row_count. - iceberg.parquet_read (graph_source/r2rml.rs, per file): the actual read+decode. Created before tokio::spawn (which does not propagate the current span) and instrumented on the read future inside the task, so each concurrent read gets a distinct span parented under the query. Pure additive instrumentation: no behavior change, no new dependencies, existing log events retained. Feature-gated exactly as the surrounding iceberg scan bridge. Add a deterministic unit test asserting the oauth_token span is emitted for a token fetch (existing wiremock scaffolding; installs a process-global span-capturing subscriber so the callsite-interest cache is rebuilt and the assertion is order-independent under parallel tests). The load_table/scan_plan/ parquet_read spans need a live catalog + S3 to drive and so rely on build + review; they share the identical .instrument(debug_span!()) mechanism the test exercises.
Add a cheap "peek" at Iceberg table data — the first N rows, or the first N values of a column — as public fluree-db-api entries `sample_iceberg_rows` and `sample_column_values`. Powers the LLM agent data-peek tool and solo #756 (row preview). Reader change (fluree-db-iceberg): wire a `max_rows` budget through `decode_batches_arrow`. When set, the decode is restricted to the first surviving row group, the decode batch is sized to the budget, and the row loop stops (slicing the final batch) once N rows are produced. Combined with the range-backed chunk reader this fetches only the footer plus the first row group's projected column chunks — never a full-file scan. Unbounded callers pass `None` and are behavior-identical. `SendParquetReader::read_task_sample` drives that bounded decode from the source store (no disk-cache fill) for any file size. Sampler (fluree-db-api graph_source::iceberg_sample): mirrors `preview_iceberg_table`'s catalog/storage/snapshot handling, plans a projected single-table scan, reads the first task via `read_task_sample`, and renders cells to JSON through the shared `typed_value_to_json` path (temporals ISO-8601, decimals scaled, bytes hex, nulls JSON null). Tests: bounded-N + first-row-group-only + projection over the in-memory 2-row-group fixture (fluree-db-iceberg); pure JSON-assembly unit tests (bounded-N, projection, types/nulls/decimal) in the sampler; and an env-gated `#[ignore]` live regression (FLUREE_ICEBERG_LIVE=1 + PAT) that samples a real Snowflake table.
Add `Fluree::query_provisional_r2rml(conn: IcebergConnectionConfig, mapping_ttl, sparql)` — the LLM
agent's empirical validation lane. It compiles a candidate R2RML mapping in
memory and runs real SPARQL against `(mapping + iceberg connection)` through
the same R2RML operator the persisted path uses, creating NO graph-source or
nameservice record.
EphemeralR2rmlProvider implements both R2rmlProvider (returns the injected
compiled mapping) and R2rmlTableProvider (scans from the injected
IcebergConnectionConfig). Its scan mirrors the WP-DB2 sampler's
catalog/storage/metadata drive — reusing the shared rest_catalog_client +
build_preview_storage helpers and the SendScanPlanner + read_task primitives —
but streams ALL of a table's data files (not a bounded first-row-group peek).
It deliberately does not touch FlureeR2rmlProvider::scan_table (the persisted,
nameservice-backed scan); the small duplicated scan-drive glue is the
low-collision choice, notable for a future WP-DB1 consolidation.
The probe runs against a minimal genesis view tagged with a synthetic
graph-source alias, so maybe_wrap_for_graph_source routes patterns to the
injected provider (mirrors resolve_graph_source, minus the nameservice
lookup). Returns the same QueryResult shape the normal query path yields.
Scope: targeted SPARQL (?s a <Class> LIMIT k, joins, property filters); REST
catalogs only (Direct mode returns the shared typed error). The wildcard
View-Instances crawl is left to the persisted path.
Tests (all via a stubbed provider — injected mapping + in-memory batches, a la
MockCrawlProvider, no live catalog): a matching `?s a <Class>` binds rows and a
non-matching class is empty; a property-column probe binds the mapped value;
and — proving the real entry does NOT depend on the caller pre-registering the
mapping's namespaces — a two-class discrimination test over a view built
exactly as query_provisional_r2rml builds it (no insert_namespace_code),
asserting Person->2, Order->3 (distinct counts prove the class constraint
discriminates, not scan-all) and an absent class->0. Class matching is pure
string comparison: the parser encodes an unregistered-namespace IRI as
Sid{namespace_code: EMPTY, name: <full IRI>}, which decode_sid returns verbatim
to find_maps_for_class. Plus an env-gated (#[ignore] + FLUREE_ICEBERG_LIVE=1 +
PAT) live regression against a real Snowflake table.
… into one scan
The virtual-dataset "View Instances" wildcard crawl ({"?s":["*"]}) timed out at
scale (Customer 1.7M rows at LIMIT 20). Two causes, both fixed here and scoped to
the browse crawl only:
1. RefObjectMap objects were materialized by full-scanning each FK-parent
dimension table. Add trust_fk_refs (QueryExecutionOptions -> ContextConfig ->
ExecutionContext, set only by expand_wildcard_crawl) so the operator renders a
RefObjectMap object as a templated parent IRI from the child row's own FK
columns, with no parent scan. Byte-identical to the scan for a matched row; a
present-but-dangling FK renders the templated IRI instead of no triple (the
browse relaxation). Gated at the operator on the true-wildcard shape, so a
predicate-filtered ref or a WHERE-bound ref keeps the scan + dangling-FK
semantics and its subject set. Fires only for a single-column FK whose parent
subject is a pure IRI template over the join columns; refuses column/constant/
blank-node parent subjects and composite FKs (falls back to the scan).
2. The injected `?s a ?type` ran as a separate topmost LIMIT-budgeted scan,
starving the wildcard scan of the budget (a full ~512K-row materialize
window). Merge the projected type-var into the wildcard in
try_fuse_wildcard_class (removing the standalone type-var pattern) so the crawl
is a single budgeted scan; the operator's POM branch emits the per-(pred,obj)
x declared-class cartesian, equivalent to the two-scan inner join (the crawl
regroup dedups). Gated on crawl_active (threaded from ctx.trust_fk_refs) so
hand-written SPARQL {?s a :C . ?s ?p ?o . ?s a ?t} keeps its known-correct
two-scan plan.
Live-verified on large2/Customer: only DW.DIM_CUSTOMER scanned (was also
DIM_GEOGRAPHY + DIM_ACCOUNT), a single r2rml.scan_table, refs templated from the
child, 20 rows, ~9s cold vs the prior timeout; a non-crawl flat ref query still
scans the FK parent, confirming the scoping. Tests: rewrite merge/refusal +
non-crawl two-scan preserved, build_ref_shortcut reject-matrix, crawl e2e
(ref-templated with dangling FK, two-FKs-to-same-parent distinct, multi-class,
vertical-partition, subject-set parity); no regression across the r2rml and
graph-source suites.
…nto #1450 Brings feat/iceberg-act-step-engine onto feat/iceberg-virtual-dataset-support. Disjoint from the browse-crawl work (6fdcedb) — no file overlap; merge-tree clean. - WP-DB2 b8af743: bounded row/column Iceberg sampler (also solo #756) - WP-DB3 ac5e15c: query a provisional (unpersisted) R2RML mapping
…ancellation + crawl OFFSET Three fixes for virtual-dataset (R2RML/Iceberg) queries: 1. Bound-subject inspect prune (operator.rs): a `<iri> ?p ?o` query previously scanned EVERY TriplesMap (all dimension + fact tables) because a bound subject only filtered rows, never pruned maps. Skip any TriplesMap whose subject-template constant prefix the bound IRI does not start with — a provably conservative necessary-condition prune (the per-row match already enforces equality). Turns a 16-table fan-out into a point lookup on the subject's own table. Live: 16 -> 3 tables, timeout -> ~20s. 2. Cancellation poll sites (operator.rs): the R2RML scan loop never polled ctx.check_cancelled(), and next_batch can scan a whole table in one call, so a cancelled/timed-out query ran to the 900s Lambda ceiling. Add coarse polls at the next_batch loop top, before each Parquet row-group pull, and before each table scan (window/file/scan granularity, never per-row). A cancelled scan now stops promptly. 3. Crawl OFFSET (crawl.rs): the browse crawl ignored offset, so page 2 returned page 1. Apply offset to grouped subjects (flat budget covers offset+limit; skip(offset).take(limit) after grouping). Reuses constant_prefix (now pub(crate)). Tests: bound-subject prune scans only the matching table; offset paginates non-overlapping pages; poll sites review-verified. No regression across the r2rml + graph-source suites.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
Makes an R2RML/Iceberg virtual (query-in-place) dataset a first-class citizen of the dataset-info APIs and fixes deterministic generation for real Snowflake schemas. Four changes, from a downstream audit of fluree/solo's "generate a dataset from a catalog" flow:
ledger_info— the dataset About/Data panels (stats, classes/properties, counts) were empty for a virtual dataset becauseinfo/data-modelderive everything from a committedLedgerStatea virtual graph doesn't have.client_secret(a live PAT) to clients. A second instance of the same class (the SSEns-recordstream) was also leaking.decimal(38,0), i.e.NUMBER(38,0)) were silently dropped before the name match — no joins, no diagnostic.Details
Virtual-aware
ledger_info(fluree-db-api/src/ledger_info.rs,fluree-db-server/src/routes/ledger.rs)LedgerInfoBuilder::executenow falls back tonameservice().lookup_graph_source()on ledger-not-found and routes through a sharedbuild_graph_source_info. The native committed-ledger path is byte-unchanged.FlureeR2rmlProvider::compiled_mapping), and per-class/predicate counts come from the Iceberg manifestrow_count(preview_iceberg_tableatStatsTier::Schema) — metadata-only, no Parquet scan (cheaper than the native index path).graph_source_info_jsonis removed; both/infosites call the shared builder.Virtual
ledger_infoJSON shape (the contract solo's panels consume;stats.classes/stats.propertiesmirror the native shape so existing consumers keep working):{ "ledger_id": "<gs_id>", "t": <snapshotId|null>, "ledger": { "alias": "<gs_id>", "t": <snapshotId|null>, "commit-t": null, "index-t": null, "flakes": <totalRows|null>, "size": 0, "named-graphs": [ /* urn:default */ ] }, "graph": "urn:default", "stats": { "flakes": <totalRows|null>, "size": 0, "properties": { "<predIri>": { "count": <int|null>, "datatypes": { "<xsd|@id>": <int|null> } } }, "classes": { "<classIri>": { "count": <int|null> } } }, "commit": null, "nameservice": { /* gs_record_to_jsonld — config REDACTED */ }, "source": { "virtual": true, "type": "Iceberg"|"R2RML", "tables": ["ns.table", ...], "snapshot": <id|null>, "catalog": { "type": "...", "uri": "...", "warehouse": "...|null" }, "table-row-counts": { "ns.table": <int>, ... } } // NO credentials }Counts are
null(not0) when unknown; thesourceblock is the virtual-only addition.Secret redaction
redact_graph_source_config(recursive; masksclient_secret/token/secret/password/AWS keys/default_val, keeps an env-var name, and is a byte-identical no-op for secret-free configs so BM25/Vector/etc. are unchanged). Applied at every client-facing emission:gs_record_to_jsonld, the generic stub, and the SSE stream. The virtual builder structurally never emits auth.Serialize: persistence (IcebergGsConfig::to_json) and query-time.resolve()need the real secret, and that round-trip is covered by existing tests. Redaction lives at the emission boundary instead.Decimal-key FK detection (
fluree-db-r2rml/src/emit/{heuristic,input,mod,diagnostic}.rs,fluree-db-api/src/graph_source/iceberg_generate.rs)is_key_type);key_types_matchtreats scale-0 decimals as matching across precisions, integers by exact width, and never cross-matches int↔decimal.child == parent.pkname+type match now joins on name∧type alone; range-containment is demoted to a disambiguator when several same-named parents survive (Snowflake often supplies no integer bounds). Decimal-string stat bounds are now parsed.*_KEY/*_IDcolumn skipped for a non-key type emits aNonKeyTypeSkippeddiagnostic instead of failing silently.Subject-key auto-fallback
subject_strategy(autodefault /identifierstrict) onEmitOptions+ per-table override. Underauto, a table with an unverifiable-but-present key uses it with a downgradedSubjectKeyUnverifiedwarning (kept join-compatible); a truly keyless table gets a deterministic composite subject + aSubjectKeySynthesizedwarning (never a fabricated rownum, and never used as an FK parent —index_as_pkguard). Override +identifier_field_idspaths stay strict.subject_key: ["A","B"].New generate API surface (companion solo PR wires it)
subject_strategyon the generate request options + per-table override ("auto"default |"identifier").subject_keyon the per-table override (precedence over singleprimary_key).nonKeyTypeSkipped,subjectKeySynthesized.Testing
cargo fmt --all -- --checkclean ·cargo clippy -p fluree-db-r2rml -p fluree-db-api -p fluree-db-server --all-features --all-targets -- -D warningsclean · tests: r2rml 121/0, api 797/0 (+icebergledger_info17/0), server 114/0 · theenterprise.ttlgolden is byte-identical (all-integer-keyed → exercises no new branch = the no-regression proof). Virtual-info count derivation is unit-tested against a constructed mapping + stubbed manifest counts (no live catalog in CI).Companion / note
A fluree/solo PR re-pins this and wires the new generate surface. Security note: the live PAT exposure users see is served by solo's own query-lambda
handle_infofallback — it closes when solo re-pins (so itsledger_infocall becomes virtual-aware + redacted) and redacts its remaining fallback + proxy. This PR closes every db-side path.