explorer: #248 Flavor B — search by EXTERNAL concept URIs (Getty AAT, UBERON/OBO) via a pid→concept-URI projection

[rdhyee]

The ask (Eric Kansa)

Search material samples by a concept's URI/PID using linked data, not strings (2026-05-29 call). Eric's four guide queries — these are the acceptance tests:

1. 'bucchero ware' → https://vocab.getty.edu/aat/300387149
2. 'tibia' → https://purl.obolibrary.org/obo/UBERON_0000979
3. 'femur' → https://purl.obolibrary.org/obo/UBERON_0000981 (in geographic context: Italy)
4. 'whorls (spindle flywheels)' → https://vocab.getty.edu/aat/300263796

Key finding (verified 2026-06-01)

Flavor A (#248/#252) returns 0 for ALL four — these are EXTERNAL vocab URIs, not iSamples-vocabulary, and they are not in object_type/material/context of sample_facets_v2. They live in the PQG graph as IdentifiedConcept nodes referenced by p__keywords / p__has_* (BIGINT[] node refs in the wide parquet). The keyword labels ARE abundant in text (bucchero 2,693 · tibia 16,577 · femur 13,388 · whorl 1,473), so A1 free-text approximates these today.

Proposed work

1. Confirm the graph path: resolve Eric's 4 URIs through p__keywords (+ category relations) in the narrow/wide PQG to verify reachability and get pid sets. (Narrow: isamples_202601_narrow.)
2. Build a denormalized projection pid → [concept_uri, concept_label, relation] (a new facets-style parquet) so described-by=<any-uri> can semi-join it — eliminating joins at query time. (Eric & Andrea repeatedly flagged this 'new denormalized parquet' need in the MVP notes.)
3. Frontend: point buildConceptFilter at the new projection (or UNION it with the existing iSamples-vocab columns); add URI→label resolution + free-text fallback for unknown URIs; paste-a-URI input.
4. Compose with geography (query 3: femur in Italy) — already works via concept filter + viewport.

Repos

Data/projection work likely in pqg/; frontend in isamplesorg.github.io. Cross-repo.

[rdhyee]

Flavor B investigation done — design doc: pqg/docs/FLAVOR_B_CONCEPT_INDEX_DESIGN.md (branch codex/flavor-b-concept-index).

TL;DR — the projection design is sound, but the data has a gap for Eric's exact URIs.

1. The PQG has rich concept edges (MaterialSampleRecord → IdentifiedConcept), so a sample-centric concept index is buildable today:

   • keywords 15.06M · has_material_category 8.30M · has_sample_object_type 7.65M · has_context_category 6.53M edges; 55,893 IdentifiedConcept nodes total.

2. But the 4 acceptance URIs resolve to ZERO IdentifiedConcept nodes in the published PQG (checked remote wide + narrow, local caches, and the facets mirror; exact + fragment scans):

   term	external URI	concept node?	graph pids	free-text count
   bucchero	getty/aat/300387149	no	0	2,693
   tibia	obo/UBERON_0000979	no	0	16,577
   femur	obo/UBERON_0000981	no	0	13,388
   whorls	getty/aat/300263796	no	0	1,473

   → The OpenContext→iSamples export isn't carrying these external (Getty/UBERON) concept URIs into the PQG as IdentifiedConcept nodes. The free-text counts are lexical matches in label/description, not concept-URI links.

Proposed projection (build via DuckDB over wide.parquet): flat isamples_YYYYMM_sample_concepts.parquet — pid, source, concept_uri, concept_label, relation_type — published beside the Explorer parquet on R2; frontend semi-joins pid on concept_uri (no graph joins at query time). Full schema + build SQL in the doc.

Decisions needed (esp. for Eric / OpenContext):

• Can the iSamples export carry external concept URIs (Getty AAT, UBERON) as IdentifiedConcept nodes / scheme_uri? Without that, Flavor B can match the iSamples-vocab concepts that ARE in the graph, but not Eric's external-URI examples.
• Interim: free-text on the concept label already approximates these queries today.
• Exact-URI vs broader/hierarchy expansion?

[rdhyee]

Pinpointed where the URIs are lost — it's upstream of PQG, at the OpenContext→iSamples export.

Checked the iSamples export directly (isamples_export_2025_04_21_16_23_46_geo.parquet, the format upstream of PQG):

field	carries	Eric's URIs present?
has_sample_object_type / has_material_category / has_context_category	STRUCT(identifier VARCHAR)[] — URIs, but only iSamples-vocab URIs	0 for all 4 Getty/UBERON URIs
keywords	STRUCT(keyword VARCHAR)[] — text only, no identifier	bucchero: 0 · tibia: 86 (text labels, no URI)
informal_classification	VARCHAR[]	0

So the external concept URIs (Getty AAT, UBERON) are absent from the export entirely — not just from PQG. OpenContext has them (its own obj=…getty…300387149 queries resolve), but the OpenContext→iSamples export isn't carrying them: the category fields only receive controlled-vocab (iSamples) URIs, and keywords is text-only with no identifier slot.

Action (OpenContext / Eric): extend the export so external descriptor concepts carry their URIs (e.g. a keyword/descriptor struct with an identifier, or as additional has_*_category/IdentifiedConcept entries). Once the URIs are in the export → PQG → our sample_concepts.parquet projection makes described-by=<getty|uberon uri> work end-to-end.

Meanwhile (no upstream dependency): the projection + described-by semi-join can be built/demoed now for the 55,893 iSamples-vocab IdentifiedConcepts already in the data — only the external-URI examples wait on the export.

[rdhyee]

Blocked by #263 — Flavor B's external-URI examples can't work until the OpenContext→iSamples export carries the external concept URIs (verified: Getty/UBERON URIs are absent from the export itself, not just PQG). #263 has the detailed ask + data-shape spec for Eric/OpenContext.

Unblocked half: the sample_concepts.parquet projection + described-by=<uri> rail can be built now against the 55,893 iSamples-vocab IdentifiedConcepts already in the data — the external (Getty/UBERON) URIs light up once #263 lands.