Skip to content

feat(search-pipeline)!: apply a label source’s transforms to its referents - #1096

Merged
ddeboer merged 2 commits into
mainfrom
feat/search-referent-document-labels
Nov 16, 2026
Merged

ddeboer merged 2 commits into
mainfrom
feat/search-referent-document-labels

Conversation

@ddeboer

@ddeboer ddeboer commented Nov 16, 2026 •

Copy link
Copy Markdown
Member

A searchable reference is now searched by its referents’ labels as the target type’s transforms make them, not only as the referring graph states them (#1093). Before projection, each stage passes its referents’ key and label quads through the target type’s own transforms, inside the batch, and projects what comes back. Decision and alternatives: ADR 39.

In Linked Open Limburg, the Term transform replaces a keyed term’s label with the Authority’s, so the Term document for the CHT concept is called “boerenbont”. Works referencing that term were searched only by their publisher’s label for it, which is often missing or untagged (limburg/lol#318). With this change, those works are searched by the Authority’s label.

This keeps memory bounded by batchSize (ADR 12) and needs no stage order, so referents from any dataset are covered. A first version kept target documents’ labels in memory for the dataset pass, and review rejected it for breaking ADR 12. ADR 39 lists it as a rejected alternative.

Limits:

  • A transform that replaces a label also removes the publisher’s label from search, just as it does on the target’s document.
  • A referent stated by IRI alone has no quad of its own, so a transform has nothing to act on.
  • The target’s transforms come from its SearchStageType. searchIndexerPipeline stages every schema type, but a caller wiring searchStages directly must include the target type, with its transforms on the default reader.
  • The target’s transforms run with the referring stage’s context, so its distribution, stage name and counts. For a reference with several targets, each target’s transforms see every referent, not only those of its type. The projection still reads each referent through its own type.
  • local lookups such as creator still refuse searchable.

Part of #1058. It stays open for the local lookups.

BREAKING CHANGE: a root type’s transforms also run on its referents in the stages of types whose searchable references name it. They receive nodes that carry only the key and label field, once per referring batch.

Before: a Term transform saw only Term roots.
After: it also sees each about referent of a CreativeWork batch when about is searchable. Make the transform cope with a node holding only _sameAs and name, or drop searchable from the reference.


@ddeboer
ddeboer force-pushed the feat/search-referent-document-labels branch from 208fb82 to 6fa1916 Compare November 16, 2026 16:16
…rents

- Pass the key and label quads of each searchable reference’s
  referents through the target type’s own transforms, inside the batch
- Search the reference by the labels those transforms leave, as the
  target’s documents carry them
- Record the decision in ADR 39

BREAKING CHANGE: a root type’s transforms also run on its referents in
the stages of types whose searchable references name it, receiving
nodes that carry only the key and label field. Make such a transform
cope with that input, or drop searchable from the referring reference.
- Route the referents of an inline reference whose graph leaves the edge
  out, read off the inline node itself, through the endpoint’s target
- Key the referents’ transforms by class, as the schema resolves a
  class-equal lookalike
- Append routed quads in place instead of copying the array per quad
- Move referentHopFields above the private helpers and restore
  keyedFieldOf’s doc comment
@ddeboer
ddeboer force-pushed the feat/search-referent-document-labels branch from aad14f0 to 60cd630 Compare November 16, 2026 18:22
@ddeboer
ddeboer merged commit 90d7794 into main Nov 16, 2026
4 checks passed
@ddeboer
ddeboer deleted the feat/search-referent-document-labels branch November 16, 2026 18:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant