Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
# 39. Apply a label source’s transforms to its referents

Date: 2026-11-16

## Status

Accepted

Amends [ADR 38](0038-search-a-document-by-its-referents-labels.md).

## Context

ADR 38 searches a reference by the labels the referring graph states for its
referents. A label a transform gives the target’s documents never reaches the
referrer. In Linked Open Limburg, the `Term` type’s transform replaces a keyed
term’s content with its Authority’s, so the Term document for a Cultuurhistorische
Thesaurus concept is called “boerenbont”, while a work referencing it is
searched only by what its own publisher calls the term – nothing, where the
publisher’s label is untagged and so outside the label field’s locales
([limburg/lol#318](https://codeberg.org/limburg/lol/issues/318),
[lde#1058](https://github.com/ldelements/lde/issues/1058)).

## Decision

Before projection, a stage passes the referents of its `searchable`
references, its inline referents’ included, through each target type’s own
transforms.

- **Only the referent’s key and label quads go in**, under the target’s IR
Aliases, and what the transforms return replaces them. The referent’s
`rdf:type`, and every quad the referrer reads for its own fields, stay out.
- **It runs inside the batch**, after the stage’s own transforms, so memory
stays bounded by `batchSize` ([ADR 12](0012-bound-memory-by-the-unit-of-work-not-the-input.md)).
- **A reference to its own type is skipped**: the type’s transform already
sees the batch. A target wired with its own `readers` has no transform to
apply.

## Consequences

- The referrer is searched by the labels the target’s transform leaves. Where it
replaces a publisher’s label with an Authority’s, the publisher’s is no longer
searched, as it is no longer on the target’s document.
- No stage order is needed, and a referent is covered whichever dataset its
target document comes from, as long as the referring graph states its key or
label. One described only in another dataset’s graph still adds nothing.
- A transform on a label source must cope with nodes carrying only a key and a
label, and runs once per referring batch as well as for its own roots. A
transform doing I/O pays for it there too, unless it caches.
- A referent stated by IRI alone has no quad for a transform to act on, so it
adds nothing to search.
- Labels are copies, changed only when the referrer is reindexed.

## Alternatives considered

- **Copy labels from the target’s projected documents, held in memory for the
dataset pass.** No transform runs twice, but the index grows with the dataset,
which ADR 12 rules out; it also needs label sources’ stages to run first and
misses referents from other datasets.
- **The same index on disk.** Bounded memory, but a store to manage per run for
a copy the transform can recompute.
- **Patch referrers in the writer before commit**, paging through the
collections. Bounded and complete, but it rewrites whole collections through
the engine on every run, outside the engine-neutral projection.
23 changes: 15 additions & 8 deletions docs/reference/search.md
Original file line number Diff line number Diff line change
Expand Up @@ -592,14 +592,21 @@ points at the term “Starluster” matches the query `starluster`. The extracti
reads each referent’s label field off the referent, one hop from the document,
beside its key. The projection folds the labels in the label field’s own
`locales` into the reference’s `${name}_search` companion, so a query still
searches one collection. The label is the one the document’s own graph states
for its referent. A referent stated by IRI alone, or one whose label only an
enrichment or another dataset supplies, adds nothing to search, and the label
field must therefore have a `path`. A copied label changes when the document is
reindexed. A reference naming no label source still folds its IRIs, and a
`derive`d or `from` reference reads no referent, so it is never searched by
labels
([ADR 38](../decisions/0038-search-a-document-by-its-referents-labels.md)).
searches one collection. The label field must therefore have a `path`.

Before projecting, `@lde/search-pipeline` passes each referent’s key and label,
as read, through the target type’s own transforms, so the reference is searched
by the labels the target’s documents get: where a transform replaces a term’s
label with an Authority’s, the referring document is searched by the
Authority’s. A transform on a label source therefore also receives nodes that
carry only those two fields, once per referring batch. A referent stated by IRI
alone, with no triple of its own, gives a transform nothing to act on and adds
nothing to search. A copied label changes when the document is reindexed.

A reference naming no label source still folds its IRIs, and a `derive`d or
`from` reference reads no referent, so it is never searched by labels
([ADR 38](../decisions/0038-search-a-document-by-its-referents-labels.md),
[ADR 39](../decisions/0039-apply-a-label-sources-transforms-to-its-referents.md)).

The resolved label carries **one word** wherever it surfaces: on the label
source’s own type, on the reference that resolves against it (`dataset { id
Expand Down
2 changes: 1 addition & 1 deletion package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion packages/search-pipeline/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -26,14 +26,14 @@
],
"dependencies": {
"@lde/search": "^0.46.0",
"@rdfjs/types": "^2.0.1",
"@traqula/generator-sparql-1-1": "^1.3.1",
"@traqula/parser-sparql-1-1": "^1.3.1",
"@traqula/rules-sparql-1-1": "^1.2.0",
"tslib": "^2.3.0"
},
"devDependencies": {
"@lde/search-typesense": "^0.61.1",
"@rdfjs/types": "^2.0.1",
"n3": "^2.13.12",
"testcontainers": "^12.1.0",
"typesense": "^3.2.0"
Expand Down
35 changes: 22 additions & 13 deletions packages/search-pipeline/src/extraction.ts
Original file line number Diff line number Diff line change
Expand Up @@ -141,6 +141,24 @@ export function extractionQueryString(
return generator.generate(extractionQuery(searchType, schema, options));
}

/**
* The fields a reference reads off a referent of a target it does not expand
* `local`ly: the target’s key field, where it is keyed, and – for a
* `searchable` reference – its label field, by which the reference is searched.
*/
export function referentHopFields(
field: SearchField,
target: RootType,
): PathField[] {
return [
keyedFieldOf(target),
// `searchSchema` guarantees a searchable reference’s label a path.
field.searchable === undefined
? undefined
: (labelFieldOf(target) as PathField),
].filter((hopField) => hopField !== undefined);
}

/** A per-query counter minting distinct value/referent variable names. */
interface VariableCounter {
next: number;
Expand Down Expand Up @@ -450,19 +468,10 @@ function buildReferentHops(
// is one of them, so emitting the hop as well would state it twice.
// A searchable reference is searched by its referents’ labels, so it
// reads them off the referent beside the key.
const hopFields = [
keyedFieldOf(target),
// `searchSchema` guarantees a searchable reference’s label a path.
field.searchable === undefined
? undefined
: (labelFieldOf(target) as PathField),
];
for (const hopField of hopFields) {
if (hopField !== undefined) {
const built = buildFieldHop(target, hopField, referent, counter);
template.push(built.triple);
expansions.push(built.pattern);
}
for (const hopField of referentHopFields(field, target)) {
const built = buildFieldHop(target, hopField, referent, counter);
template.push(built.triple);
expansions.push(built.pattern);
}
continue;
}
Expand Down
3 changes: 3 additions & 0 deletions packages/search-pipeline/src/search-indexer-pipeline.ts
Original file line number Diff line number Diff line change
Expand Up @@ -110,6 +110,9 @@ export interface SearchIndexerPipelineOptions {
* Named by `name` rather than by class IRI, like {@link registryTypes}; a
* name the schema does not declare throws at wiring time rather than
* attaching nothing and shipping an unenriched collection.
*
* A type’s transforms also run on its referents in the stages of the types
* searching it by its labels ({@link SearchStageType.transform}).
*/
transforms?: Readonly<
Record<
Expand Down
Loading
Loading