Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -86,7 +86,7 @@ Each package uses conditional exports with a `development` condition for local d
- Test files use `.test.ts` suffix in `test/` directory
- Fixtures in `test/fixtures/`
- HTTP mocking with Nock
- Tests that start a local SPARQL endpoint (`@lde/local-sparql-endpoint`) must use unique ports across packages to avoid conflicts when Nx runs tests in parallel. Current port allocations: `dataset-registry-client` (3002), `pipeline` sparqlQuery (3001), `pipeline` executor (3003), `pipeline` provenance store (3004), `pipeline-void` namespace-normalization (3005), `search-pipeline` extraction round-trip (3007), `search-pipeline` searchIndexerPipeline end-to-end (3008), `search-pipeline` registry-sourced extraction (3009), `search-pipeline` joins end-to-end (3010), `search-pipeline` keyed roots (3011), `search-pipeline` nested fan-out (3012), `pipeline-void` per-property partitions (3013), `pipeline-void` subject URI spaces (3014), `pipeline-void` distinct counts (3016), `change-detection` content hash reader (3017), `sparql-qlever` remote mode (3015 for QLever, 7070 for the task agent)
- Tests that start a local SPARQL endpoint (`@lde/local-sparql-endpoint`) must use unique ports across packages to avoid conflicts when Nx runs tests in parallel. Current port allocations: `dataset-registry-client` (3002), `pipeline` sparqlQuery (3001), `pipeline` executor (3003), `pipeline` provenance store (3004), `pipeline-void` namespace-normalization (3005), `search-pipeline` extraction round-trip (3007), `search-pipeline` searchIndexerPipeline end-to-end (3008), `search-pipeline` registry-sourced extraction (3009), `search-pipeline` joins end-to-end (3010), `search-pipeline` keyed roots (3011), `search-pipeline` nested fan-out (3012), `pipeline-void` per-property partitions (3013), `pipeline-void` subject URI spaces (3014), `pipeline-void` distinct counts (3016), `change-detection` content hash reader (3017), `search-pipeline` referents’ label transforms (3019), `search-pipeline` lookup presence (3020), `sparql-qlever` remote mode (3015 for QLever, 7070 for the task agent)

### Key Dependencies

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,9 @@ Accepted
Supersedes the last consequence of
[ADR 36](0036-facet-a-root-document-on-its-entries-leaves.md) for a Root
Type’s own field. Extended to an inline entry’s leaf by
[ADR 41](0041-filter-and-facet-a-root-document-on-whether-an-entrys-leaf-has-a-value.md).
[ADR 41](0041-filter-and-facet-a-root-document-on-whether-an-entrys-leaf-has-a-value.md);
its refusal of a facetable field lifted by
[ADR 42](0042-count-a-referents-leaf-and-a-fields-facets-in-one-object.md).

## Context

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,9 @@ Date: 2026-11-16
Accepted

Extends [ADR 40](0040-filter-and-facet-a-field-on-whether-it-has-a-value.md)
to a leaf of an inline reference.
to a leaf of an inline reference. Extended to a lookup’s referents, and its
refusals for the facet shape lifted, by
[ADR 42](0042-count-a-referents-leaf-and-a-fields-facets-in-one-object.md).

## Context

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,124 @@
# 42. Count a referent’s leaf, and a field’s facets in one object

Date: 2026-11-16

## Status

Accepted

Extends [ADR 39](0039-apply-a-label-sources-transforms-to-its-referents.md)
and [ADR 41](0041-filter-and-facet-a-root-document-on-whether-an-entrys-leaf-has-a-value.md);
lifts the refusals ADR 36, ADR 40 and ADR 41 made for the facet shape.

## Context

Linked Open Limburg’s map view filters works on whether a place they name has
coordinates. `contentLocation` is a lookup into `Place`, whose coordinates
come from GeoNames enrichment, not from the work’s graph. The root document
stores only the places’ ids, so LOL guesses from each place’s key: a GeoNames
IRI counts as located. When a GeoNames lookup fails, the place is indexed
without coordinates and the work still says it has them.

ADR 41 counts a leaf of an inline reference’s entries, which the root
document stores. A lookup’s referents are other documents, possibly from
other datasets, so reading the leaf off the target’s collection would need a
pass after every dataset, or an order across them.

Separately, the facets object had one key per facetable field, typed as a
bucket list. ADR 36, ADR 40 and ADR 41 therefore refused anything that would
give one key two meanings: presence on a facetable field, leaves on a
facetable reference, a leaf faceted both by value and by presence.

## Decision

- **A lookup declares `presenceLeaves`** naming leaves of its target, as an
inline reference does since ADR 41:
`ref: { strategy: 'lookup', target: 'Place', presenceLeaves: ['latitude'] }`.
One boolean per root document, `contentLocation_latitude_exists`, in
`PhysicalFields.leafExists`. Only on a lookup naming one target: a leaf is
read through one declaration.
- **Counted at index time, in the referring batch.** The extraction reads each
presence leaf one hop from the root, beside the target’s key and label. The
stage passes the referents through the target type’s transforms, as ADR 39
does for labels, and the projection reads the leaf off each referent’s node
as the target declares it. No pass after all datasets: what the root’s graph
states, or a transform can say from the referent’s key, is counted;
anything else is not, and the work counts as holding none – so a failed
lookup gives `false`, not the guess’s `true`. Only a referent named by IRI
counts: a blank node is no id the lookup stores.
- **A referent named by IRI alone is seeded.** ADR 39 hands a transform only
the quads the batch has, so such a referent reached no transform. A referent
a presence leaf is counted for enters its target’s transforms as
`<referent> rdf:type <class>`, and the seed is dropped from what comes
back. A transform describing a node from its key, as LOL’s place transform
does, then supplies the leaf. A lookup into its own type seeds through that
type’s transforms too, though they saw the batch: they never saw a referent
the batch states nothing of.
- **The target declares the leaf, with a `path`.** A leaf is read off the
referent’s node through the target’s declaration, not derived.
- **A plain criterion on the root selects it**, naming the reference and the
leaf: `{ field: 'contentLocation', leaf: 'latitude', exists: true }`. The
value is the root document’s, so it combines with other criteria like any,
and each key of a lookup’s GraphQL `where` is one such criterion, ANDed like
sibling keys – two presence leaves in one `where` are two conditions. A
lookup takes no weld: it has no entries.
- **One facet object per field.** A field counting more than its own values
is served as `‹Type›‹Field›Facet`: its own values in `buckets` and
`totalValues` (a numeric field’s range facet parts instead), capped by the
field’s `limit`; `exists` for its presence; one key per leaf, a plain facet
type for a leaf counted by value only and `‹Type›‹Field›‹Leaf›Facet` for
one whose presence is counted. A field counting only its own values keeps
`IRIFacet`, `ValueFacet` or its range facet type. A leaf is not named
`buckets`, `totalValues` or `exists`.
- **One filter input per reference counting leaves.** `‹Type›‹Field›Filter`
holds the reference’s `in` and a `where`, `‹Type›‹Field›Where`, naming its
filterable entry leaves and its presence leaves. Two references nesting one
Reference Type may then declare different leaves, which ADR 41 refused.
Adding leaves to a reference therefore changes its filter type, from the
shared `‹Edge›Filter` or `‹Target›Filter` to its own – a breaking change
for a client whose variables name the old type.

## Consequences

- LOL declares `presenceLeaves: ['latitude']` on `contentLocation` and
`locationCreated`, and drops its key-based guess.
- Lifted: presence on a facetable field (ADR 40), leaves on a facetable
reference (ADR 36, ADR 41), a leaf faceted by value and by presence (ADR
41), and one set of presence leaves per Reference Type (ADR 41).
- Breaking at the GraphQL surface: ADR 36’s `‹Type›‹Reference›Facets` and ADR
40 and 41’s `…Facets` objects are renamed `…Facet`; a field that gains a
leaf or `exists` changes from `IRIFacet`/`ValueFacet` to its own object; a
reference counting leaves takes `‹Type›‹Field›Filter` instead of
`‹Edge›Filter` or `‹Target›Filter`, so the refined discovery strategy no
longer reaches it – and a reference that gains its first leaf later breaks
the variables typed with its old filter.
- Each part of a field’s facet object is charged as a facet key, so selecting
`buckets` with `totalValues` costs two where a plain facet costs one – an
overestimate, never a part that costs nothing.
- A target’s transform now also receives nodes carrying a type alone, once per
referring batch, and pays for any I/O it does there unless it caches.
- A value is a copy, changed only when the referring document is reindexed.
- Refused for now: presence leaves on a lookup naming several targets, on a
`joinable` lookup, whose `where` already joins, and on a lookup reading its
literals as periods. A lookup’s leaves faceted by value have no consumer
and are not offered.

## Alternatives considered

- **A pass after all datasets, from the collection being built**, as the
issue first proposed. Complete for referents described only in another
dataset’s graph, but a pass over every referring document per run, skipped
datasets included, outside the batch.
- **A Typesense join facet** (`facet_by=$places(latitude)`). On 30.2 it counted
the joined documents, not the roots, and returned nothing on a text query
without a filter.
- **A weld of one `exists` condition**, as for an inline entry:
`{ field: 'contentLocation', entry: [{ field: 'latitude', exists }] }`.
It pretends a lookup has entries, needs a special case wherever a weld is
read, and cannot hold two leaves, which a weld puts on one entry.
- **Counting through any of several targets.** A referent of either kind
would be read through its own target’s declaration, at the cost of a leaf
every target must declare alike; no consumer names several targets yet.
- **The reference’s own buckets under `id`**, beside its leaves. Every
facetable reference would change shape, where a field counting only its own
values needs none.
88 changes: 64 additions & 24 deletions docs/reference/search-api-graphql.md
Original file line number Diff line number Diff line change
Expand Up @@ -124,8 +124,14 @@ fails this – accepts. The endpoint path defaults to `/graphql`
| A joined reference (`lookup`), at any depth | 1 per target type |
| A facet key | 1 |
| A range facet’s `all` extent | 1 more |
| Each part of a `‹Type›‹Field›Facet` object | 1 |
| A plain stored field or an inline reference | 0, at any depth |

In a field’s facet object every part selected is charged on its own –
`buckets`, `totalValues`, `exists`, a range facet’s `all`, `matching` and
`buckets`, and each part of a leaf’s – so `buckets` with `totalValues`
costs 2 where the same pair on a plain `IRIFacet` costs 1.

A lookup fetches the referents of a whole page in one request, and the facet
keys of a search share one request, so `perPage` does not enter the cost.
Fragments cost what their contents cost, a selection under `@skip` or
Expand Down Expand Up @@ -280,22 +286,41 @@ role: { in: ["etser"] } } }` asks for _this person in this role_, where the
without the keys of fields declaring presence alone. A type declaring none
keeps one `‹Type›Where` for both.

An inline reference’s
[presence leaves](./search#presence-on-an-entry-s-leaf) take `exists` in its
reference type’s `‹Edge›Where`, typed as a field of the root’s own is
(`ExistsFilter`, or `‹Filter›OrExists` for a `filterable` leaf):
`where: { associatedMedia: { where: { thumbnailUrl: { exists: true } } } }`.
Whether any entry holds a value is recorded per document, so `exists` is
welded to nothing else: setting it beside another key of the same
`‹Edge›Where` is an error naming the rewrite into two conditions. Every
reference nesting one reference type declares the same presence leaves, so
`‹Edge›Where` stays shared; through a join, where no companion is read, a
reference with presence leaves takes `‹Edge›JoinedFilter`, without `exists`.
A reference counting leaves of its referents – an inline reference’s
[faceted](./search#faceting-on-an-entry-s-leaf) or
[presence leaves](./search#presence-on-an-entry-s-leaf), a lookup’s
[presence leaves](./search#on-a-lookup-s-referents) – takes a filter of its
own, **`‹Type›‹Field›Filter`**: `@oneOf` over `in` (the ids it holds, for a
lookup or an inline reference naming an identity) and `where`, typed
**`‹Type›‹Field›Where`**. For an inline reference that `where` names the
entries’ filterable leaves, as the shared `‹Edge›Where` does; for both
kinds it names each presence leaf, taking `exists` – `ExistsFilter`, or
`‹Filter›OrExists` for an inline entry’s filterable leaf:
`where: { contentLocation: { where: { latitude: { exists: true } } } }`.
A lookup’s `where` names its presence leaves alone, and each key is a plain
criterion on the root document, ANDed like sibling keys:
`{ latitude: { exists: true }, longitude: { exists: true } }` is two
conditions, each on whether any referent holds that leaf; inside an `or`
alternative it sets one key, as a joined condition does. In an inline
reference’s `where`, whether any entry holds a value is recorded per
document, so `exists` is welded to nothing else: setting it beside another
key of the same `where` is an error naming the rewrite into two conditions.
Through a join, where no companion is read, an inline reference with
presence leaves takes `‹Type›‹Field›JoinedFilter`, without `exists`, and a
lookup its plain `‹Target›Filter`.

**Adding leaves changes the reference’s filter type.** A reference that
gains faceted or presence leaves moves from the shared `‹Edge›Filter` or
`‹Target›Filter` to its own `‹Type›‹Field›Filter`, so a client’s variables
typed with the old name stop validating – a breaking change to the API,
even where the reference’s `in` arm is unchanged. Like `‹Filter›OrExists`,
the per-reference filter is a type of its own, so the refined discovery
strategy does not reach it.

- **`orderBy`**: `RELEVANCE` plus every `sortable` field, as an enum – field
names SCREAMING_SNAKE_CASEd (`datePosted` → `DATE_POSTED`); `direction`
defaults to `DESC`.
- **Facets**: a keyed object with one field per `facetable` field, typed by
- **Facets**: a keyed object with one key per field anything is counted on, typed by
the field’s declaration:
- a **reference** facet returns an `IRIFacet!` – `buckets: [IRIBucket!]!`,
each a `value` (an `IRI`) + `count` + the resolved data `label`, and
Expand Down Expand Up @@ -348,18 +373,33 @@ role: { in: ["etser"] } } }` asks for _this person in this role_, where the
the engine counted documents for it, so a uniform result set yields one
bucket, not two.

An inline reference’s [faceted leaves](./search#faceting-on-an-entry-s-leaf)
are served one level down, under the reference: `facets { associatedMedia {
license { buckets { value count } } } }`, typed `‹Type›‹Reference›Facets`. A presence
facet is served the same way, under the field it counts: `facets {
dateCreated { exists { value count } } }`, typed `‹Type›‹Field›Facets`
(`WorkDateCreatedFacets`) with `exists: [BooleanBucket!]!`. The object
leaves room for a field’s value buckets beside it, once a field may be both
`facetable` and declare presence. A presence leaf’s facet sits under the
reference, one level further down: `facets { associatedMedia { thumbnailUrl {
exists { value count } } } }`, typed `‹Type›‹Reference›‹Leaf›Facets`
(`WorkAssociatedMediaThumbnailUrlFacets`), beside the reference’s faceted
leaves.
**One object per field.** A field counting more than its own values –
its presence, or a leaf of its referents – is served as an object of its
own, **`‹Type›‹Field›Facet`**, holding everything counted on it:

```graphql
facets {
material(limit: 20) { buckets { value count } totalValues }
dateCreated { exists { value count } }
contentLocation(limit: 20) {
buckets { value count }
totalValues
latitude { exists { value count } }
}
associatedMedia { license(limit: 5) { buckets { value count } } }
}
```

Its own values are `buckets` and `totalValues` (a numeric field’s `all`,
`matching` and `buckets` instead), capped by the field’s `limit`, which caps
nothing else; `exists` counts its presence. Each leaf is a key of its own: a
leaf counted only by value keeps its plain facet type (`license(limit):
IRIFacet!`), and a leaf whose presence is counted is an object again,
**`‹Type›‹Field›‹Leaf›Facet`** (`WorkContentLocationLatitudeFacet`), with
`exists` and – when it is faceted by value too – its `buckets`,
`totalValues` and `limit`. A field counting only its own values keeps
`IRIFacet`, `ValueFacet`, its range facet type or `[BooleanBucket!]!`. A
name a declaration took already is refused with a clear error.

Selecting facet fields IS
the request, and the whole selection is **batched per request**: one
Expand Down
32 changes: 32 additions & 0 deletions docs/reference/search-pipeline.md
Original file line number Diff line number Diff line change
Expand Up @@ -398,6 +398,38 @@ readers:
the stage’s projected documents into the write. A projected document is far
heavier than a quad, so lower it where documents are large.

## Referents through their type’s transforms

A stage passes the referents of its references through each target type’s own
`transforms`, inside the batch, so memory stays bounded by `batchSize`: the
quads the extraction read off each referent go in, under the target’s IR
Aliases, and what the transforms return replaces them. Two kinds of reference
are described this way:

- a `searchable` reference, whose referents’ key and label are read, so it is
searched by the label the target’s transform leaves
([ADR 39](../decisions/0039-apply-a-label-sources-transforms-to-its-referents.md));
- a `lookup` declaring [presence leaves](./search#on-a-lookup-s-referents),
whose referents’ key and leaves are read, so the projection can write whether
any referent holds a leaf.

A referent a presence leaf is counted for, named by IRI, and that the batch
states nothing about – the root’s graph names it by IRI alone – enters its
target’s transforms all the same, as `<referent> rdf:type <class>`, so a
transform that describes a node from its key (an Authority’s coordinates for a
GeoNames place) supplies the leaf. The seed is not kept. A transform on such a
target therefore also receives nodes carrying a type alone, or only a key and
a few leaves, once per referring batch; a transform doing I/O pays for that
unless it caches. A blank-node referent is not seeded: the lookup stores no id
for it, so it is not counted.

A reference to the stage’s own type passes no referent’s quads through again,
since its transforms already saw the batch – but a lookup into its own type
declaring presence leaves still has the referents the batch states nothing
about seeded through them. A target wired with its own `readers` has no
transform to apply
([ADR 42](../decisions/0042-count-a-referents-leaf-and-a-fields-facets-in-one-object.md)).

## Per-component isolation

Each root type is an independent engine run – its own collection, alias and
Expand Down
Loading
Loading