Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 23 additions & 8 deletions docs/reference/search-api-graphql.md
Original file line number Diff line number Diff line change
Expand Up @@ -118,14 +118,15 @@ fails this – accepts. The endpoint path defaults to `/graphql`
asks for, measured with
[graphql-query-complexity](https://github.com/slicknode/graphql-query-complexity):

| Selection | Cost |
| ------------------------------------------- | ----------------- |
| A search field (each alias counts) | 1 |
| A joined reference (`lookup`), at any depth | 1 per target type |
| A facet key | 1 |
| A range facet’s `all` extent | 1 more |
| Each part of a `‹Type›‹Field›Facet` object | 1 |
| A plain stored field or an inline reference | 0, at any depth |
| Selection | Cost |
| ------------------------------------------- | ---------------------- |
| A search field (each alias counts) | 1 |
| A joined reference (`lookup`), at any depth | 1 per target type |
| A facet key | 1 |
| A range facet’s `all` extent | 1 more |
| Each part of a `‹Type›‹Field›Facet` object | 1 |
| A `query` on a reference facet | 1 more per target type |
| A plain stored field or an inline reference | 0, at any depth |

In a field’s facet object every part selected is charged on its own –
`buckets`, `totalValues`, `exists`, a range facet’s `all`, `matching` and
Expand Down Expand Up @@ -341,6 +342,20 @@ role: { in: ["etser"] } } }` asks for _this person in this role_, where the
buckets came back, so a client can show “10 of 195” and knows whether a
larger `limit` would bring more; it may be an estimate on large result
sets, and is `null` when the engine does not report it;

A facet whose values are text someone can type – a plain value facet, or a
reference facet with a label source – also takes a `query` argument, which
searches its values: `creator(query: "rem", limit: 10)` returns the
creators matching “rem”, counted over the current results, highest count
first and capped at `limit`. A reference facet matches its referents by
their own type’s free-text search, so a label or alternate name finds them.
Under a query, `totalValues` is the number of matching values, and `null`
when the engine cannot tell. A selected value that does not match is left
out – the one exception to a facet reporting its selected values – so read
the selection from the facet without a query, which an alias can load in
the same request. Text shorter than three characters is no query. A reference facet without a
label source, and a boolean or range facet, has no `query` argument
([facet queries](./search#searching-a-facet-s-values));
- a numeric field is a [range facet](./search#range-facets) and returns an
object rather than a list: `IntRangeFacet`, `FloatRangeFacet` or
`DateRangeFacet`. Its `all` is the extent over the whole collection,
Expand Down
44 changes: 44 additions & 0 deletions docs/reference/search-typesense.md
Original file line number Diff line number Diff line change
Expand Up @@ -276,6 +276,47 @@ rejects on engine failure.
The pure halves `buildSearchParams` and `parseSearchResponse` are exported for
direct use and testing.

### Facet queries

A query searching a facet’s values ([`facetQueries`](./search#searching-a-facet-s-values))
compiles by what the facet holds.

A **literal** (`keyword`) facet compiles to Typesense’s `facet_query` on the
facet’s own search: Typesense matches the values by prefix, with typo
tolerance, and returns the matching values with the highest counts. A colon
in the text separates words like a space, because Typesense reads one as the
separator between field and text.
Non-matching values cannot crowd them out, and under `facet_query`
`stats.total_values` counts the matching values only, so `totalValues` is
exact.

A **reference** facet’s values are IRIs, which `facet_query` would match
instead of the labels someone types. It takes two steps:

1. Search the collection of each type the reference names with the query,
using that type’s own free-text search – its searchable fields, weights,
typo tolerance and prefix matching – and take at most
`maxFacetQueryCandidates` ids from each. These are the candidates.
2. Count each candidate with a search of its own: the facet’s search – the
query’s documents without the facet’s own filter – plus that candidate
selected, asking for no document. The facet keeps the candidates that
occur, the highest counts first, capped at the limit.

Counting each candidate apart is what keeps the answer exact. A single facet
search over the documents carrying any candidate would count their other
values too, which can outnumber every candidate and crowd it out of the
buckets – a match on two works that also carry twenty prolific creators would
fall out of a top-ten – and fetching all of them instead makes the response
grow with the data. The counts travel in `multi_search` requests of 50, so with the default cap
of 50 a query costs one request of counts per type the reference names, beside
step one’s: on a local index of 19694 works, 50 counts took 13 ms against
10 ms for one facet search. A reference naming two types (`Person`,
`Organization`) can need 100 counts.

`totalValues` is the number of candidates that occur in the documents. It is
exact only when no step-one search filled its cap; when one did, more
referents may match than were found, and `totalValues` is absent.

### Engine options

Beyond `collections`, `TypesenseSearchEngineOptions` carries:
Expand All @@ -297,6 +338,9 @@ Beyond `collections`, `TypesenseSearchEngineOptions` carries:
[facet policy](./search#facet-policy) keeps out of the facet – sent
together in as few `multi_search` requests as Typesense accepts (50
searches each).
- **`maxFacetQueryCandidates`** – how many referents a query on a reference
facet may match, per type the reference names (default 50, at most 250:
Typesense’s `per_page` cap). Each candidate costs a count search. See [facet queries](#facet-queries).
- **`onLabelError`** – called when reference-label resolution fails; the
search then degrades to id-only references (and reference-facet buckets to
unlabelled ones) rather than failing. Omit it and the degradation is
Expand Down
47 changes: 47 additions & 0 deletions docs/reference/search.md
Original file line number Diff line number Diff line change
Expand Up @@ -1153,6 +1153,53 @@ like any other: the `status` facet counts across every status and reports
`facetSearches` from `@lde/search/adapter`
([ADR 29](../decisions/0029-make-skip-own-filter-what-a-facet-means.md)).

#### Searching a facet’s values

A query’s `facetQueries` searches a facet’s values by text: type “rem” in the
`creator` facet and get the creators matching it, counted over the current
results. Where `facetLimits` shows only the values with the highest counts,
this finds the one a person is looking for among thousands.

```ts
{
...query,
facets: ['creator'],
facetQueries: { creator: 'rem' },
facetLimits: { creator: 10 },
}
```

A facet that takes a query is one whose values are text someone can type
(`isQueryableFacet`):

- a **`keyword`** facet, matched on its values’ own text;
- a **reference** facet whose values have a [label source](#field-model),
matched by the free-text search of the types its values belong to – their
labels, alternate names, whatever that type searches, as a search of that
type for the same text would. A label only the target’s collection holds,
supplied there by enrichment, finds the value too.

A reference without a label source holds bare IRIs nobody types, and a
boolean, range or presence facet has no values to search, so `validateQuery`
rejects a query on any of them (`not-queryable`). Text shorter than three
characters, once trimmed (`MIN_FACET_QUERY_LENGTH`), is no query: below that,
prefix and typo matching match next to nothing or nearly everything.

Under a query a facet counts as it does without one – over the documents
every `where` clause except its own leaves – and returns only the values that
match, highest count first, capped at its limit. Its `totalValues` is how many
matching values occur in those documents, absent where the engine cannot tell.
`facetSearches` gives each facet with a query a search of its own.

::: warning A selected value that does not match is left out
Under a query a facet reports only the values matching it, **selected or
not**. This is the one exception to a facet reporting every value its own
clause selects: a selection that does not match the text is absent. The same
facet without a query still reports it, so a selection can always be undone
there – show the selected values from the facet without a query, and the
query’s matches beside them.
:::

A `FacetBucket` always carries its `value` and `count`; what else it carries is
decided by the facet’s kind, and a surface types the bucket accordingly (see
[bucket types](./search-api-graphql#what-it-builds-per-root-type)).
Expand Down
82 changes: 74 additions & 8 deletions packages/search-api-graphql/src/build-schema.ts
Original file line number Diff line number Diff line change
Expand Up @@ -17,13 +17,15 @@ import {
type GraphQLEnumValueConfigMap,
type GraphQLFieldConfig,
type GraphQLArgumentConfig,
type GraphQLFieldExtensions,
type GraphQLInputFieldConfig,
type GraphQLInputType,
type GraphQLOutputType,
type GraphQLResolveInfo,
} from 'graphql';
import {
NESTED_DOCUMENT_TYPE,
facetQueryText,
type Criterion,
type FacetBucket,
type Filter,
Expand Down Expand Up @@ -55,6 +57,7 @@ import {
ID_FIELD,
isAbsoluteIri,
OR_KEY,
isQueryableFacet,
isRangeFacet,
isBinnedRangeFacet,
isoToUnixSeconds,
Expand Down Expand Up @@ -82,7 +85,11 @@ import {
type FacetScope,
} from './facet-batch.js';
import { projectionFor } from './projection.js';
import { costExtensions, operationCost } from './query-cost.js';
import {
costExtensions,
facetQueryCostExtensions,
operationCost,
} from './query-cost.js';

/** Populated per request by the transport; no framework type appears here. */
export interface SearchContext {
Expand Down Expand Up @@ -145,6 +152,12 @@ export interface BuildGraphQLSchemaOptions {

type Source = Record<string, unknown>;

/** What a value facet’s field takes ({@link valueArgumentsFor}). */
interface ValueFacetArguments {
readonly limit?: number | null;
readonly query?: string | null;
}

const nonNullListOf = (type: GraphQLOutputType): GraphQLOutputType =>
new GraphQLNonNull(new GraphQLList(new GraphQLNonNull(type)));

Expand Down Expand Up @@ -1986,12 +1999,18 @@ export function buildGraphQLSchema(
fields,
}),
),
...(limited && { args: { limit: limitArgument } }),
...(limited && {
args: valueArgumentsFor(values as FacetDeclaration),
// Its `buckets` and `totalValues` are charged below; a query adds
// its own requests.
extensions: queryCostFor(values as FacetDeclaration, 0),
}),
// Resolved synchronously, so what its keys load lands in the same batch
// as the sibling facets. A range facet’s keys read `load`.
resolve: (source: Source, args: { limit?: number | null }) => ({
resolve: (source: Source, args: ValueFacetArguments) => ({
loadFacet: source.loadFacet,
limit: limited ? checkedLimit(args.limit) : undefined,
query: limited ? facetQueryText(args.query ?? undefined) : undefined,
...(range && {
load: (scope: FacetScope) =>
(source.loadFacet as FacetLoader)(
Expand Down Expand Up @@ -2036,13 +2055,53 @@ export function buildGraphQLSchema(
type: new GraphQLNonNull(
field.kind === 'reference' ? iriFacet : valueFacet,
),
extensions: facetCost,
args: { limit: limitArgument },
resolve: (source: Source, args: { limit?: number | null }) =>
valueFacetOf(source, facet, checkedLimit(args.limit)),
extensions: queryCostFor(facet, 1),
args: valueArgumentsFor(facet),
resolve: (source: Source, args: ValueFacetArguments) =>
valueFacetOf(
source,
facet,
checkedLimit(args.limit),
facetQueryText(args.query ?? undefined),
),
};
}

/** A value facet’s arguments: `limit`, and `query` on a facet whose values
* are text someone can type – a literal one, or a reference whose values
* have labels ({@link isQueryableFacet}). */
function valueArgumentsFor(
facet: FacetDeclaration,
): Record<string, GraphQLArgumentConfig> {
return {
limit: limitArgument,
...(isQueryableFacet(facet, schema) && {
query: {
type: GraphQLString,
description:
facet.field.kind === 'reference'
? 'Only the values whose referent matches this text – by its label, alternate names or whatever else a search of its type matches – counted as without it and capped at `limit`. A selected value that does not match is left out. Text shorter than 3 characters is no query.'
: 'Only the values matching this text, counted as without it and capped at `limit`. A selected value that does not match is left out. Text shorter than 3 characters is no query.',
},
}),
};
}

/** What a value facet costs on top of `own`: under a query on a reference
* facet, one more per target type – the search for its candidates and the
* request that counts them. */
function queryCostFor(
facet: FacetDeclaration,
own: number,
): GraphQLFieldExtensions<unknown, unknown> {
return facet.field.kind === 'reference' && isQueryableFacet(facet, schema)
? facetQueryCostExtensions(
own,
labelTargetNamesOf(facet.field, schema).length,
)
: costExtensions(own * operationCost);
}

/**
* A value facet’s `buckets` and `totalValues`, read through the loader.
* `IRI`-keyed buckets for a reference, literal keys otherwise: the reference
Expand All @@ -2053,11 +2112,13 @@ export function buildGraphQLSchema(
source: Source,
facet: FacetDeclaration,
limit: number | undefined,
query: string | undefined,
): Promise<{ buckets: readonly FacetBucket[]; totalValues: number | null }> {
const { buckets, totalValues } = await (source.loadFacet as FacetLoader)(
facet.name,
'matching',
limit,
query,
);
return {
// A bucket keyed on something that is not an IRI is not a selection key
Expand Down Expand Up @@ -2087,7 +2148,12 @@ export function buildGraphQLSchema(
facet.field.kind === 'reference' ? iriFacet : valueFacet
).toConfig().fields;
const read = (source: Source) =>
valueFacetOf(source, facet, source.limit as number | undefined);
valueFacetOf(
source,
facet,
source.limit as number | undefined,
source.query as string | undefined,
);
return {
buckets: {
...shape.buckets,
Expand Down
30 changes: 23 additions & 7 deletions packages/search-api-graphql/src/facet-batch.ts
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ export type FacetLoader = (
field: string,
scope: FacetScope,
limit?: number,
query?: string,
) => Promise<Facet>;

/** Which documents a {@link FacetLoader} computes a facet over. */
Expand All @@ -24,9 +25,10 @@ export type FacetScope = 'matching' | 'all';
* listing’s `query` for the `matching` loads, and one over
* `unfilteredQuery` – the deployment’s policy without the caller’s filters –
* for the `all` loads, dispatched as ONE `engine.searchFacets` call. Each
* load’s bucket `limit` travels in the query’s `facetLimits`; only the same
* field loaded twice with different limits (two aliases) needs a second
* query in its scope. The
* load’s bucket `limit` travels in the query’s `facetLimits`, and its text
* `query` in its `facetQueries`; only the same field loaded twice with
* different limits or queries (two aliases) needs a second query in its
* scope. The
* engine counts each facet without its own filter (skip-own-filter) and keeps
* its selected values among the buckets, so the whole sidebar costs one
* engine round-trip.
Expand Down Expand Up @@ -67,6 +69,7 @@ export function createFacetLoader(
}
const queries = groups.map(({ scope, keys: groupKeys }) => {
const limited = groupKeys.filter((key) => key.limit !== undefined);
const queried = groupKeys.filter((key) => key.query !== undefined);
return {
keys: groupKeys,
// `searchFacets` answers every query facet-only, whatever its
Expand All @@ -79,6 +82,11 @@ export function createFacetLoader(
limited.map((key) => [key.field, key.limit as number]),
),
}),
...(queried.length > 0 && {
facetQueries: Object.fromEntries(
queried.map((key) => [key.field, key.query as string]),
),
}),
},
};
});
Expand Down Expand Up @@ -129,20 +137,28 @@ export function createFacetLoader(
},
{ cacheKeyFn: cacheKeyOf },
);
return (field, scope, limit) => loader.load({ field, scope, limit });
return (field, scope, limit, query) =>
loader.load({ field, scope, limit, query });
}

/** One facet load: a field, over the documents a scope names, capped at
* `limit` buckets (absent: the engine’s default). */
* `limit` buckets (absent: the engine’s default) and, with a `query`,
* only the values matching it. */
interface FacetKey {
readonly field: string;
readonly scope: FacetScope;
readonly limit?: number;
readonly query?: string;
}

/** A field name never contains a space, so this cannot conflate two keys. */
/** Every part of the key, so no two loads are conflated. */
function cacheKeyOf(key: FacetKey): string {
return `${key.scope} ${key.limit ?? ''} ${key.field}`;
return JSON.stringify([
key.scope,
key.limit ?? null,
key.query ?? null,
key.field,
]);
}

/** What a facet degrades to: no buckets, and no extent. */
Expand Down
Loading
Loading