diff --git a/docs/reference/search-api-graphql.md b/docs/reference/search-api-graphql.md index 05386c98..e3f72b66 100644 --- a/docs/reference/search-api-graphql.md +++ b/docs/reference/search-api-graphql.md @@ -118,14 +118,15 @@ fails this – accepts. The endpoint path defaults to `/graphql` asks for, measured with [graphql-query-complexity](https://github.com/slicknode/graphql-query-complexity): - | Selection | Cost | - | ------------------------------------------- | ----------------- | - | A search field (each alias counts) | 1 | - | A joined reference (`lookup`), at any depth | 1 per target type | - | A facet key | 1 | - | A range facet’s `all` extent | 1 more | - | Each part of a `‹Type›‹Field›Facet` object | 1 | - | A plain stored field or an inline reference | 0, at any depth | + | Selection | Cost | + | ------------------------------------------- | ---------------------- | + | A search field (each alias counts) | 1 | + | A joined reference (`lookup`), at any depth | 1 per target type | + | A facet key | 1 | + | A range facet’s `all` extent | 1 more | + | Each part of a `‹Type›‹Field›Facet` object | 1 | + | A `query` on a reference facet | 1 more per target type | + | A plain stored field or an inline reference | 0, at any depth | In a field’s facet object every part selected is charged on its own – `buckets`, `totalValues`, `exists`, a range facet’s `all`, `matching` and @@ -341,6 +342,20 @@ role: { in: ["etser"] } } }` asks for _this person in this role_, where the buckets came back, so a client can show “10 of 195” and knows whether a larger `limit` would bring more; it may be an estimate on large result sets, and is `null` when the engine does not report it; + + A facet whose values are text someone can type – a plain value facet, or a + reference facet with a label source – also takes a `query` argument, which + searches its values: `creator(query: "rem", limit: 10)` returns the + creators matching “rem”, counted over the current results, highest count + first and capped at `limit`. A reference facet matches its referents by + their own type’s free-text search, so a label or alternate name finds them. + Under a query, `totalValues` is the number of matching values, and `null` + when the engine cannot tell. A selected value that does not match is left + out – the one exception to a facet reporting its selected values – so read + the selection from the facet without a query, which an alias can load in + the same request. Text shorter than three characters is no query. A reference facet without a + label source, and a boolean or range facet, has no `query` argument + ([facet queries](./search#searching-a-facet-s-values)); - a numeric field is a [range facet](./search#range-facets) and returns an object rather than a list: `IntRangeFacet`, `FloatRangeFacet` or `DateRangeFacet`. Its `all` is the extent over the whole collection, diff --git a/docs/reference/search-typesense.md b/docs/reference/search-typesense.md index 6ba53139..9de2f986 100644 --- a/docs/reference/search-typesense.md +++ b/docs/reference/search-typesense.md @@ -276,6 +276,47 @@ rejects on engine failure. The pure halves `buildSearchParams` and `parseSearchResponse` are exported for direct use and testing. +### Facet queries + +A query searching a facet’s values ([`facetQueries`](./search#searching-a-facet-s-values)) +compiles by what the facet holds. + +A **literal** (`keyword`) facet compiles to Typesense’s `facet_query` on the +facet’s own search: Typesense matches the values by prefix, with typo +tolerance, and returns the matching values with the highest counts. A colon +in the text separates words like a space, because Typesense reads one as the +separator between field and text. +Non-matching values cannot crowd them out, and under `facet_query` +`stats.total_values` counts the matching values only, so `totalValues` is +exact. + +A **reference** facet’s values are IRIs, which `facet_query` would match +instead of the labels someone types. It takes two steps: + +1. Search the collection of each type the reference names with the query, + using that type’s own free-text search – its searchable fields, weights, + typo tolerance and prefix matching – and take at most + `maxFacetQueryCandidates` ids from each. These are the candidates. +2. Count each candidate with a search of its own: the facet’s search – the + query’s documents without the facet’s own filter – plus that candidate + selected, asking for no document. The facet keeps the candidates that + occur, the highest counts first, capped at the limit. + +Counting each candidate apart is what keeps the answer exact. A single facet +search over the documents carrying any candidate would count their other +values too, which can outnumber every candidate and crowd it out of the +buckets – a match on two works that also carry twenty prolific creators would +fall out of a top-ten – and fetching all of them instead makes the response +grow with the data. The counts travel in `multi_search` requests of 50, so with the default cap +of 50 a query costs one request of counts per type the reference names, beside +step one’s: on a local index of 19694 works, 50 counts took 13 ms against +10 ms for one facet search. A reference naming two types (`Person`, +`Organization`) can need 100 counts. + +`totalValues` is the number of candidates that occur in the documents. It is +exact only when no step-one search filled its cap; when one did, more +referents may match than were found, and `totalValues` is absent. + ### Engine options Beyond `collections`, `TypesenseSearchEngineOptions` carries: @@ -297,6 +338,9 @@ Beyond `collections`, `TypesenseSearchEngineOptions` carries: [facet policy](./search#facet-policy) keeps out of the facet – sent together in as few `multi_search` requests as Typesense accepts (50 searches each). +- **`maxFacetQueryCandidates`** – how many referents a query on a reference + facet may match, per type the reference names (default 50, at most 250: + Typesense’s `per_page` cap). Each candidate costs a count search. See [facet queries](#facet-queries). - **`onLabelError`** – called when reference-label resolution fails; the search then degrades to id-only references (and reference-facet buckets to unlabelled ones) rather than failing. Omit it and the degradation is diff --git a/docs/reference/search.md b/docs/reference/search.md index 992a2ba0..a8999838 100644 --- a/docs/reference/search.md +++ b/docs/reference/search.md @@ -1153,6 +1153,53 @@ like any other: the `status` facet counts across every status and reports `facetSearches` from `@lde/search/adapter` ([ADR 29](../decisions/0029-make-skip-own-filter-what-a-facet-means.md)). +#### Searching a facet’s values + +A query’s `facetQueries` searches a facet’s values by text: type “rem” in the +`creator` facet and get the creators matching it, counted over the current +results. Where `facetLimits` shows only the values with the highest counts, +this finds the one a person is looking for among thousands. + +```ts +{ + ...query, + facets: ['creator'], + facetQueries: { creator: 'rem' }, + facetLimits: { creator: 10 }, +} +``` + +A facet that takes a query is one whose values are text someone can type +(`isQueryableFacet`): + +- a **`keyword`** facet, matched on its values’ own text; +- a **reference** facet whose values have a [label source](#field-model), + matched by the free-text search of the types its values belong to – their + labels, alternate names, whatever that type searches, as a search of that + type for the same text would. A label only the target’s collection holds, + supplied there by enrichment, finds the value too. + +A reference without a label source holds bare IRIs nobody types, and a +boolean, range or presence facet has no values to search, so `validateQuery` +rejects a query on any of them (`not-queryable`). Text shorter than three +characters, once trimmed (`MIN_FACET_QUERY_LENGTH`), is no query: below that, +prefix and typo matching match next to nothing or nearly everything. + +Under a query a facet counts as it does without one – over the documents +every `where` clause except its own leaves – and returns only the values that +match, highest count first, capped at its limit. Its `totalValues` is how many +matching values occur in those documents, absent where the engine cannot tell. +`facetSearches` gives each facet with a query a search of its own. + +::: warning A selected value that does not match is left out +Under a query a facet reports only the values matching it, **selected or +not**. This is the one exception to a facet reporting every value its own +clause selects: a selection that does not match the text is absent. The same +facet without a query still reports it, so a selection can always be undone +there – show the selected values from the facet without a query, and the +query’s matches beside them. +::: + A `FacetBucket` always carries its `value` and `count`; what else it carries is decided by the facet’s kind, and a surface types the bucket accordingly (see [bucket types](./search-api-graphql#what-it-builds-per-root-type)). diff --git a/packages/search-api-graphql/src/build-schema.ts b/packages/search-api-graphql/src/build-schema.ts index e6ccd92c..7fc4f87b 100644 --- a/packages/search-api-graphql/src/build-schema.ts +++ b/packages/search-api-graphql/src/build-schema.ts @@ -17,6 +17,7 @@ import { type GraphQLEnumValueConfigMap, type GraphQLFieldConfig, type GraphQLArgumentConfig, + type GraphQLFieldExtensions, type GraphQLInputFieldConfig, type GraphQLInputType, type GraphQLOutputType, @@ -24,6 +25,7 @@ import { } from 'graphql'; import { NESTED_DOCUMENT_TYPE, + facetQueryText, type Criterion, type FacetBucket, type Filter, @@ -55,6 +57,7 @@ import { ID_FIELD, isAbsoluteIri, OR_KEY, + isQueryableFacet, isRangeFacet, isBinnedRangeFacet, isoToUnixSeconds, @@ -82,7 +85,11 @@ import { type FacetScope, } from './facet-batch.js'; import { projectionFor } from './projection.js'; -import { costExtensions, operationCost } from './query-cost.js'; +import { + costExtensions, + facetQueryCostExtensions, + operationCost, +} from './query-cost.js'; /** Populated per request by the transport; no framework type appears here. */ export interface SearchContext { @@ -145,6 +152,12 @@ export interface BuildGraphQLSchemaOptions { type Source = Record; +/** What a value facet’s field takes ({@link valueArgumentsFor}). */ +interface ValueFacetArguments { + readonly limit?: number | null; + readonly query?: string | null; +} + const nonNullListOf = (type: GraphQLOutputType): GraphQLOutputType => new GraphQLNonNull(new GraphQLList(new GraphQLNonNull(type))); @@ -1986,12 +1999,18 @@ export function buildGraphQLSchema( fields, }), ), - ...(limited && { args: { limit: limitArgument } }), + ...(limited && { + args: valueArgumentsFor(values as FacetDeclaration), + // Its `buckets` and `totalValues` are charged below; a query adds + // its own requests. + extensions: queryCostFor(values as FacetDeclaration, 0), + }), // Resolved synchronously, so what its keys load lands in the same batch // as the sibling facets. A range facet’s keys read `load`. - resolve: (source: Source, args: { limit?: number | null }) => ({ + resolve: (source: Source, args: ValueFacetArguments) => ({ loadFacet: source.loadFacet, limit: limited ? checkedLimit(args.limit) : undefined, + query: limited ? facetQueryText(args.query ?? undefined) : undefined, ...(range && { load: (scope: FacetScope) => (source.loadFacet as FacetLoader)( @@ -2036,13 +2055,53 @@ export function buildGraphQLSchema( type: new GraphQLNonNull( field.kind === 'reference' ? iriFacet : valueFacet, ), - extensions: facetCost, - args: { limit: limitArgument }, - resolve: (source: Source, args: { limit?: number | null }) => - valueFacetOf(source, facet, checkedLimit(args.limit)), + extensions: queryCostFor(facet, 1), + args: valueArgumentsFor(facet), + resolve: (source: Source, args: ValueFacetArguments) => + valueFacetOf( + source, + facet, + checkedLimit(args.limit), + facetQueryText(args.query ?? undefined), + ), + }; + } + + /** A value facet’s arguments: `limit`, and `query` on a facet whose values + * are text someone can type – a literal one, or a reference whose values + * have labels ({@link isQueryableFacet}). */ + function valueArgumentsFor( + facet: FacetDeclaration, + ): Record { + return { + limit: limitArgument, + ...(isQueryableFacet(facet, schema) && { + query: { + type: GraphQLString, + description: + facet.field.kind === 'reference' + ? 'Only the values whose referent matches this text – by its label, alternate names or whatever else a search of its type matches – counted as without it and capped at `limit`. A selected value that does not match is left out. Text shorter than 3 characters is no query.' + : 'Only the values matching this text, counted as without it and capped at `limit`. A selected value that does not match is left out. Text shorter than 3 characters is no query.', + }, + }), }; } + /** What a value facet costs on top of `own`: under a query on a reference + * facet, one more per target type – the search for its candidates and the + * request that counts them. */ + function queryCostFor( + facet: FacetDeclaration, + own: number, + ): GraphQLFieldExtensions { + return facet.field.kind === 'reference' && isQueryableFacet(facet, schema) + ? facetQueryCostExtensions( + own, + labelTargetNamesOf(facet.field, schema).length, + ) + : costExtensions(own * operationCost); + } + /** * A value facet’s `buckets` and `totalValues`, read through the loader. * `IRI`-keyed buckets for a reference, literal keys otherwise: the reference @@ -2053,11 +2112,13 @@ export function buildGraphQLSchema( source: Source, facet: FacetDeclaration, limit: number | undefined, + query: string | undefined, ): Promise<{ buckets: readonly FacetBucket[]; totalValues: number | null }> { const { buckets, totalValues } = await (source.loadFacet as FacetLoader)( facet.name, 'matching', limit, + query, ); return { // A bucket keyed on something that is not an IRI is not a selection key @@ -2087,7 +2148,12 @@ export function buildGraphQLSchema( facet.field.kind === 'reference' ? iriFacet : valueFacet ).toConfig().fields; const read = (source: Source) => - valueFacetOf(source, facet, source.limit as number | undefined); + valueFacetOf( + source, + facet, + source.limit as number | undefined, + source.query as string | undefined, + ); return { buckets: { ...shape.buckets, diff --git a/packages/search-api-graphql/src/facet-batch.ts b/packages/search-api-graphql/src/facet-batch.ts index 3d9bd95a..a934fc09 100644 --- a/packages/search-api-graphql/src/facet-batch.ts +++ b/packages/search-api-graphql/src/facet-batch.ts @@ -11,6 +11,7 @@ export type FacetLoader = ( field: string, scope: FacetScope, limit?: number, + query?: string, ) => Promise; /** Which documents a {@link FacetLoader} computes a facet over. */ @@ -24,9 +25,10 @@ export type FacetScope = 'matching' | 'all'; * listing’s `query` for the `matching` loads, and one over * `unfilteredQuery` – the deployment’s policy without the caller’s filters – * for the `all` loads, dispatched as ONE `engine.searchFacets` call. Each - * load’s bucket `limit` travels in the query’s `facetLimits`; only the same - * field loaded twice with different limits (two aliases) needs a second - * query in its scope. The + * load’s bucket `limit` travels in the query’s `facetLimits`, and its text + * `query` in its `facetQueries`; only the same field loaded twice with + * different limits or queries (two aliases) needs a second query in its + * scope. The * engine counts each facet without its own filter (skip-own-filter) and keeps * its selected values among the buckets, so the whole sidebar costs one * engine round-trip. @@ -67,6 +69,7 @@ export function createFacetLoader( } const queries = groups.map(({ scope, keys: groupKeys }) => { const limited = groupKeys.filter((key) => key.limit !== undefined); + const queried = groupKeys.filter((key) => key.query !== undefined); return { keys: groupKeys, // `searchFacets` answers every query facet-only, whatever its @@ -79,6 +82,11 @@ export function createFacetLoader( limited.map((key) => [key.field, key.limit as number]), ), }), + ...(queried.length > 0 && { + facetQueries: Object.fromEntries( + queried.map((key) => [key.field, key.query as string]), + ), + }), }, }; }); @@ -129,20 +137,28 @@ export function createFacetLoader( }, { cacheKeyFn: cacheKeyOf }, ); - return (field, scope, limit) => loader.load({ field, scope, limit }); + return (field, scope, limit, query) => + loader.load({ field, scope, limit, query }); } /** One facet load: a field, over the documents a scope names, capped at - * `limit` buckets (absent: the engine’s default). */ + * `limit` buckets (absent: the engine’s default) and, with a `query`, + * only the values matching it. */ interface FacetKey { readonly field: string; readonly scope: FacetScope; readonly limit?: number; + readonly query?: string; } -/** A field name never contains a space, so this cannot conflate two keys. */ +/** Every part of the key, so no two loads are conflated. */ function cacheKeyOf(key: FacetKey): string { - return `${key.scope} ${key.limit ?? ''} ${key.field}`; + return JSON.stringify([ + key.scope, + key.limit ?? null, + key.query ?? null, + key.field, + ]); } /** What a facet degrades to: no buckets, and no extent. */ diff --git a/packages/search-api-graphql/src/query-cost.ts b/packages/search-api-graphql/src/query-cost.ts index 1c11d160..fc5ad429 100644 --- a/packages/search-api-graphql/src/query-cost.ts +++ b/packages/search-api-graphql/src/query-cost.ts @@ -7,6 +7,7 @@ import { type GraphQLFieldExtensions, } from 'graphql'; import type { Plugin } from 'graphql-yoga'; +import { facetQueryText } from '@lde/search'; import { fieldExtensionsEstimator, getComplexity, @@ -58,7 +59,7 @@ export function queryCostPlugin(maxCost: number): Plugin { setResultAndStopExecution({ errors: [ new GraphQLError( - `Query costs ${cost}; at most ${maxCost} is allowed. Each search, facet calculation and joined reference lookup per target type costs ${operationCost}.`, + `Query costs ${cost}; at most ${maxCost} is allowed. Each search, facet calculation and joined reference lookup per target type costs ${operationCost}, and a query on a reference facet ${operationCost} more per target type.`, ), ], }); @@ -80,6 +81,32 @@ export function costExtensions( return { complexity: cost }; } +/** + * The field extensions of a reference facet that takes a `query`: `own` facet + * calculations, and under a query one more per target type – the search for + * its candidates and the request that counts them. + */ +export function facetQueryCostExtensions( + own: number, + targets: number, +): GraphQLFieldExtensions { + return { + complexity: ({ + args, + childComplexity, + }: { + args: { query?: string | null }; + childComplexity: number; + }) => + childComplexity + + operationCost * + (own + + (facetQueryText(args.query ?? undefined) === undefined + ? 0 + : targets)), + }; +} + /** * `document` without its directives, which validation has limited to `@skip` * and `@include`: the only executable directives the schema has. diff --git a/packages/search-api-graphql/test/__snapshots__/generator-stability.test.ts.snap b/packages/search-api-graphql/test/__snapshots__/generator-stability.test.ts.snap index c2b1328c..6f975a86 100644 --- a/packages/search-api-graphql/test/__snapshots__/generator-stability.test.ts.snap +++ b/packages/search-api-graphql/test/__snapshots__/generator-stability.test.ts.snap @@ -71,6 +71,11 @@ type ThingFacets { At most this many buckets, between 1 and 100: the values with the highest counts. A larger limit returns a superset of a smaller one; there is no next page. Omit it for the deployment’s default. """ limit: Int + + """ + Only the values matching this text, counted as without it and capped at \`limit\`. A selected value that does not match is left out. Text shorter than 3 characters is no query. + """ + query: String ): ValueFacet! creator( """ @@ -95,6 +100,11 @@ type ThingFacets { At most this many buckets, between 1 and 100: the values with the highest counts. A larger limit returns a superset of a smaller one; there is no next page. Omit it for the deployment’s default. """ limit: Int + + """ + Only the values matching this text, counted as without it and capped at \`limit\`. A selected value that does not match is left out. Text shorter than 3 characters is no query. + """ + query: String ): ValueFacet! open: [BooleanBucket!]! } diff --git a/packages/search-api-graphql/test/build-schema.test.ts b/packages/search-api-graphql/test/build-schema.test.ts index 76645b5c..30fbfe1c 100644 --- a/packages/search-api-graphql/test/build-schema.test.ts +++ b/packages/search-api-graphql/test/build-schema.test.ts @@ -814,6 +814,93 @@ describe('buildGraphQLSchema', () => { ]); }); + it('searches a facet’s values with its query, beside its limit, in the one query', async () => { + const { engine, facetBatches } = fakeEngine({ + total: 0, + hits: [], + facets: {}, + }); + const result = await run( + `{ datasets { facets { + keyword(query: "kaart", limit: 5) { totalValues } + publisher(query: "rijks") { totalValues } + status(query: " ") { totalValues } + terminologySource(query: "ka") { totalValues } + } } }`, + { engine, acceptLanguage: ['nl'] }, + ); + expect(result.errors).toBeUndefined(); + expect(facetBatches()).toHaveLength(1); + const [batch] = facetBatches(); + // A blank query is none, and so is one under three characters. + expect( + batch.map((query) => [ + query.facets, + query.facetLimits, + query.facetQueries, + ]), + ).toEqual([ + [ + ['keyword', 'publisher', 'status', 'terminologySource'], + { keyword: 5 }, + { keyword: 'kaart', publisher: 'rijks' }, + ], + ]); + }); + + it('loads one facet under two queries in two queries of its scope', async () => { + const { engine, facetBatches } = fakeEngine({ + total: 0, + hits: [], + facets: {}, + }); + const result = await run( + `{ datasets { facets { + all: keyword { totalValues } + matching: keyword(query: "kaart") { totalValues } + } } }`, + { engine, acceptLanguage: ['nl'] }, + ); + expect(result.errors).toBeUndefined(); + const [batch] = facetBatches(); + expect(batch.map((query) => query.facetQueries)).toEqual([ + undefined, + { keyword: 'kaart' }, + ]); + }); + + it('offers a query only on a facet whose values someone can type', () => { + const work: SearchType = { + name: 'Work', + class: 'https://example.org/Work', + fields: [ + { name: 'keyword', kind: 'keyword', facetable: true }, + { + name: 'publisher', + kind: 'reference', + facetable: true, + ref: { strategy: 'idOnly' }, + labelSource: 'Organization', + }, + { + name: 'class', + kind: 'reference', + facetable: true, + ref: { strategy: 'idOnly' }, + }, + { name: 'iiif', kind: 'boolean', facetable: true }, + ], + }; + const facets = printSchema( + buildGraphQLSchema(searchSchema(work, organization)), + ).match(/type WorkFacets \{[^}]*\}/)?.[0]; + expect(facets).toContain('keyword(\n'); + expect(facets).toMatch(/keyword\([^)]*query: String/); + expect(facets).toMatch(/publisher\([^)]*query: String/); + expect(facets).not.toMatch(/class\([^)]*query/); + expect(facets).toMatch(/iiif: \[BooleanBucket!\]!/); + }); + it('refuses a maxFacetLimit that is not a positive integer', () => { for (const maxFacetLimit of [0, 2.5]) { expect(() => @@ -1244,7 +1331,9 @@ describe('buildGraphQLSchema', () => { expect(sdl).toMatch(/SIZE/); // Facets are a keyed object, one field per facetable field, typed by kind. expect(sdl).toMatch(/type DatasetFacets/); - expect(sdl).toMatch(/keyword\([\s\S]*?limit: Int\s*\): ValueFacet!/); + expect(sdl).toMatch( + /keyword\([\s\S]*?limit: Int[\s\S]*?query: String\s*\): ValueFacet!/, + ); // A numeric facet is a range facet: its extent, plus a bucket per bin // where the field declares bins and per distinct value where it doesn’t. expect(sdl).toMatch(/size: IntBinnedRangeFacet!/); @@ -1274,7 +1363,9 @@ describe('buildGraphQLSchema', () => { // `value`, send it back as `where: { publisher: { in: [...] } }`. Both sides // are `IRI`, so no `String`-to-`IRI` boundary sits in the middle – the same // contract BooleanBucket keeps for `is`. - expect(sdl).toMatch(/publisher\([\s\S]*?limit: Int\s*\): IRIFacet!/); + expect(sdl).toMatch( + /publisher\([\s\S]*?limit: Int[\s\S]*?query: String\s*\): IRIFacet!/, + ); expect(sdl).toMatch(/type IRIBucket \{\s+value: IRI!/); expect(sdl).toMatch(/publisher: OrganizationFilter/); // A literal-keyed facet keeps the literal-keyed bucket. diff --git a/packages/search-api-graphql/test/handler.test.ts b/packages/search-api-graphql/test/handler.test.ts index 6d117989..ce9480f7 100644 --- a/packages/search-api-graphql/test/handler.test.ts +++ b/packages/search-api-graphql/test/handler.test.ts @@ -414,7 +414,7 @@ describe('createSearchGraphQLHandler', () => { expect(response.status).toBe(200); expect(data).toBeUndefined(); expect(errors[0].message).toBe( - 'Query costs 2; at most 1 is allowed. Each search, facet calculation and joined reference lookup per target type costs 1.', + 'Query costs 2; at most 1 is allowed. Each search, facet calculation and joined reference lookup per target type costs 1, and a query on a reference facet 1 more per target type.', ); }); diff --git a/packages/search-api-graphql/test/lookup-presence.test.ts b/packages/search-api-graphql/test/lookup-presence.test.ts index d1efdb42..81fcb052 100644 --- a/packages/search-api-graphql/test/lookup-presence.test.ts +++ b/packages/search-api-graphql/test/lookup-presence.test.ts @@ -135,7 +135,7 @@ describe('one facet object per field', () => { it('gives a field counting more an object holding everything counted on it', () => { expect(printed).toMatch( - /contentLocation\(\s*"""[^"]*"""\s*limit: Int\s*\): WorkContentLocationFacet!/, + /contentLocation\(\s*"""[^"]*"""\s*limit: Int\s*"""[^"]*"""\s*query: String\s*\): WorkContentLocationFacet!/, ); expect(printed).toMatch( /type WorkContentLocationFacet \{[^}]*buckets: \[IRIBucket!\]![^}]*totalValues: Int\s*latitude: WorkContentLocationLatitudeFacet!\s*longitude: WorkContentLocationLongitudeFacet!\s*\}/, @@ -176,6 +176,17 @@ describe('one facet object per field', () => { }); }); + it('searches the field’s own values with its query, beside its limit', async () => { + const { response, faceted } = await run( + `{ works { facets { contentLocation(limit: 5, query: "ven") { totalValues } } } }`, + ); + + expect(response.errors).toBeUndefined(); + expect( + faceted.map((query) => [query.facetLimits, query.facetQueries]), + ).toEqual([[{ contentLocation: 5 }, { contentLocation: 'ven' }]]); + }); + it('refuses a limit out of range', async () => { const { response } = await run( `{ works { facets { contentLocation(limit: 0) { totalValues } } } }`, diff --git a/packages/search-api-graphql/test/query-cost.test.ts b/packages/search-api-graphql/test/query-cost.test.ts index 8807e082..8f6d7bb5 100644 --- a/packages/search-api-graphql/test/query-cost.test.ts +++ b/packages/search-api-graphql/test/query-cost.test.ts @@ -219,6 +219,20 @@ describe('the cost of a query', () => { ).toBe(1 + 2); }); + it('charges a query on a reference facet one more per target type, and a literal one nothing extra', async () => { + expect( + await costOf( + '{ creativeWorks(perPage: 0) { facets { author(query: "rem") { totalValues } genre(query: "kaa") { totalValues } } } }', + ), + ).toBe(1 + (1 + 1) + 1); + // Under three characters a query is none, and costs nothing extra. + expect( + await costOf( + '{ creativeWorks(perPage: 0) { facets { author(query: "re") { totalValues } } } }', + ), + ).toBe(1 + 1); + }); + it('charges one per selected facet key', async () => { expect( await costOf( diff --git a/packages/search-api-graphql/vite.config.ts b/packages/search-api-graphql/vite.config.ts index 453e092b..399b642a 100644 --- a/packages/search-api-graphql/vite.config.ts +++ b/packages/search-api-graphql/vite.config.ts @@ -31,7 +31,7 @@ export default mergeConfig( // from a direct caller, since GraphQL validates the query first. // Re-anchored when facet grouping moved to `@lde/search`, taking its // covered branches with it. - branches: 99.15, + branches: 99.18, statements: 100, }, }, diff --git a/packages/search-typesense/src/query-compiler.ts b/packages/search-typesense/src/query-compiler.ts index b6ec346d..68c774cc 100644 --- a/packages/search-typesense/src/query-compiler.ts +++ b/packages/search-typesense/src/query-compiler.ts @@ -1,6 +1,7 @@ import type { SearchParams } from 'typesense/lib/Typesense/Documents.js'; import { fold } from '@lde/text-normalization'; import { + facetQueryText, type Criterion, type FacetRange, type Filter, @@ -33,6 +34,7 @@ import { nestedReferenceType, resolvePath, searchableFields, + type FacetDeclaration, } from '@lde/search/adapter'; /** One hop of a join path, resolved: the Root Type it reaches and the @@ -156,10 +158,49 @@ export function buildSearchParams( if (maxFacetValues !== undefined) { params.max_facet_values = maxFacetValues; } + // A literal facet’s values are their own text, so Typesense matches them + // itself. A reference facet’s are IRIs nobody types: the engine finds the + // referents matching the query first, then counts those. + const facetQuery = facetQueryOf(query, searchType, options.schema); + if ( + facetQuery !== undefined && + facetQuery.facet.field.kind !== 'reference' + ) { + // Typesense reads a colon as the field separator, so one in the text + // would break the match; it separates words like a space does. + params.facet_query = `${facetQuery.facet.engineField}:${facetQuery.text.replaceAll(':', ' ')}`; + } } return params; } +/** + * The facet a query searches the values of, with the text it searches them + * with: the first of the query’s facets carrying a + * {@link SearchQuery.facetQueries} entry long enough to act on + * (`facetQueryText`), or `undefined` when none does. + * Typesense takes one `facet_query` per search, and through the engine each + * facet query has a search of its own (`facetSearches`), so the first is the + * only one. + */ +export function facetQueryOf( + query: SearchQuery, + searchType: SearchType, + schema: SearchSchema | undefined, +): { readonly facet: FacetDeclaration; readonly text: string } | undefined { + for (const name of query.facets) { + const text = facetQueryText(query.facetQueries?.[name]); + if (text === undefined) { + continue; + } + const facet = facetNamed(searchType, name, schema); + if (facet !== undefined) { + return { facet, text }; + } + } + return undefined; +} + /** * The bucket cap a query’s facets are compiled with (`max_facet_values`), * which Typesense applies to every facet of a search alike: the largest diff --git a/packages/search-typesense/src/search.ts b/packages/search-typesense/src/search.ts index 3bf67932..c4672279 100644 --- a/packages/search-typesense/src/search.ts +++ b/packages/search-typesense/src/search.ts @@ -55,6 +55,7 @@ import { buildSearchParams, escapeFilterValue, facetCapOf, + facetQueryOf, type BuildSearchParamsOptions, } from './query-compiler.js'; import { deriveCollectionName } from './collection-name.js'; @@ -99,6 +100,16 @@ export interface TypesenseSearchEngineOptions< * {@link fetchLabels} behaviour unchanged. */ readonly labelCacheTtlMs?: number; + /** + * How many referents a query on a reference facet may match + * ({@link SearchQuery.facetQueries}), per type the reference names: the + * candidates whose counts the facet then reports. A referent matching + * beyond them is not counted, so a facet whose search filled this cap + * reports no `totalValues` – it cannot tell how many match. Between 1 and + * 250, Typesense’s `per_page` cap; defaults to 50. Each candidate costs a + * count search of its own. + */ + readonly maxFacetQueryCandidates?: number; } /** @@ -173,6 +184,17 @@ export function createTypesenseSearchEngine< }; }); assertNoAccidentalSharing(resolved); + const maxCandidates = + options.maxFacetQueryCandidates ?? DEFAULT_MAX_FACET_QUERY_CANDIDATES; + if ( + !Number.isInteger(maxCandidates) || + maxCandidates < 1 || + maxCandidates > TYPESENSE_MAX_PER_PAGE + ) { + throw new Error( + `maxFacetQueryCandidates must be an integer between 1 and ${TYPESENSE_MAX_PER_PAGE}; got ${maxCandidates}.`, + ); + } const collections = new Map( resolved.map(({ searchType, collection }) => [ searchType.class, @@ -450,15 +472,25 @@ export function createTypesenseSearchEngine< callers: readonly (readonly PlannedSearch[])[], ): Promise<(TypesenseSearchResponse | Error)[]> { const collection = collections.get(searchType.class) as string; - const planned = callers.flat(); - const dispatched = planned.filter(({ query }) => !isUnsatisfiable(query)); - const results = await performInChunks( + const planned = await withCandidates(searchType, callers.flat()); + // A query on a reference facet is answered by counting its candidates, + // never by a facet search of its own. + const counted = await countCandidates( client, - dispatched.map(({ query }) => ({ - collection, - ...buildSearchParams(query, searchType, searchParamsOptions), - })), + collection, + searchType, + planned, + searchParamsOptions, ); + const dispatched = planned.filter( + (search, index) => + !counted.has(index) && answeredWithoutSearch(search) === undefined, + ); + const paramsFor = (search: PlannedSearch) => ({ + collection, + ...buildSearchParams(search.query, searchType, searchParamsOptions), + }); + const results = await performInChunks(client, dispatched.map(paramsFor)); const transportFailure = results.find((result) => result instanceof Error); if (transportFailure !== undefined) { throw transportFailure; @@ -467,9 +499,10 @@ export function createTypesenseSearchEngine< searchType.class, ) as ReadonlyMap; let entry = 0; - const responses = planned.map((search) => { - if (isUnsatisfiable(search.query)) { - return EMPTY_RESPONSE; + const responses = planned.map((search, index) => { + const answered = counted.get(index) ?? answeredWithoutSearch(search); + if (answered !== undefined) { + return answered; } const result = results[entry++] as TypesenseSearchResponse | TypesenseErrorEntry | undefined; @@ -505,6 +538,99 @@ export function createTypesenseSearchEngine< }); } + /** + * Step one of a query on a reference facet: each planned search whose facet + * query is on a reference gets the referents matching it – searched in the + * collection of every type the reference names, with that type’s own + * free-text search, as a search of that type for the same text would – + * at most {@link TypesenseSearchEngineOptions.maxFacetQueryCandidates} + * per type. Every such search travels in one `multi_search` round-trip. + * A failed search fails exactly the planned search it was for. + */ + async function withCandidates( + searchType: RootType, + planned: readonly PlannedSearch[], + ): Promise { + const steps = planned.map((search) => { + const facetQuery = facetQueryOf(search.query, searchType, schema); + if ( + facetQuery === undefined || + facetQuery.facet.field.kind !== 'reference' || + isUnsatisfiable(search.query) + ) { + return undefined; + } + return { + ...facetQuery, + locale: search.query.locale, + // `assertValidQuery` admitted only a facet with label targets, and + // every target is a Root Type with a collection. + targets: labelTargetNamesOf(facetQuery.facet.field, schema).map( + (name) => typesByName.get(name) as RootType, + ), + }; + }); + const searches = steps.flatMap((step) => + step === undefined + ? [] + : step.targets.map((target) => ({ + collection: collections.get(target.class) as string, + ...buildSearchParams( + { + text: step.text, + where: [], + orderBy: [], + limit: maxCandidates, + offset: 0, + facets: [], + locale: step.locale, + }, + target, + searchParamsOptions, + ), + include_fields: 'id', + })), + ); + if (searches.length === 0) { + return [...planned]; + } + const results = await performInChunks(client, searches); + const transportFailure = results.find((result) => result instanceof Error); + if (transportFailure !== undefined) { + throw transportFailure; + } + let next = 0; + return planned.map((search, index) => { + const step = steps[index]; + if (step === undefined) { + return search; + } + const answered = results.slice(next, (next += step.targets.length)) as ( + TypesenseSearchResponse | TypesenseErrorEntry | undefined + )[]; + const iris = new Set(); + let capped = false; + for (const [position, result] of answered.entries()) { + if (result === undefined || 'error' in result) { + const target = step.targets[position] as RootType; + return { + ...search, + candidates: entryError( + `search of “${collections.get(target.class) as string}” for values of facet “${step.facet.name}”`, + result, + ), + }; + } + const hits = result.hits ?? []; + capped ||= result.found > hits.length; + for (const hit of hits) { + iris.add(String(hit.document.id)); + } + } + return { ...search, candidates: { facet: step.facet, iris, capped } }; + }); + } + const engine: TypesenseSearchEngine = { schema, collectionNameFor(searchType: RootType): string { @@ -525,7 +651,12 @@ export function createTypesenseSearchEngine< // its `where`; each own-filtered facet is counted by a search of its own // (`facetSearches`), in the same round-trip. const searches = facetSearches(query); - const shared = searches.find((search) => search.own === undefined); + // A facet query’s search counts its facet alone and may narrow the + // documents it counts, so it never carries the hits. + const shared = searches.find( + (search) => + search.own === undefined && search.query.facetQueries === undefined, + ); const [response] = await answer(searchType, [ [ { @@ -783,6 +914,162 @@ function withDeclaredFacetNames( * counts facets alone or also returns the caller’s hits. */ interface PlannedSearch extends FacetSearch { readonly facetOnly: boolean; + /** For a query on a reference facet, the referents matching it + * (`withCandidates`), or why they could not be found. */ + readonly candidates?: Candidates | Error; +} + +/** The referents a query on a reference facet matches: the only values the + * facet may report. */ +interface Candidates { + readonly facet: FacetDeclaration; + readonly iris: ReadonlySet; + /** Whether a search for them filled its cap, so more may match. */ + readonly capped: boolean; +} + +/** + * The response a planned search has without being sent, or `undefined` when + * it must be sent: nothing for a search asking for no document (its only + * filter would compile away and hand back the whole collection, + * {@link isUnsatisfiable}), the failure of a search whose candidates could + * not be found, and an empty facet for one whose query matched no referent. + */ +function answeredWithoutSearch( + search: PlannedSearch, +): TypesenseSearchResponse | Error | undefined { + if (isUnsatisfiable(search.query)) { + return EMPTY_RESPONSE; + } + const { candidates } = search; + if (candidates instanceof Error) { + return candidates; + } + if (candidates !== undefined && candidates.iris.size === 0) { + return { + found: 0, + facet_counts: [ + { + field_name: candidates.facet.name, + counts: [], + stats: { total_values: 0 }, + }, + ], + }; + } + return undefined; +} + +/** + * Step two of a query on a reference facet: each candidate is counted by a + * search of its own – the facet’s search narrowed to that candidate on the + * field the facet buckets, asking for no document. + * The facet keeps the candidates that occur, the highest counts first and + * capped at its limit, and reports how many occur, unless step one filled its + * cap, when more may match than it found and no total is reported. + * + * Counting each candidate apart, rather than faceting the documents that carry + * any of them, keeps their other values from crowding a candidate out of the + * buckets, and bounds the work by the candidate cap instead of by the data. + * The counts travel together, in as few `multi_search` requests as Typesense + * accepts. A failed count fails exactly the search it was for. + * + * Returns the response per planned search index, for the searches it answers. + */ +async function countCandidates( + client: Pick, + collection: string, + searchType: RootType, + planned: readonly PlannedSearch[], + options: BuildSearchParamsOptions, +): Promise> { + const toCount = planned.flatMap((search, entry) => { + const { candidates } = search; + // `withCandidates` sets none on a search asking for no document. + return candidates === undefined || + candidates instanceof Error || + candidates.iris.size === 0 + ? [] + : [...candidates.iris].map((iri) => ({ entry, candidates, iri })); + }); + const results = await performInChunks( + client, + toCount.map(({ entry, candidates, iri }) => { + // The facet query is answered here; the count carries none. + const { facetQueries: _, ...query } = (planned[entry] as PlannedSearch) + .query; + const params = buildSearchParams( + { ...query, facets: [], orderBy: [], limit: 0, offset: 0 }, + searchType, + options, + ); + // On the field the facet buckets, not on the field itself: under a facet + // policy only that one leaves out the values the policy keeps out of the + // facet, which a query must not bring back. + const candidate = `${candidates.facet.engineField}:=${escapeFilterValue(iri)}`; + return { + collection, + ...params, + filter_by: + params.filter_by === undefined + ? candidate + : `(${params.filter_by}) && ${candidate}`, + }; + }), + ); + const counts = new Map(); + toCount.forEach(({ entry, candidates, iri }, index) => { + const sofar = counts.get(entry) ?? []; + if (sofar instanceof Error) { + return; + } + const result = results[index]; + if (result instanceof Error) { + counts.set(entry, result); + } else if (result === undefined || 'error' in result) { + counts.set( + entry, + entryError( + `count of facet value “${iri}” on “${candidates.facet.name}”`, + result, + ), + ); + } else { + counts.set(entry, [...sofar, { value: iri, count: result.found }]); + } + }); + const answered = new Map(); + for (const [entry, found] of counts) { + if (found instanceof Error) { + answered.set(entry, found); + continue; + } + const search = planned[entry] as PlannedSearch; + const candidates = search.candidates as Candidates; + const occurring = found.filter(({ count }) => count > 0).sort(byCount); + answered.set(entry, { + found: 0, + facet_counts: [ + { + field_name: candidates.facet.name, + counts: occurring.slice(0, bucketCapOf(search.query, options)), + ...(candidates.capped + ? {} + : { stats: { total_values: occurring.length } }), + }, + ], + }); + } + return answered; +} + +/** The buckets a facet search returns at most: its limit, or Typesense’s + * own default. */ +function bucketCapOf( + query: SearchQuery, + options: BuildSearchParamsOptions, +): number { + return facetCapOf(query, options) ?? TYPESENSE_DEFAULT_MAX_FACET_VALUES; } /** A planned facet search as sent: no hits, and with no hits no ordering. */ @@ -858,6 +1145,13 @@ async function performInChunks( * document, which is never dispatched. */ const EMPTY_RESPONSE: TypesenseSearchResponse = { found: 0, hits: [] }; +/** The referents a query on a reference facet matches at most, per type, + * unless a deployment says otherwise. */ +const DEFAULT_MAX_FACET_QUERY_CANDIDATES = 50; + +/** The most hits Typesense returns for one search (`per_page`). */ +const TYPESENSE_MAX_PER_PAGE = 250; + /** The bucket cap Typesense applies when `max_facet_values` is unset. */ const TYPESENSE_DEFAULT_MAX_FACET_VALUES = 10; diff --git a/packages/search-typesense/test/facet-policy.test.ts b/packages/search-typesense/test/facet-policy.test.ts index 6fdefac7..8dfd8725 100644 --- a/packages/search-typesense/test/facet-policy.test.ts +++ b/packages/search-typesense/test/facet-policy.test.ts @@ -381,6 +381,32 @@ describe('facet policy end to end (integration)', () => { ]); }); + it('leaves a value the policy keeps out of the facet out of a facet query too', async () => { + // Every Kessel matches “kes” in the place collection, and works carry + // them, but the policy admits none of them as a bucket. + const [kessel, venlo] = await engine.searchFacets(work, [ + { + ...baseQuery, + limit: 0, + facets: ['locationCreated'], + facetQueries: { locationCreated: 'kes' }, + }, + { + ...baseQuery, + limit: 0, + facets: ['locationCreated'], + facetQueries: { locationCreated: 'ven' }, + }, + ]); + if ('error' in kessel || 'error' in venlo) { + throw new Error('Expected facets outcomes.'); + } + expect(kessel.facets.locationCreated?.buckets).toEqual([]); + expect(venlo.facets.locationCreated?.buckets).toEqual([ + { value: VENLO, count: 1, label: { und: ['Venlo'] } }, + ]); + }); + it('keeps the field whole: a local place still displays and still filters exactly', async () => { const result = await engine.search(work, { ...baseQuery, diff --git a/packages/search-typesense/test/facet-query.integration.test.ts b/packages/search-typesense/test/facet-query.integration.test.ts new file mode 100644 index 00000000..2b57ca18 --- /dev/null +++ b/packages/search-typesense/test/facet-query.integration.test.ts @@ -0,0 +1,392 @@ +import { afterAll, beforeAll, describe, expect, it } from 'vitest'; +import type { Client } from 'typesense'; +import { + defineSearchType, + searchSchema, + type Facet, + type FacetsOutcome, + type SearchQuery, + type SearchType, + type TextField, +} from '@lde/search'; +import { physicalFields } from '@lde/search/adapter'; +import { buildCollectionDefinition } from '../src/collection-definition.js'; +import { createTypesenseSearchEngine } from '../src/search.js'; +import { TypesenseContainer } from './typesense-container.js'; + +const SCHEMA_ORG = 'https://schema.org/'; + +const person = defineSearchType({ + name: 'Person', + class: `${SCHEMA_ORG}Person`, + fields: [ + { + name: 'label', + kind: 'text', + path: `${SCHEMA_ORG}name`, + locales: ['und'], + output: true, + searchable: { weight: 2 }, + }, + { + // Searched, never shown: an alternate name only the people collection + // holds, as enrichment supplies it. + name: 'alternateName', + kind: 'text', + path: `${SCHEMA_ORG}alternateName`, + locales: ['und'], + searchable: { weight: 1 }, + }, + ], +}); + +const organization = defineSearchType({ + name: 'Organization', + class: `${SCHEMA_ORG}Organization`, + fields: [ + { + name: 'label', + kind: 'text', + path: `${SCHEMA_ORG}name`, + locales: ['und'], + output: true, + searchable: { weight: 1 }, + }, + ], +}); + +const work = defineSearchType({ + name: 'Work', + class: `${SCHEMA_ORG}CreativeWork`, + fields: [ + { + name: 'keyword', + kind: 'keyword', + path: `${SCHEMA_ORG}keywords`, + array: true, + output: true, + filterable: true, + facetable: true, + }, + { + name: 'creator', + kind: 'reference', + path: `${SCHEMA_ORG}creator`, + array: true, + output: true, + filterable: true, + facetable: true, + ref: { strategy: 'idOnly' }, + labelSource: ['Person', 'Organization'], + }, + { + name: 'genre', + kind: 'reference', + path: `${SCHEMA_ORG}genre`, + array: true, + output: true, + filterable: true, + facetable: true, + ref: { strategy: 'idOnly' }, + }, + ], +}); + +const rembrandt = 'https://ex/rembrandt'; +const rembert = 'https://ex/rembert'; +const harmen = 'https://ex/harmen'; +const remonstrants = 'https://ex/remonstrants'; +// Twenty people who share every work Rembert made, and have made more. +const others = Array.from({ length: 20 }, (_, index) => `https://ex/p${index}`); + +/** + * A facet’s values searched by text (`facetQueries`), end to end through a + * real Typesense: a literal facet through `facet_query`, a reference facet in + * two steps – the referents matching the text in their own collections, then + * the facet counted over those. + */ +describe('a facet query', () => { + const container = new TypesenseContainer(); + let client: Client; + const schema = searchSchema(work, person, organization); + const collections = { + Work: 'works', + Person: 'people', + Organization: 'organizations', + }; + let engine: ReturnType; + const base: SearchQuery = { + where: [], + orderBy: [], + limit: 0, + offset: 0, + facets: [], + locale: 'und', + }; + const facetOf = (outcome: FacetsOutcome | undefined, name: string): Facet => { + if (outcome === undefined || 'error' in outcome) { + throw new Error(`Expected facets, got ${String(outcome)}`); + } + return outcome.facets[name] as Facet; + }; + const counts = (facet: Facet) => + facet.buckets.map((bucket) => [bucket.value, bucket.count]); + const searchName = (type: SearchType, name: string) => + physicalFields( + type.fields.find((field) => field.name === name) as TextField, + ).search[0] as string; + const named = ( + type: typeof person | typeof organization, + id: string, + label: string, + alternateName?: string, + ) => ({ + id, + label_und: label, + [searchName(type, 'label')]: label.toLowerCase(), + ...(alternateName !== undefined + ? { [searchName(person, 'alternateName')]: alternateName.toLowerCase() } + : {}), + }); + + beforeAll(async () => { + client = await container.start(); + for (const [type, name] of [ + [work, 'works'], + [person, 'people'], + [organization, 'organizations'], + ] as const) { + await client.collections().create( + buildCollectionDefinition(type, { + collectionNameFor: () => name, + schema, + }), + ); + } + const imports = [ + await client + .collections('people') + .documents() + .import( + [ + named(person, rembrandt, 'Rembrandt van Rijn'), + named(person, rembert, 'Rembert Dodoens'), + // Found by “rem” through the alternate name alone. + named(person, harmen, 'Harmen Gerritsz', 'Remmert'), + ...others.map((id, index) => named(person, id, `Painter ${index}`)), + ], + { action: 'create' }, + ), + await client + .collections('organizations') + .documents() + .import( + [named(organization, remonstrants, 'Remonstrantse Broederschap')], + { + action: 'create', + }, + ), + await client + .collections('works') + .documents() + .import( + [ + ...[1, 2, 3].map((index) => ({ + id: `https://ex/painting-${index}`, + keyword: ['schilderij'], + creator: [rembrandt], + genre: ['https://ex/portrait'], + })), + ...[1, 2].map((index) => ({ + id: `https://ex/herbal-${index}`, + keyword: + index === 1 + ? ['boek', 'kruid', 'kruidenboek'] + : ['boek', 'kruid'], + creator: [rembert, ...others], + })), + ...[1, 2, 3].map((index) => ({ + id: `https://ex/other-${index}`, + keyword: ['boek'], + creator: others, + })), + { + id: 'https://ex/pamphlet', + keyword: ['pamflet'], + creator: [remonstrants], + }, + { + id: 'https://ex/drawing', + keyword: ['schilderij'], + creator: [harmen, ...others], + }, + ], + { action: 'create' }, + ), + ]; + expect(imports.flat().every((result) => result.success)).toBe(true); + engine = createTypesenseSearchEngine(client, schema, { collections }); + }, 120_000); + + afterAll(async () => { + await container.stop(); + }); + + it('returns only the literal values matching it, by count, capped at the limit', async () => { + const [outcome] = await engine.searchFacets(work, [ + { + ...base, + facets: ['keyword'], + facetQueries: { keyword: 'krui' }, + facetLimits: { keyword: 1 }, + }, + ]); + const facet = facetOf(outcome, 'keyword'); + + expect(counts(facet)).toEqual([['kruid', 2]]); + // `stats.total_values` counts the values matching the query, not all. + expect(facet.totalValues).toBe(2); + }); + + it('finds a referent by its label, an alternate name and in every target type', async () => { + const [outcome] = await engine.searchFacets(work, [ + { ...base, facets: ['creator'], facetQueries: { creator: 'rem' } }, + ]); + const facet = facetOf(outcome, 'creator'); + + expect(counts(facet)).toEqual([ + [rembrandt, 3], + [rembert, 2], + // Equal counts come in the engine’s order. + ...counts(facet).slice(2), + ]); + expect( + new Set( + counts(facet) + .slice(2) + .map(([value]) => value), + ), + ).toEqual(new Set([harmen, remonstrants])); + expect(facet.totalValues).toBe(4); + expect(facet.buckets[0]?.label).toEqual({ und: ['Rembrandt van Rijn'] }); + }); + + it('keeps a match that co-occurs with many values that outnumber it', async () => { + // Rembert’s two works and Harmen’s drawing carry twenty other people, so + // over the works carrying a match each of them counts 3: more than Rembert + // and Harmen. They must not crowd either out of the buckets. + const [outcome] = await engine.searchFacets(work, [ + { + ...base, + facets: ['creator'], + facetQueries: { creator: 'rem' }, + facetLimits: { creator: 2 }, + }, + ]); + const facet = facetOf(outcome, 'creator'); + + expect(counts(facet)).toEqual([ + [rembrandt, 3], + [rembert, 2], + ]); + expect(facet.totalValues).toBe(4); + }); + + it('counts under the other filters, without its own', async () => { + const [outcome] = await engine.searchFacets(work, [ + { + ...base, + where: [ + { or: [{ field: 'keyword', in: ['boek'] }] }, + { or: [{ field: 'creator', in: [others[0] as string] }] }, + ], + facets: ['creator'], + facetQueries: { creator: 'rem' }, + }, + ]); + const facet = facetOf(outcome, 'creator'); + + // Only books: Rembert’s herbals. The selected Painter 0 does not match + // “rem”, so it does not come back. + expect(counts(facet)).toEqual([[rembert, 2]]); + expect(facet.totalValues).toBe(1); + }); + + it('leaves out a selected value that does not match, which the facet without a query keeps', async () => { + const selected: SearchQuery = { + ...base, + where: [ + { or: [{ field: 'keyword', in: ['pamflet'] }] }, + { or: [{ field: 'creator', in: [others[0] as string] }] }, + ], + facets: ['keyword', 'creator'], + }; + const [queried, unqueried] = await engine.searchFacets(work, [ + { + ...selected, + facetQueries: { keyword: 'kruidenboek', creator: 'rembrandt' }, + }, + selected, + ]); + + // No work is a pamphlet by Painter 0. Without a query both selections + // come back at 0, to undo them; under one only matching values do. + expect(counts(facetOf(queried, 'keyword'))).toEqual([['kruidenboek', 1]]); + expect(counts(facetOf(queried, 'creator'))).toEqual([]); + expect(counts(facetOf(unqueried, 'keyword'))).toContainEqual([ + 'pamflet', + 0, + ]); + expect(counts(facetOf(unqueried, 'creator'))).toContainEqual([ + others[0], + 0, + ]); + }); + + it('reports no total when the referent search filled its cap', async () => { + const capped = createTypesenseSearchEngine(client, schema, { + collections, + maxFacetQueryCandidates: 1, + }); + const [outcome] = await capped.searchFacets(work, [ + { ...base, facets: ['creator'], facetQueries: { creator: 'rem' } }, + ]); + const facet = facetOf(outcome, 'creator'); + + // One person and one organization: the best match of each. + expect(facet.buckets.length).toBeLessThanOrEqual(2); + expect(facet.totalValues).toBeUndefined(); + }); + + it('answers a query no referent matches with an empty facet', async () => { + const [outcome] = await engine.searchFacets(work, [ + { ...base, facets: ['creator'], facetQueries: { creator: 'xqzzv' } }, + ]); + + expect(facetOf(outcome, 'creator')).toEqual({ + buckets: [], + totalValues: 0, + }); + }); + + it('leaves the hits of a search alone', async () => { + const result = await engine.search(work, { + ...base, + limit: 20, + facets: ['creator', 'keyword'], + facetQueries: { creator: 'dodoens' }, + }); + + expect(result.total).toBe(10); + expect(counts(result.facets.creator as Facet)).toEqual([[rembert, 2]]); + expect(counts(result.facets.keyword as Facet)).toContainEqual(['boek', 5]); + }); + + it('rejects a query on a reference facet without a label source', async () => { + await expect( + engine.searchFacets(work, [ + { ...base, facets: ['genre'], facetQueries: { genre: 'portrait' } }, + ]), + ).rejects.toThrow(/not-queryable/); + }); +}); diff --git a/packages/search-typesense/test/facet-query.test.ts b/packages/search-typesense/test/facet-query.test.ts new file mode 100644 index 00000000..99d52914 --- /dev/null +++ b/packages/search-typesense/test/facet-query.test.ts @@ -0,0 +1,240 @@ +import { describe, expect, it } from 'vitest'; +import { searchSchema, type SearchQuery, type SearchType } from '@lde/search'; +import { createTypesenseSearchEngine } from '../src/search.js'; +import { fakeTypesenseClient } from './fake-typesense-client.js'; + +// The failure paths of a query on a reference facet, which a live Typesense +// does not produce on demand. The behaviour itself is pinned against a real +// Typesense in facet-query.integration.test.ts. + +const person: SearchType = { + name: 'Person', + class: 'https://schema.org/Person', + fields: [ + { + name: 'label', + kind: 'text', + locales: ['und'], + output: true, + searchable: { weight: 1 }, + }, + ], +}; + +const work: SearchType = { + name: 'Work', + class: 'https://schema.org/CreativeWork', + fields: [ + { + name: 'creator', + kind: 'reference', + array: true, + facetable: true, + ref: { strategy: 'idOnly' }, + labelSource: 'Person', + }, + ], +}; + +const schema = searchSchema(person, work); +const collections = { Person: 'people', Work: 'works' }; + +const query: SearchQuery = { + where: [], + orderBy: [], + limit: 0, + offset: 0, + facets: ['creator'], + facetQueries: { creator: 'rem' }, + facetLimits: { creator: 1 }, + locale: 'und', +}; + +/** Step one’s search: the referents, in the target’s collection. */ +const isCandidateSearch = (search: Record) => + search.collection === 'people' && search.include_fields === 'id'; + +/** A step-two search: one candidate counted, no facet and no hits. */ +const isCountSearch = (search: Record) => + search.collection === 'works' && + search.facet_by === undefined && + search.per_page === 0; + +/** The works each referent occurs in. */ +const occurrences: Record = { + 'https://ex/rembrandt': 5, + 'https://ex/remigius': 2, + 'https://ex/remco': 0, +}; + +/** Answers step one with three referents, and each count with how many works + * carry that referent. */ +function answer(search: Record): Record { + if (isCandidateSearch(search)) { + return { + found: 3, + hits: Object.keys(occurrences) + .slice(0, Number(search.per_page)) + .map((id) => ({ document: { id } })), + }; + } + if (isCountSearch(search)) { + const iri = Object.keys(occurrences).find((candidate) => + String(search.filter_by).includes(candidate), + ) as string; + return { found: occurrences[iri], hits: [] }; + } + // A label lookup. + return { found: 0, hits: [] }; +} + +describe('a query on a reference facet', () => { + it('refuses a candidate cap Typesense cannot return', () => { + for (const maxFacetQueryCandidates of [0, 251, 1.5]) { + expect(() => + createTypesenseSearchEngine(fakeTypesenseClient().client, schema, { + collections, + maxFacetQueryCandidates, + }), + ).toThrow(/maxFacetQueryCandidates must be an integer between 1 and 250/); + } + }); + + it('fails exactly its own query when the referent search fails', async () => { + const { client } = fakeTypesenseClient({ + multiSearch: (search) => + isCandidateSearch(search) + ? { error: 'Not found', code: 404 } + : answer(search), + }); + const engine = createTypesenseSearchEngine(client, schema, { collections }); + + const [queried, unqueried] = await engine.searchFacets(work, [ + query, + { ...query, facetQueries: undefined }, + ]); + + expect(queried).toEqual({ + error: new Error( + 'Typesense search of “people” for values of facet “creator” failed (404): Not found', + ), + }); + expect(unqueried).not.toHaveProperty('error'); + }); + + it('rejects when the referent search cannot be sent at all', async () => { + const { client } = fakeTypesenseClient({ + multiSearch: (search) => { + if (isCandidateSearch(search)) { + throw new Error('Connection refused'); + } + return answer(search); + }, + }); + const engine = createTypesenseSearchEngine(client, schema, { collections }); + + await expect(engine.searchFacets(work, [query])).rejects.toThrow( + 'Connection refused', + ); + }); + + it('reports the total when step one returned exactly its cap', async () => { + const { client } = fakeTypesenseClient({ multiSearch: answer }); + const engine = createTypesenseSearchEngine(client, schema, { + collections, + maxFacetQueryCandidates: 3, + }); + + const [outcome] = await engine.searchFacets(work, [query]); + + expect(outcome).toEqual({ + facets: { + creator: { + buckets: [{ value: 'https://ex/rembrandt', count: 5 }], + totalValues: 2, + }, + }, + }); + }); + + it('counts each candidate by a search of its own, and keeps those that occur by count', async () => { + const { client, performs } = fakeTypesenseClient({ multiSearch: answer }); + const engine = createTypesenseSearchEngine(client, schema, { collections }); + + const [outcome] = await engine.searchFacets(work, [query]); + + // Capped at the limit of 1; Remco occurs nowhere and does not count. + expect(outcome).toEqual({ + facets: { + creator: { + buckets: [{ value: 'https://ex/rembrandt', count: 5 }], + totalValues: 2, + }, + }, + }); + const sent = performs.flat(); + expect(sent.filter(isCountSearch)).toHaveLength(3); + // No facet search over the works: the counts answer the facet. + expect( + sent.some( + (search) => search.collection === 'works' && 'facet_by' in search, + ), + ).toBe(false); + }); + + it('reports no total when step one left referents out', async () => { + const { client } = fakeTypesenseClient({ multiSearch: answer }); + const engine = createTypesenseSearchEngine(client, schema, { + collections, + maxFacetQueryCandidates: 2, + }); + + const [outcome] = await engine.searchFacets(work, [query]); + + // Three referents match and two were counted: the total is unknown. + expect(outcome).toEqual({ + facets: { + creator: { buckets: [{ value: 'https://ex/rembrandt', count: 5 }] }, + }, + }); + }); + + it('fails exactly its own query when a count fails', async () => { + const { client } = fakeTypesenseClient({ + multiSearch: (search) => + isCountSearch(search) && + String(search.filter_by).includes('https://ex/remigius') + ? { error: 'Timeout' } + : answer(search), + }); + const engine = createTypesenseSearchEngine(client, schema, { collections }); + + const [queried, unqueried] = await engine.searchFacets(work, [ + query, + { ...query, facetQueries: undefined }, + ]); + + expect(queried).toEqual({ + error: new Error( + 'Typesense count of facet value “https://ex/remigius” on “creator” failed: Timeout', + ), + }); + expect(unqueried).not.toHaveProperty('error'); + }); + + it('fails exactly its own query when a count cannot be sent', async () => { + const { client } = fakeTypesenseClient({ + multiSearch: (search) => { + if (isCountSearch(search)) { + throw new Error('Connection reset'); + } + return answer(search); + }, + }); + const engine = createTypesenseSearchEngine(client, schema, { collections }); + + const [outcome] = await engine.searchFacets(work, [query]); + + expect(outcome).toEqual({ error: new Error('Connection reset') }); + }); +}); diff --git a/packages/search-typesense/test/identity-facet.integration.test.ts b/packages/search-typesense/test/identity-facet.integration.test.ts index b40d19d8..2cd5f145 100644 --- a/packages/search-typesense/test/identity-facet.integration.test.ts +++ b/packages/search-typesense/test/identity-facet.integration.test.ts @@ -81,6 +81,7 @@ const work = defineSearchType({ describe('an identity faceted on its nested leaf', () => { const container = new TypesenseContainer(); let engine: SearchEngine; + let client: Client; const schema = searchSchema(work, person, creatorRole); const base: SearchQuery = { where: [], @@ -108,7 +109,7 @@ describe('an identity faceted on its nested leaf', () => { }); beforeAll(async () => { - const client: Client = await container.start(); + client = await container.start(); const definition = buildCollectionDefinition(work, { collectionNameFor: () => 'works', schema, @@ -207,4 +208,34 @@ describe('an identity faceted on its nested leaf', () => { 'https://ex/bram': 1, }); }); + + it('searches the facet’s values by the label of each person', async () => { + const people = await client + .collections('people') + .documents() + .import( + [ + { + id: 'https://ex/anna', + label_und: 'Anna Visscher', + label_search_und: 'anna visscher', + }, + { + id: 'https://ex/bram', + label_und: 'Bram de Bie', + label_search_und: 'bram de bie', + }, + ], + { action: 'create' }, + ); + expect(people.every((result) => result.success)).toBe(true); + + const result = await engine.search(work, { + ...base, + where: [createdBy('https://ex/bram')], + facetQueries: { creator: 'visscher' }, + }); + + expect(counts(result)).toEqual({ 'https://ex/anna': 1 }); + }); }); diff --git a/packages/search-typesense/test/query-compiler.test.ts b/packages/search-typesense/test/query-compiler.test.ts index 6da5da8e..a0feb6d8 100644 --- a/packages/search-typesense/test/query-compiler.test.ts +++ b/packages/search-typesense/test/query-compiler.test.ts @@ -697,6 +697,48 @@ describe('buildSearchParams', () => { ).toBe(50); }); + it('searches a literal facet’s values with its facet query', () => { + expect( + buildSearchParams( + { + ...base, + facets: ['status', 'unknown', 'keyword'], + facetQueries: { status: ' ', unknown: 'x', keyword: 'kaart' }, + }, + schema, + ).facet_query, + ).toBe('keyword:kaart'); + }); + + it('separates words at a colon in a facet query, as Typesense reads one as its field separator', () => { + expect( + buildSearchParams( + { ...base, facets: ['keyword'], facetQueries: { keyword: 'van:rijn' } }, + schema, + ).facet_query, + ).toBe('keyword:van rijn'); + }); + + it('leaves a reference facet’s query to the engine, which matches referents', () => { + const withReference: SearchType = { + ...schema, + fields: [ + ...schema.fields, + { name: 'publisher', kind: 'reference', facetable: true }, + ], + }; + expect( + buildSearchParams( + { + ...base, + facets: ['publisher'], + facetQueries: { publisher: 'rijks' }, + }, + withReference, + ).facet_query, + ).toBeUndefined(); + }); + it('caps a search at the largest limit among its facets', () => { expect( buildSearchParams( diff --git a/packages/search-typesense/vite.config.ts b/packages/search-typesense/vite.config.ts index 6138942b..56ae6c43 100644 --- a/packages/search-typesense/vite.config.ts +++ b/packages/search-typesense/vite.config.ts @@ -22,10 +22,10 @@ export default mergeConfig( // by-name map for the shared `rootTypeNamed`: one COVERED arrow fewer // over one function fewer, which lowers the ratio without uncovering // anything. - functions: 99.4, + functions: 99.43, // Lines, branches and statements moved the same way when the range // bound reading went to `@lde/search` as `storedRangeBound`. - lines: 99.68, + lines: 99.7, // Re-anchored for the projection lookup: its guards against a // projection naming what no lookup reaches are unreachable through // the port, since `assertValidQuery` rejects such a query first. @@ -34,8 +34,8 @@ export default mergeConfig( // companion: covered branches deleted, nothing uncovered. And when // a field declaring presence alone went back to being internal, // deleting the covered branches that kept its value unindexed. - branches: 97.48, - statements: 99.69, + branches: 97.66, + statements: 99.71, }, }, }, diff --git a/packages/search/src/adapter.ts b/packages/search/src/adapter.ts index 75e6e711..513c20f8 100644 --- a/packages/search/src/adapter.ts +++ b/packages/search/src/adapter.ts @@ -28,6 +28,7 @@ export { TARGET_FIELD, facetNamed, facetsOf, + isQueryableFacet, presenceFacetName, holdsNoIds, identityFieldOf, diff --git a/packages/search/src/engine.ts b/packages/search/src/engine.ts index e5f7ff1d..4294e636 100644 --- a/packages/search/src/engine.ts +++ b/packages/search/src/engine.ts @@ -36,7 +36,10 @@ import type { * value its own clause selects, at count 0 when nothing else leaves it a * document ({@link SearchQuery.facets}). An adapter plans the searches with * `facetSearches` from `@lde/search/adapter`, so ownership cannot drift - * between engines. + * between engines. A facet with a {@link SearchQuery.facetQueries query} + * reports only the values matching it, selected or not, and its + * {@link Facet.totalValues} counts those values – absent where the engine + * cannot tell how many match. * * `Types` is the literal tuple {@link searchSchema} captured: with a * `defineSearchType` declaration, `search()` only accepts the deployment’s diff --git a/packages/search/src/index.ts b/packages/search/src/index.ts index 573a0d29..8d37b360 100644 --- a/packages/search/src/index.ts +++ b/packages/search/src/index.ts @@ -76,7 +76,7 @@ export type { JoinGraph } from './join-graph.js'; // `filterOn` builds the ordinary one-criterion clause – the shape a consumer // policy (`queryDefaults`) appends to `where`, so it belongs on the authoring // surface and not only on the adapter one. -export { filterOn } from './query.js'; +export { filterOn, facetQueryText, MIN_FACET_QUERY_LENGTH } from './query.js'; export type { SearchQuery, ReferenceProjection, diff --git a/packages/search/src/query.ts b/packages/search/src/query.ts index 24fd54ce..8c8206a5 100644 --- a/packages/search/src/query.ts +++ b/packages/search/src/query.ts @@ -5,6 +5,7 @@ import { ID_FIELD, holdsNoIds, isInlineReference, + isQueryableFacet, nestedReferenceType, physicalFields, presenceFacetName, @@ -52,6 +53,26 @@ export interface SearchQuery { * still caps each facet at its own. */ readonly facetLimits?: Readonly>; + /** + * Per facet field, text to search that facet’s values with: only the values + * matching it come back, counted over the matching documents as without a + * query (skip-own-filter), highest count first and capped at the facet’s + * {@link facetLimits} limit. A literal facet matches its values’ own text; a + * reference facet matches its referents by the free-text search of the type + * they belong to – their labels, alternate names, whatever that type + * searches. Text shorter than {@link MIN_FACET_QUERY_LENGTH} characters, + * once trimmed, is no query ({@link facetQueryText}). + * + * Only a facet {@link isQueryableFacet} admits takes one: a keyword facet, + * or a reference facet whose values have a label source. + * + * Under a query, a facet reports only the values that match it, selected or + * not – the one exception to a facet reporting every value its own clause + * selects. The same facet without a query still does, so a selection can + * always be undone there. {@link facetSearches} gives each facet with a + * query a search of its own. + */ + readonly facetQueries?: Readonly>; /** Selects the per-locale fields to query/sort on (from `Accept-Language`). */ readonly locale: string; /** @@ -277,6 +298,25 @@ export function isUnsatisfiable(query: SearchQuery): boolean { }); } +/** + * The fewest characters a facet query must have, once trimmed, to be acted + * on. Below it, prefix and typo matching match next to nothing or nearly + * everything, so a shorter query counts as none. + */ +export const MIN_FACET_QUERY_LENGTH = 3; + +/** + * A {@link SearchQuery.facetQueries} entry as a search acts on it: the text, + * trimmed, or `undefined` when there is none or it is shorter than + * {@link MIN_FACET_QUERY_LENGTH} characters once trimmed. + */ +export function facetQueryText(text: string | undefined): string | undefined { + const trimmed = text?.trim(); + return trimmed === undefined || trimmed.length < MIN_FACET_QUERY_LENGTH + ? undefined + : trimmed; +} + /** * One search a query’s facets need, with the values each of its facets must * report whatever their count. See {@link facetSearches}. @@ -290,7 +330,8 @@ export interface FacetSearch { * values. */ readonly selected: Readonly>; /** The facet whose own clauses this search removed; absent on the search - * that every facet without a clause of its own shares. */ + * that every facet without a clause of its own shares, and on a facet + * query’s search when the facet has no clause of its own. */ readonly own?: string; } @@ -332,6 +373,11 @@ export interface FacetSearch { * A default a surface adds to `where` (a valid-only `status`) is a clause * like any other: its facet counts across every value and reports the * default’s values as selected. + * + * A facet with a {@link SearchQuery.facetQueries query} gets a search of its + * own as well, carrying that query alone and selecting nothing: under a + * query, a facet reports only the values matching it. Every other search + * carries no facet query. */ export function facetSearches(query: SearchQuery): FacetSearch[] { const owners = query.where.map(ownedField); @@ -339,27 +385,55 @@ export function facetSearches(query: SearchQuery): FacetSearch[] { owners.filter((field): field is string => field !== undefined), ); const facets = [...new Set(query.facets)]; - // The facets without a clause of their own share a search per limit. + const facetQueryOf = (field: string): string | undefined => + facetQueryText(query.facetQueries?.[field]); + // A search without a facet query carries none, so a facet query is read + // only by the one search it belongs to. + const { facetQueries, ...unqueried } = query; + const base: SearchQuery = + facetQueries === undefined ? query : (unqueried as SearchQuery); + // The facets without a clause or a query of their own share a search per + // limit. const shared = new Map(); - for (const field of facets.filter((field) => !owned.has(field))) { + for (const field of facets.filter( + (field) => !owned.has(field) && facetQueryOf(field) === undefined, + )) { const limit = query.facetLimits?.[field]; shared.set(limit, [...(shared.get(limit) ?? []), field]); } const searches: FacetSearch[] = [...shared.values()].map((fields) => ({ - query: { ...query, facets: fields }, + query: { ...base, facets: fields }, selected: {}, })); - for (const field of facets.filter((field) => owned.has(field))) { + for (const field of facets.filter( + (field) => owned.has(field) || facetQueryOf(field) !== undefined, + )) { + const text = facetQueryOf(field); + const ownClauses = owned.has(field); + const where = ownClauses + ? query.where.filter((_, index) => owners[index] !== field) + : query.where; + if (text !== undefined) { + // Under a query a facet reports only the values matching it, so it + // selects nothing: a selected value that does not match stays out. + searches.push({ + query: { + ...base, + where, + facets: [field], + facetQueries: { [field]: text }, + }, + selected: {}, + ...(ownClauses ? { own: field } : {}), + }); + continue; + } const own = query.where.filter((_, index) => owners[index] === field); const selected = [ ...new Set(own.flatMap((filter) => filter.or).flatMap(selectedValues)), ]; searches.push({ - query: { - ...query, - where: query.where.filter((_, index) => owners[index] !== field), - facets: [field], - }, + query: { ...base, where, facets: [field] }, selected: selected.length > 0 ? { [field]: selected } : {}, own: field, }); @@ -388,6 +462,10 @@ export interface QueryIssue { | 'not-filterable' | 'operator-mismatch' | 'not-facetable' + /** A {@link SearchQuery.facetQueries facet query} on a facet that takes + * none ({@link isQueryableFacet}): a boolean, range or presence facet, or + * a reference facet whose values have no label source. */ + | 'not-queryable' /** The criterion’s `on` path crosses more than {@link MAX_JOIN_DEPTH} * *joins*. Nested hops cost no round trip and do not count. */ | 'join-too-deep' @@ -769,6 +847,25 @@ export function validateQuery( collectLeafIssues(criterion, constrained, field, issues, schema); } } + for (const [name, text] of Object.entries(query.facetQueries ?? {})) { + if (facetQueryText(text) === undefined) { + continue; + } + const facet = facetNamed(searchType, name, schema); + if (facet !== undefined && isQueryableFacet(facet, schema)) { + continue; + } + issues.push({ + part: 'facets', + field: name, + reason: + facet !== undefined + ? 'not-queryable' + : fieldNamed(searchType, name) === undefined + ? 'unknown-field' + : 'not-facetable', + }); + } for (const name of query.facets) { if (facetNamed(searchType, name, schema) !== undefined) { continue; diff --git a/packages/search/src/schema.ts b/packages/search/src/schema.ts index ebf9956f..f7208217 100644 --- a/packages/search/src/schema.ts +++ b/packages/search/src/schema.ts @@ -3244,6 +3244,28 @@ export function facetNamed( return facetsOf(searchType, schema).find((facet) => facet.name === name); } +/** + * Whether a facet takes a query (`SearchQuery.facetQueries`) – text to + * search its values with. A facet whose values are text a person can type + * against does: a `keyword` facet, matched on its values’ own text, and a + * reference facet whose values have a label source ({@link labelTargetNamesOf}), + * matched by the free-text search of the types they belong to. A reference + * without one holds bare IRIs, which nobody types; a boolean, range or presence + * facet has no values to search. + */ +export function isQueryableFacet( + facet: FacetDeclaration, + schema: SearchSchema | undefined, +): boolean { + if (facet.presence === true) { + return false; + } + return ( + facet.field.kind === 'keyword' || + labelTargetNamesOf(facet.field, schema).length > 0 + ); +} + /** The logical name of the facet counting whether a field declaring * {@link SearchFieldBase.presence} holds a value: `‹field›.exists`. */ export function presenceFacetName(name: string): string { diff --git a/packages/search/src/testing.ts b/packages/search/src/testing.ts index 886e18c8..3881bcad 100644 --- a/packages/search/src/testing.ts +++ b/packages/search/src/testing.ts @@ -1,6 +1,7 @@ import { describe, expect, it } from 'vitest'; import type { SearchEngine } from './engine.js'; import { + facetQueryText, filterOn, filterOperatorFor, type Filter, @@ -10,6 +11,7 @@ import { import { facetableFields, facetsOf, + isQueryableFacet, isRangeFacet, filterableFields, ID_FIELD, @@ -18,6 +20,7 @@ import { referencedTargetsOf, nestedReferenceType, outputFields, + type FacetDeclaration, type RootType, type SearchField, type SearchType, @@ -436,6 +439,144 @@ export function describeSearchEngineContract( } }); + it('rejects a facet query on a facet that takes none', async () => { + for (const searchType of types()) { + const facet = facetsOf(searchType, engine().schema).find( + (candidate) => !isQueryableFacet(candidate, engine().schema), + ); + if (facet === undefined) { + continue; + } + await expect( + engine().searchFacets(searchType, [ + { + ...browse(searchType), + limit: 0, + facets: [facet.name], + facetQueries: { [facet.name]: 'xyz' }, + }, + ]), + ).rejects.toThrow(/not-queryable/); + } + }); + + it('under a facet query, finds a value by its text and reports only values the facet holds, by count, capped at the limit', async () => { + // The text a person would type for a bucket: a literal value’s own text, + // a reference’s label. + const textOf = ( + facet: FacetDeclaration, + bucket: { + readonly value: unknown; + readonly label?: Readonly>; + }, + ): string | undefined => + facet.field.kind === 'reference' + ? Object.values(bucket.label ?? {})[0]?.[0] + : typeof bucket.value === 'string' + ? bucket.value + : undefined; + for (const searchType of types()) { + const facets = facetsOf(searchType, engine().schema).filter((facet) => + isQueryableFacet(facet, engine().schema), + ); + if (facets.length === 0) { + continue; + } + const [unqueried] = await engine().searchFacets(searchType, [ + { + ...browse(searchType), + limit: 0, + facets: facets.map((facet) => facet.name), + }, + ]); + if (unqueried === undefined || 'error' in unqueried) { + continue; // The batch test above fails on this already. + } + for (const facet of facets) { + const all = unqueried.facets[facet.name]; + const [top] = all?.buckets ?? []; + // A value too short to search by says nothing about matching. + const text = + top === undefined ? undefined : facetQueryText(textOf(facet, top)); + if (all === undefined || top === undefined || text === undefined) { + continue; + } + const limit = all.buckets.length; + const [outcome] = await engine().searchFacets(searchType, [ + { + ...browse(searchType), + limit: 0, + facets: [facet.name], + facetQueries: { [facet.name]: text }, + facetLimits: { [facet.name]: limit }, + }, + ]); + if (outcome === undefined || 'error' in outcome) { + throw new Error(`Expected facets for a query on ${facet.name}.`); + } + const queried = outcome.facets[facet.name]?.buckets ?? []; + expect(queried).toContainEqual( + expect.objectContaining({ value: top.value, count: top.count }), + ); + expect(queried.length).toBeLessThanOrEqual(limit); + const counts = queried.map((bucket) => bucket.count); + expect(counts).toEqual( + [...counts].sort((left, right) => right - left), + ); + const totalValues = outcome.facets[facet.name]?.totalValues; + if (totalValues !== undefined) { + expect(totalValues).toBeGreaterThanOrEqual(queried.length); + } + // Where the facet without a query reported every value, each match + // is one of them, at the same count. + if (all.totalValues !== undefined && all.totalValues <= limit) { + for (const bucket of queried) { + expect(all.buckets).toContainEqual( + expect.objectContaining({ + value: bucket.value, + count: bucket.count, + }), + ); + } + } + } + } + }); + + it('under a facet query, leaves out a selected value that does not match', async () => { + // The one exception to a facet reporting every value its own clause + // selects: under a query only matches come back. Without the query the + // value is reported at count 0, which the test above pins. + const absent = 'urn:lde:testing:absent-facet-value'; + for (const searchType of types()) { + const facet = facetsOf(searchType, engine().schema).find( + (candidate) => + isQueryableFacet(candidate, engine().schema) && + candidate.reference === undefined && + candidate.field.filterable === true && + filterOperatorFor(candidate.field) === 'in', + ); + if (facet === undefined) { + continue; + } + const [outcome] = await engine().searchFacets(searchType, [ + { + ...browse(searchType), + limit: 0, + where: [filterOn({ field: facet.name, in: [absent] })], + facets: [facet.name], + facetQueries: { [facet.name]: 'qxzvj' }, + }, + ]); + if (outcome === undefined || 'error' in outcome) { + throw new Error(`Expected facets for a query on ${facet.name}.`); + } + expect(outcome.facets[facet.name]?.buckets ?? []).not.toContainEqual( + expect.objectContaining({ value: absent }), + ); + } + }); + it('answers a browse query with hits, a total and facets', async () => { for (const searchType of types()) { const result = await engine().search(searchType, browse(searchType)); diff --git a/packages/search/test/facet-queries.test.ts b/packages/search/test/facet-queries.test.ts new file mode 100644 index 00000000..32acaa2f --- /dev/null +++ b/packages/search/test/facet-queries.test.ts @@ -0,0 +1,185 @@ +import { describe, expect, it } from 'vitest'; +import { + facetSearches, + validateQuery, + type SearchQuery, +} from '../src/query.js'; +import { + facetNamed, + isQueryableFacet, + searchSchema, + type FacetDeclaration, + type SearchType, +} from '../src/schema.js'; + +const person: SearchType = { + name: 'Person', + class: 'https://schema.org/Person', + fields: [ + { + name: 'label', + kind: 'text', + locales: ['nl'], + output: true, + searchable: { weight: 1 }, + }, + ], +}; + +const work: SearchType = { + name: 'CreativeWork', + class: 'https://schema.org/CreativeWork', + fields: [ + { name: 'keyword', kind: 'keyword', facetable: true, filterable: true }, + { + name: 'creator', + kind: 'reference', + facetable: true, + filterable: true, + ref: { strategy: 'idOnly' }, + labelSource: 'Person', + }, + { + name: 'class', + kind: 'reference', + facetable: true, + filterable: true, + ref: { strategy: 'idOnly' }, + }, + { name: 'iiif', kind: 'boolean', facetable: true, filterable: true }, + { name: 'size', kind: 'integer', facetable: true, filterable: true }, + { name: 'note', kind: 'keyword', output: true }, + // A keyword, but its presence facet counts booleans. + { name: 'subtitle', kind: 'keyword', output: true, presence: true }, + ], +}; + +const schema = searchSchema(person, work); + +const base: SearchQuery = { + where: [], + orderBy: [], + limit: 0, + offset: 0, + facets: [], + locale: 'nl', +}; + +const facet = (name: string): FacetDeclaration => + facetNamed(work, name, schema) as FacetDeclaration; + +describe('isQueryableFacet', () => { + it('admits a keyword facet and a reference facet with a label source', () => { + expect(isQueryableFacet(facet('keyword'), schema)).toBe(true); + expect(isQueryableFacet(facet('creator'), schema)).toBe(true); + }); + + it('refuses a reference without a label source, a boolean, a range and a presence facet', () => { + expect(isQueryableFacet(facet('class'), schema)).toBe(false); + expect(isQueryableFacet(facet('iiif'), schema)).toBe(false); + expect(isQueryableFacet(facet('size'), schema)).toBe(false); + expect(isQueryableFacet(facet('subtitle.exists'), schema)).toBe(false); + }); +}); + +describe('validateQuery on facet queries', () => { + it('accepts a query on a queryable facet, and ignores one too short to act on anywhere', () => { + expect( + validateQuery( + { + ...base, + facets: ['keyword', 'creator', 'class'], + facetQueries: { keyword: 'boe', creator: 'rem', class: 're' }, + }, + work, + schema, + ), + ).toEqual([]); + }); + + it('rejects a query on a facet that takes none, naming why', () => { + expect( + validateQuery( + { + ...base, + facets: ['class', 'iiif', 'size'], + facetQueries: { + class: 'xyz', + iiif: 'xyz', + size: 'xyz', + note: 'xyz', + nonexistent: 'xyz', + }, + }, + work, + schema, + ), + ).toEqual([ + { part: 'facets', field: 'class', reason: 'not-queryable' }, + { part: 'facets', field: 'iiif', reason: 'not-queryable' }, + { part: 'facets', field: 'size', reason: 'not-queryable' }, + { part: 'facets', field: 'note', reason: 'not-facetable' }, + { part: 'facets', field: 'nonexistent', reason: 'unknown-field' }, + ]); + }); +}); + +describe('facetSearches with facet queries', () => { + it('gives a facet with a query a search of its own, and keeps the query off the others', () => { + const query: SearchQuery = { + ...base, + facets: ['keyword', 'creator', 'iiif'], + facetQueries: { creator: 'rem' }, + }; + const { facetQueries: _, ...unqueried } = query; + expect(facetSearches(query)).toEqual([ + { query: { ...unqueried, facets: ['keyword', 'iiif'] }, selected: {} }, + { + query: { + ...unqueried, + facets: ['creator'], + facetQueries: { creator: 'rem' }, + }, + selected: {}, + }, + ]); + }); + + it('removes the facet’s own clause under a query, and selects nothing', () => { + const query: SearchQuery = { + ...base, + facets: ['keyword'], + where: [ + { or: [{ field: 'keyword', in: ['selected'] }] }, + { or: [{ field: 'iiif', is: true }] }, + ], + facetQueries: { keyword: 'boe' }, + }; + expect(facetSearches(query)).toEqual([ + { + query: { + ...query, + where: [{ or: [{ field: 'iiif', is: true }] }], + facets: ['keyword'], + }, + selected: {}, + own: 'keyword', + }, + ]); + }); + + it.each(['', ' ', 're', ' re '])( + 'treats %j, shorter than three characters, as no query', + (text) => { + const query: SearchQuery = { + ...base, + facets: ['keyword'], + facetQueries: { keyword: text }, + }; + const { facetQueries: _, ...unqueried } = query; + expect(facetSearches(query)).toEqual([ + { query: { ...unqueried, facets: ['keyword'] }, selected: {} }, + ]); + }, + ); +}); diff --git a/packages/search/vite.config.ts b/packages/search/vite.config.ts index ef98e91b..168ceb83 100644 --- a/packages/search/vite.config.ts +++ b/packages/search/vite.config.ts @@ -12,7 +12,7 @@ export default mergeConfig( thresholds: { functions: 100, lines: 100, - branches: 99.92, + branches: 99.93, statements: 100, }, },