Skip to content

Introduce kind: text requests for BM25 over a text index - #185

Merged
agourlay merged 2 commits into
devfrom
bm25-text-query
Oct 7, 2026
Merged

agourlay merged 2 commits into
devfrom
bm25-text-query

Conversation

@agourlay

@agourlay agourlay commented Oct 7, 2026 •

Copy link
Copy Markdown
Member

Adds BM25 over a payload field's text index to the config path, the Query.text variant Qdrant dev added for 1.19.3 (qdrant/qdrant#10786 and the series before it).

Builds against qdrant/rust-client's dev branch, which carries the new API since qdrant/rust-client#299 merged (71143d8).

Search config: kind: text

requests:
  - kind: text
    using: body                    # payload field with a scoring text index
    k: 1.2                         # optional, BM25 term-frequency saturation
    b: 0.75                        # optional, BM25 length normalization, 0..1
    filters: [...]                 # optional, as for other kinds
    source:                        # random words, or a file of queries
      type: random                 # random | file
      vocab_size: 20000
      min_length: 2
      max_length: 5
      distribution: zipf
  • random draws word_<id> words, the same vocabulary text payloads are generated from, so the queries hit the corpus.
  • file reads one query per line (path, strategy: from-start | random-sample), loaded once at startup. Blank lines are skipped, and an empty file fails at startup.
  • using names the payload field and is required. k must be >= 0 and b within [0, 1], the same bounds the server validates.
  • --search-quality is refused with a text request, because BM25 over a text index is exact and there's no exact reference to compare against. --prefetch is refused too, since there's no vector to rescore. --acorn works on the request's filters.

Upload config: text fields

fields:
  - name: body
    type: text
    tokenizer: word
    scoring: bm25                  # TextIndexParams.scoring; implies phrase matching
    lowercase: true                # optional
    stopwords: english             # optional, stopword language
    stemmer: english               # optional, Snowball stemmer language

scoring, lowercase, stopwords and stemmer apply only to type: text fields with index: true, and are refused elsewhere so they can't be silently dropped.

Other changes

  • create_request_builder takes the query as a single MainQuery (a vector or a text query) instead of a vector plus a text argument, which keeps it under clippy's argument limit.
  • New examples upload-bm25-text and search-bm25-text, plus a README section.
  • The schema reference example gains a text field carrying the new options.

Verification

  • cargo test --release: 207 passed, 7 of them new (config parsing and validation, random and file query sources, request building, flag guards, text index options).
  • cargo clippy --release --all-targets -- -D warnings: clean. cargo fmt --check: clean.
  • End to end against Qdrant dev (850859ec9): upload, random-source and file-source searches all pass, with the server reporting the BM25-scoring index. See the comment below.
  • Re-run after moving to the client's dev branch: 207 tests, clippy and fmt pass. 71143d8's tree is identical to the d960517 the change was built against.

🤖 Generated with Claude Code

Qdrant 1.19.3-dev ranks points by BM25 over a payload field's text index:
`Query.text` carries `TextQuery { query, k, b }`, `using` names the field,
and the field's text index needs `TextIndexParams.scoring`. This builds
bfb against qdrant/rust-client#299, which regenerates the client from
that proto (pinned to its commit until it merges).

Search config:
- `kind: text` with `using` (the payload field), optional `k` and `b`,
  `filters`, and a query `source`: `random` words drawn from the same
  `word_<id>` vocabulary text payloads are generated from, or `file`
  with one query per line (`from-start` or `random-sample`).
- `--search-quality` and `--prefetch` are refused with a text request:
  BM25 over a text index is exact, and it has no vector to rescore.

Upload config, `type: text` fields:
- `scoring: bm25`, plus the tokenizer options `lowercase`, `stopwords`
  and `stemmer` (Snowball), all refused on other field types.

The request builder takes the query as one `MainQuery` (a vector, or a
text query) instead of a vector alone. New examples `upload-bm25-text`
and `search-bm25-text`, and a README section.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@agourlay

agourlay commented Oct 7, 2026

Copy link
Copy Markdown
Member Author

End-to-end check against Qdrant dev at 850859ec9 (1.19.3-dev, --profile perf), whose proto is identical to rust-client#299's. Run on a laptop for correctness, not as a benchmark.

  • bfb upload --example upload-bm25-text -n 20000: 20,000 points uploaded, collection green. The server reports body as {"type": "text", "tokenizer": "word", "phrase_matching": true, "scoring": {"type": "bm25"}}, with phrase matching turned on by scoring as documented.
  • bfb search --example search-bm25-text -n 5000 --search-limit 10 -p 4: no errors, median 688 q/s, server p99 40 ms.
  • kind: text from a type: file source (strategy: from-start, k: 2.0, b: 0.0): no errors, median 1,244 q/s.
  • A direct REST {"query": {"text": "word_1 word_42"}, "using": "body"} on the same collection returns ranked points with BM25 sums (2.677, 2.638, 2.576 for the top three).
  • No ERROR lines in the server log.

The Client version 1.16.1-dev is not compatible with server version 1.19.3-dev warning comes from the client crate's version string on the dev line; the run is unaffected.

🤖 Generated with Claude Code

qdrant/rust-client#299 merged as 71143d8, whose tree is identical to the
d960517 this branch was built and tested against.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@agourlay
agourlay marked this pull request as ready for review October 7, 2026 12:15
@agourlay
agourlay merged commit caaeeef into dev Oct 7, 2026
3 checks passed
agourlay added a commit to agourlay/strawmann that referenced this pull request Oct 7, 2026
W15 needs #185's `kind: text` requests and text index scoring options.
Read for what it changes in a request §4's other rows send: the dense
request is built from the same VectorInput through the new MainQuery,
text index options are sent only when a field sets them, and the client
moves to rust-client 71143d8, which adds proto fields and changes none a
dense request sets. bfb_pin is a stamp key, so pairs measured from here
are STALE against earlier ones, as the next pairs already are for the
client change. The pinned worktree is checked out at caaeeef and built.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants