From 5de59bfd9db025143ce35ef20d34ddebc0b6f757 Mon Sep 17 00:00:00 2001 From: zhaoshangzi Date: Fri, 17 Jul 2026 16:35:17 +0800 Subject: [PATCH 1/4] Improve full-text index docs and scoring details Expand the full-text search guide with practical details: clarify that the STANDARD parser lowercases text for case-insensitive matching; add a "Managing full-text indexes" section explaining index naming, viewing (SHOW INDEX / INFORMATION_SCHEMA), dropping, and examples for CREATE/ALTER/CREATE FULLTEXT INDEX; add sections on tokenization and multi-word search semantics (fts_match_word uses tokenization and OR semantics, no phrase or prefix matching), explain effect of repeated terms on BM25 scoring, and document the BM25Tanvity relevance algorithm with the standard k1=1.2 and b=0.75 defaults. --- .../vector-search-full-text-search-sql.md | 117 +++++++++++++++++- 1 file changed, 116 insertions(+), 1 deletion(-) diff --git a/ai/guides/vector-search-full-text-search-sql.md b/ai/guides/vector-search-full-text-search-sql.md index ca64f281c64dc..af73baeb16567 100644 --- a/ai/guides/vector-search-full-text-search-sql.md +++ b/ai/guides/vector-search-full-text-search-sql.md @@ -70,10 +70,61 @@ ALTER TABLE stock_items ADD FULLTEXT INDEX (title) WITH PARSER MULTILINGUAL ADD_ The following parsers are accepted in the `WITH PARSER ` clause: -- `STANDARD`: fast, works for English content, splitting words by spaces and punctuation. +- `STANDARD`: fast, works for English content, splitting words by spaces and punctuation. All text is lowercased for indexing and search (case-insensitive matching). - `MULTILINGUAL`: supports multiple languages, including English, Chinese, Japanese, and Korean. +### Managing full-text indexes + +When creating a full-text index, the index name is optional. If not specified, TiDB automatically uses the first column name of the index as the index name. + +```sql +-- Without specifying an index name, TiDB automatically generates the name "title" +ALTER TABLE stock_items ADD FULLTEXT INDEX (title) WITH PARSER MULTILINGUAL; + +-- Specifying an index name +ALTER TABLE stock_items ADD FULLTEXT INDEX ft_title (title) WITH PARSER MULTILINGUAL; +``` + +**Viewing existing index names:** + +```sql +-- The Key_name column shows the index name +SHOW INDEX FROM stock_items; + +-- Or query INFORMATION_SCHEMA +SELECT INDEX_NAME, COLUMN_NAME, INDEX_TYPE +FROM INFORMATION_SCHEMA.STATISTICS +WHERE TABLE_SCHEMA = 'your_database' AND TABLE_NAME = 'stock_items'; +``` + +**Dropping a full-text index:** + +```sql +-- Use SHOW INDEX to confirm the index name first +ALTER TABLE stock_items DROP INDEX title; +``` + +#### Specifying an index name + +In both `CREATE TABLE` and `ALTER TABLE` syntax, you can specify a name for the index after `FULLTEXT INDEX` or `FULLTEXT KEY`: + +```sql +-- Specifying a name in CREATE TABLE +CREATE TABLE users ( + id INT, + name TEXT, + FULLTEXT INDEX ft_name (name) WITH PARSER STANDARD +); + +-- Specifying a name in ALTER TABLE +ALTER TABLE users ADD FULLTEXT INDEX ft_name (name) WITH PARSER STANDARD; + +-- Using standalone CREATE FULLTEXT INDEX (an index name is required) +CREATE FULLTEXT INDEX ft_name ON users (name) WITH PARSER STANDARD; +``` + + ### Insert text data Inserting data into a table with a full-text index is identical to inserting data into any other tables. @@ -152,6 +203,70 @@ SELECT COUNT(*) FROM stock_items +----------+ ``` +#### Multi-word search: tokenization and query semantics + +When using `fts_match_word()`, the query string is split into individual tokens according to the parser's rules, and each token is matched independently. + +For the STANDARD parser, strings are split into words by spaces and punctuation. For the MULTILINGUAL parser, strings are split according to each language's segmentation rules. + +```sql +-- This query is tokenized into two tokens: "Alice" and "Smith" +SELECT * FROM users WHERE fts_match_word('Alice Smith', name); +``` + +`fts_match_word()` uses **OR** semantics: a document matches if it contains any of the tokens, and matching more tokens increases the relevance score. + +```sql +-- The query below returns all rows where the name column contains +-- "Alice" or "Smith" or both +SELECT * FROM users WHERE fts_match_word('Alice Smith', name); +``` + +A common misconception is that `fts_match_word('Alice X', name)` treats `"Alice X"` as a single entity for exact matching. In reality, it is tokenized into `Alice` and `X`, using OR semantics. Since `X` is a very short term, it can match many irrelevant documents. Avoid using very short query terms or single letters. + +> **Note:** TiDB full-text search does not support exact phrase matching (matching all tokens consecutively in order). + +#### Prefix search + +**Not supported.** + +#### Effect of repeated terms on relevance scores + +The relevance score returned by `fts_match_word()` is based on the **BM25** algorithm. If a query string contains repeated terms, the term frequency of that term is doubled in scoring. + +```sql +-- "Alice" appears twice; in BM25 scoring, Alice's term frequency is 2 +SELECT * FROM users WHERE fts_match_word('Alice alice bob', name); +``` + +In this example, a document matching `Alice` receives twice the weight contribution compared to `bob`. This is expected behavior of the BM25 algorithm, which evaluates relevance based on term frequency (TF). + +#### Relevance scoring algorithm + +TiDB full-text search uses the **BM25Tanvity** algorithm for computing relevance scores. It is a variant of the classic BM25 (Okapi BM25) that uses Count-Min Sketch to approximate document frequency (DF) estimation for improved performance. + +**BM25 formula (standard form):** + +``` +score(D, Q) = sum_{t in Q} IDF(t) * TF(t, D) * (k1 + 1) / (TF(t, D) + k1 * (1 - b + b * |D| / avgdl)) +``` + +Where: + +- `t`: query term +- `Q`: query string (all tokens after tokenization) +- `D`: the document being evaluated +- `TF(t, D)`: term frequency of `t` in the document +- `IDF(t)`: inverse document frequency, measuring the rarity of the term +- `|D|`: document length +- `avgdl`: average document length across all documents +- `k1`, `b`: BM25 tuning parameters + +TiDB's implementation uses fixed values of `k1 = 1.2` and `b = 0.75`, which are the standard defaults for BM25 in information retrieval. + +The returned score is a non-negative floating-point number. A higher value indicates higher relevance to the query. Scores are not directly comparable across different datasets. + + ## Advanced example: Join search results with other tables You can combine full-text search with other SQL features such as joins and subqueries. From 4ff741b6cad84db56301c8ae51b50feca8a8cd42 Mon Sep 17 00:00:00 2001 From: lilin90 Date: Fri, 17 Jul 2026 16:42:52 +0800 Subject: [PATCH 2/4] Remove two extra blank lines --- ai/guides/vector-search-full-text-search-sql.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/ai/guides/vector-search-full-text-search-sql.md b/ai/guides/vector-search-full-text-search-sql.md index af73baeb16567..00cacce22cc5a 100644 --- a/ai/guides/vector-search-full-text-search-sql.md +++ b/ai/guides/vector-search-full-text-search-sql.md @@ -124,7 +124,6 @@ ALTER TABLE users ADD FULLTEXT INDEX ft_name (name) WITH PARSER STANDARD; CREATE FULLTEXT INDEX ft_name ON users (name) WITH PARSER STANDARD; ``` - ### Insert text data Inserting data into a table with a full-text index is identical to inserting data into any other tables. @@ -266,7 +265,6 @@ TiDB's implementation uses fixed values of `k1 = 1.2` and `b = 0.75`, which are The returned score is a non-negative floating-point number. A higher value indicates higher relevance to the query. Scores are not directly comparable across different datasets. - ## Advanced example: Join search results with other tables You can combine full-text search with other SQL features such as joins and subqueries. From b18b767f48e134d272622e2bff5be69d6d156b4c Mon Sep 17 00:00:00 2001 From: Lilian Lee Date: Mon, 20 Jul 2026 14:33:54 +0800 Subject: [PATCH 3/4] Make style consistent --- ai/guides/vector-search-full-text-search-sql.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/ai/guides/vector-search-full-text-search-sql.md b/ai/guides/vector-search-full-text-search-sql.md index 00cacce22cc5a..6d8b7ccf58abb 100644 --- a/ai/guides/vector-search-full-text-search-sql.md +++ b/ai/guides/vector-search-full-text-search-sql.md @@ -74,7 +74,7 @@ The following parsers are accepted in the `WITH PARSER ` clause: - `MULTILINGUAL`: supports multiple languages, including English, Chinese, Japanese, and Korean. -### Managing full-text indexes +### Manage full-text indexes When creating a full-text index, the index name is optional. If not specified, TiDB automatically uses the first column name of the index as the index name. @@ -105,7 +105,7 @@ WHERE TABLE_SCHEMA = 'your_database' AND TABLE_NAME = 'stock_items'; ALTER TABLE stock_items DROP INDEX title; ``` -#### Specifying an index name +#### Specify an index name In both `CREATE TABLE` and `ALTER TABLE` syntax, you can specify a name for the index after `FULLTEXT INDEX` or `FULLTEXT KEY`: From dab87175eb77465784565758f76e59062e01393e Mon Sep 17 00:00:00 2001 From: zhaoshangzi Date: Wed, 22 Jul 2026 10:45:40 +0800 Subject: [PATCH 4/4] Apply suggestions from code review Co-authored-by: Lilian Lee --- .../vector-search-full-text-search-sql.md | 22 ++++++++++--------- 1 file changed, 12 insertions(+), 10 deletions(-) diff --git a/ai/guides/vector-search-full-text-search-sql.md b/ai/guides/vector-search-full-text-search-sql.md index 6d8b7ccf58abb..22f89e1eb8dbf 100644 --- a/ai/guides/vector-search-full-text-search-sql.md +++ b/ai/guides/vector-search-full-text-search-sql.md @@ -76,17 +76,17 @@ The following parsers are accepted in the `WITH PARSER ` clause: ### Manage full-text indexes -When creating a full-text index, the index name is optional. If not specified, TiDB automatically uses the first column name of the index as the index name. +When creating a full-text index, specifying an index name is optional. If you do not specify one, TiDB uses the name of the first indexed column as the index name by default. ```sql --- Without specifying an index name, TiDB automatically generates the name "title" +-- Without specifying an index name, TiDB uses the first indexed column name ("title") as the index name ALTER TABLE stock_items ADD FULLTEXT INDEX (title) WITH PARSER MULTILINGUAL; -- Specifying an index name ALTER TABLE stock_items ADD FULLTEXT INDEX ft_title (title) WITH PARSER MULTILINGUAL; ``` -**Viewing existing index names:** +**View existing index names:** ```sql -- The Key_name column shows the index name @@ -98,7 +98,7 @@ FROM INFORMATION_SCHEMA.STATISTICS WHERE TABLE_SCHEMA = 'your_database' AND TABLE_NAME = 'stock_items'; ``` -**Dropping a full-text index:** +**Drop a full-text index:** ```sql -- Use SHOW INDEX to confirm the index name first @@ -107,7 +107,7 @@ ALTER TABLE stock_items DROP INDEX title; #### Specify an index name -In both `CREATE TABLE` and `ALTER TABLE` syntax, you can specify a name for the index after `FULLTEXT INDEX` or `FULLTEXT KEY`: +In both `CREATE TABLE` and `ALTER TABLE` statements, you can specify an index name after `FULLTEXT INDEX` or `FULLTEXT KEY`: ```sql -- Specifying a name in CREATE TABLE @@ -204,9 +204,9 @@ SELECT COUNT(*) FROM stock_items #### Multi-word search: tokenization and query semantics -When using `fts_match_word()`, the query string is split into individual tokens according to the parser's rules, and each token is matched independently. +When you use `fts_match_word()`, the query string is tokenized according to the parser's rules, and each token is matched independently. -For the STANDARD parser, strings are split into words by spaces and punctuation. For the MULTILINGUAL parser, strings are split according to each language's segmentation rules. +The STANDARD parser tokenizes strings into words using spaces and punctuation as delimiters. The MULTILINGUAL parser tokenizes strings according to language-specific segmentation rules. ```sql -- This query is tokenized into two tokens: "Alice" and "Smith" @@ -221,9 +221,11 @@ SELECT * FROM users WHERE fts_match_word('Alice Smith', name); SELECT * FROM users WHERE fts_match_word('Alice Smith', name); ``` -A common misconception is that `fts_match_word('Alice X', name)` treats `"Alice X"` as a single entity for exact matching. In reality, it is tokenized into `Alice` and `X`, using OR semantics. Since `X` is a very short term, it can match many irrelevant documents. Avoid using very short query terms or single letters. +A common misconception is that `fts_match_word('Alice X', name)` treats `"Alice X"` as a single entity for exact matching. In reality, it is tokenized into `Alice` and `X`, using OR semantics. Because `X` is a very short query term, it can match many irrelevant documents. Avoid using very short query terms or single letters. -> **Note:** TiDB full-text search does not support exact phrase matching (matching all tokens consecutively in order). +> **Note:** +> +> TiDB full-text search does not support exact phrase matching, where all query tokens must appear consecutively and in the specified order. #### Prefix search @@ -242,7 +244,7 @@ In this example, a document matching `Alice` receives twice the weight contribut #### Relevance scoring algorithm -TiDB full-text search uses the **BM25Tanvity** algorithm for computing relevance scores. It is a variant of the classic BM25 (Okapi BM25) that uses Count-Min Sketch to approximate document frequency (DF) estimation for improved performance. +TiDB full-text search uses the **BM25Tantivy** algorithm to calculate relevance scores. This algorithm is a variant of the classic BM25 (Okapi BM25) algorithm that uses Count-Min Sketch to approximate document frequency (DF) for improved performance. **BM25 formula (standard form):**