ai fts: improve full-text index docs and scoring details by zhaoshangzi · Pull Request #23302 · pingcap/docs

zhaoshangzi · 2026-07-17T08:37:44Z

First-time contributors' checklist

I've signed the Contributor License Agreement, which is required for the repository owners to accept my contribution.

What is changed, added or deleted? (Required)

Expand the full-text search guide with practical details: clarify that the STANDARD parser lowercases text for case-insensitive matching; add a "Managing full-text indexes" section explaining index naming, viewing (SHOW INDEX / INFORMATION_SCHEMA), dropping, and examples for CREATE/ALTER/CREATE FULLTEXT INDEX; add sections on tokenization and multi-word search semantics (fts_match_word uses tokenization and OR semantics, no phrase or prefix matching), explain effect of repeated terms on BM25 scoring, and document the BM25Tanvity relevance algorithm with the standard k1=1.2 and b=0.75 defaults.

Which TiDB version(s) do your changes apply to? (Required)

Tips for choosing the affected version(s):

By default, CHOOSE MASTER ONLY so your changes will be applied to the next TiDB major or minor releases. If your PR involves a product feature behavior change or a compatibility change, CHOOSE THE AFFECTED RELEASE BRANCH(ES) AND MASTER.

For details, see tips for choosing the affected versions.

What is the related PR or file link(s)?

This PR is translated from:
Other reference link(s):

AI agent involvement

The changes in this PR were primarily made by an AI agent on behalf of the PR author.

Do your changes match any of the following descriptions?

Delete files
Change aliases
Need modification after applied to another branch
Might cause conflicts after applied to another branch

Expand the full-text search guide with practical details: clarify that the STANDARD parser lowercases text for case-insensitive matching; add a "Managing full-text indexes" section explaining index naming, viewing (SHOW INDEX / INFORMATION_SCHEMA), dropping, and examples for CREATE/ALTER/CREATE FULLTEXT INDEX; add sections on tokenization and multi-word search semantics (fts_match_word uses tokenization and OR semantics, no phrase or prefix matching), explain effect of repeated terms on BM25 scoring, and document the BM25Tanvity relevance algorithm with the standard k1=1.2 and b=0.75 defaults.

pingcap-cla-assistant · 2026-07-17T08:37:58Z

All committers have signed the CLA.

coderabbitai · 2026-07-17T08:38:10Z

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: f48b4472-dd4c-402a-902a-0b2d8853176f

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

ti-chi-bot · 2026-07-17T08:43:04Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from lilin90. For more information see the Code Review Process.
Please ensure that each of them provides their approval before proceeding.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ti-chi-bot · 2026-07-20T04:37:17Z

[LGTM Timeline notifier]

Timeline:

2026-07-20 04:37:17.349099407 +0000 UTC m=+1206823.385194453: ☑️ agreed by 3pointer.

lilin90 · 2026-07-20T06:37:19Z


+### Manage full-text indexes
+
+When creating a full-text index, the index name is optional. If not specified, TiDB automatically uses the first column name of the index as the index name.


Suggested change

When creating a full-text index, the index name is optional. If not specified, TiDB automatically uses the first column name of the index as the index name.

When creating a full-text index, specifying an index name is optional. If you do not specify one, TiDB uses the name of the first indexed column as the index name by default.

lilin90 · 2026-07-20T06:39:34Z

+When creating a full-text index, the index name is optional. If not specified, TiDB automatically uses the first column name of the index as the index name.
+
+```sql
+-- Without specifying an index name, TiDB automatically generates the name "title"


Suggested change

-- Without specifying an index name, TiDB automatically generates the name "title"

-- Without specifying an index name, TiDB uses the first indexed column name ("title") as the index name

lilin90 · 2026-07-20T06:40:12Z

+ALTER TABLE stock_items ADD FULLTEXT INDEX ft_title (title) WITH PARSER MULTILINGUAL;
+```
+
+**Viewing existing index names:**


Suggested change

**Viewing existing index names:**

**View existing index names:**

lilin90 · 2026-07-20T06:40:24Z

+WHERE TABLE_SCHEMA = 'your_database' AND TABLE_NAME = 'stock_items';
+```
+
+**Dropping a full-text index:**


Suggested change

**Dropping a full-text index:**

**Drop a full-text index:**

lilin90 · 2026-07-20T06:41:41Z

+
+#### Specify an index name
+
+In both `CREATE TABLE` and `ALTER TABLE` syntax, you can specify a name for the index after `FULLTEXT INDEX` or `FULLTEXT KEY`:


Suggested change

In both `CREATE TABLE` and `ALTER TABLE` syntax, you can specify a name for the index after `FULLTEXT INDEX` or `FULLTEXT KEY`:

In both `CREATE TABLE` and `ALTER TABLE` statements, you can specify an index name after `FULLTEXT INDEX` or `FULLTEXT KEY`:

lilin90 · 2026-07-20T06:42:59Z


+#### Multi-word search: tokenization and query semantics
+
+When using `fts_match_word()`, the query string is split into individual tokens according to the parser's rules, and each token is matched independently.


Suggested change

When using `fts_match_word()`, the query string is split into individual tokens according to the parser's rules, and each token is matched independently.

When you use `fts_match_word()`, the query string is tokenized according to the parser's rules, and each token is matched independently.

lilin90 · 2026-07-20T06:44:06Z

+
+When using `fts_match_word()`, the query string is split into individual tokens according to the parser's rules, and each token is matched independently.
+
+For the STANDARD parser, strings are split into words by spaces and punctuation. For the MULTILINGUAL parser, strings are split according to each language's segmentation rules.


Suggested change

For the STANDARD parser, strings are split into words by spaces and punctuation. For the MULTILINGUAL parser, strings are split according to each language's segmentation rules.

The STANDARD parser tokenizes strings into words using spaces and punctuation as delimiters. The MULTILINGUAL parser tokenizes strings according to language-specific segmentation rules.

lilin90 · 2026-07-20T06:49:58Z

+SELECT * FROM users WHERE fts_match_word('Alice Smith', name);
+```
+
+A common misconception is that `fts_match_word('Alice X', name)` treats `"Alice X"` as a single entity for exact matching. In reality, it is tokenized into `Alice` and `X`, using OR semantics. Since `X` is a very short term, it can match many irrelevant documents. Avoid using very short query terms or single letters.


Suggested change

A common misconception is that `fts_match_word('Alice X', name)` treats `"Alice X"` as a single entity for exact matching. In reality, it is tokenized into `Alice` and `X`, using OR semantics. Since `X` is a very short term, it can match many irrelevant documents. Avoid using very short query terms or single letters.

A common misconception is that `fts_match_word('Alice X', name)` treats `"Alice X"` as a single entity for exact matching. In reality, it is tokenized into `Alice` and `X`, using OR semantics. Because `X` is a very short query term, it can match many irrelevant documents. Avoid using very short query terms or single letters.

lilin90 · 2026-07-20T06:51:18Z

+
+A common misconception is that `fts_match_word('Alice X', name)` treats `"Alice X"` as a single entity for exact matching. In reality, it is tokenized into `Alice` and `X`, using OR semantics. Since `X` is a very short term, it can match many irrelevant documents. Avoid using very short query terms or single letters.
+
+> **Note:** TiDB full-text search does not support exact phrase matching (matching all tokens consecutively in order).


Suggested change

> **Note:** TiDB full-text search does not support exact phrase matching (matching all tokens consecutively in order).

> **Note:**

>

> TiDB full-text search does not support exact phrase matching, where all query tokens must appear consecutively and in the specified order.

lilin90 · 2026-07-20T06:55:36Z

+
+#### Relevance scoring algorithm
+
+TiDB full-text search uses the **BM25Tanvity** algorithm for computing relevance scores. It is a variant of the classic BM25 (Okapi BM25) that uses Count-Min Sketch to approximate document frequency (DF) estimation for improved performance.


Suggested change

TiDB full-text search uses the **BM25Tanvity** algorithm for computing relevance scores. It is a variant of the classic BM25 (Okapi BM25) that uses Count-Min Sketch to approximate document frequency (DF) estimation for improved performance.

TiDB full-text search uses the **BM25Tantivy** algorithm to calculate relevance scores. This algorithm is a variant of the classic BM25 (Okapi BM25) algorithm that uses Count-Min Sketch to approximate document frequency (DF) for improved performance.

lilin90 · 2026-07-21T06:08:02Z

@zhaoshangzi Would you please address the comments?

zhaoshangzi requested review from 3pointer and lilin90 July 17, 2026 08:37

ti-chi-bot Bot added contribution This PR is from a community contributor. missing-translation-status This PR does not have translation status info. labels Jul 17, 2026

ti-chi-bot Bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Jul 17, 2026

lilin90 added the translation/no-need No need to translate this PR. label Jul 17, 2026

ti-chi-bot Bot removed the missing-translation-status This PR does not have translation status info. label Jul 17, 2026

lilin90 added the ONCALL Relates to documentation oncall. label Jul 17, 2026

lilin90 self-assigned this Jul 17, 2026

Remove two extra blank lines

4ff741b

lilin90 changed the title ~~Improve full-text index docs and scoring details~~ ai FTS: improve full-text index docs and scoring details Jul 17, 2026

lilin90 changed the title ~~ai FTS: improve full-text index docs and scoring details~~ ai fts: improve full-text index docs and scoring details Jul 17, 2026

3pointer approved these changes Jul 20, 2026

View reviewed changes

ti-chi-bot Bot added the needs-1-more-lgtm Indicates a PR needs 1 more LGTM. label Jul 20, 2026

lilin90 reviewed Jul 20, 2026

View reviewed changes

Comment thread ai/guides/vector-search-full-text-search-sql.md Outdated

Comment thread ai/guides/vector-search-full-text-search-sql.md Outdated

Make style consistent

b18b767

lilin90 reviewed Jul 20, 2026

View reviewed changes


		### Manage full-text indexes

		When creating a full-text index, the index name is optional. If not specified, TiDB automatically uses the first column name of the index as the index name.

	When creating a full-text index, the index name is optional. If not specified, TiDB automatically uses the first column name of the index as the index name.
	When creating a full-text index, specifying an index name is optional. If you do not specify one, TiDB uses the name of the first indexed column as the index name by default.

	-- Without specifying an index name, TiDB automatically generates the name "title"
	-- Without specifying an index name, TiDB uses the first indexed column name ("title") as the index name

	Viewing existing index names:
	View existing index names:


		#### Specify an index name

		In both `CREATE TABLE` and `ALTER TABLE` syntax, you can specify a name for the index after `FULLTEXT INDEX` or `FULLTEXT KEY`:


		#### Multi-word search: tokenization and query semantics

		When using `fts_match_word()`, the query string is split into individual tokens according to the parser's rules, and each token is matched independently.

	When using `fts_match_word()`, the query string is split into individual tokens according to the parser's rules, and each token is matched independently.
	When you use `fts_match_word()`, the query string is tokenized according to the parser's rules, and each token is matched independently.


		When using `fts_match_word()`, the query string is split into individual tokens according to the parser's rules, and each token is matched independently.

		For the STANDARD parser, strings are split into words by spaces and punctuation. For the MULTILINGUAL parser, strings are split according to each language's segmentation rules.

	For the STANDARD parser, strings are split into words by spaces and punctuation. For the MULTILINGUAL parser, strings are split according to each language's segmentation rules.
	The STANDARD parser tokenizes strings into words using spaces and punctuation as delimiters. The MULTILINGUAL parser tokenizes strings according to language-specific segmentation rules.

	A common misconception is that `fts_match_word('Alice X', name)` treats `"Alice X"` as a single entity for exact matching. In reality, it is tokenized into `Alice` and `X`, using OR semantics. Since `X` is a very short term, it can match many irrelevant documents. Avoid using very short query terms or single letters.
	A common misconception is that `fts_match_word('Alice X', name)` treats `"Alice X"` as a single entity for exact matching. In reality, it is tokenized into `Alice` and `X`, using OR semantics. Because `X` is a very short query term, it can match many irrelevant documents. Avoid using very short query terms or single letters.


		A common misconception is that `fts_match_word('Alice X', name)` treats `"Alice X"` as a single entity for exact matching. In reality, it is tokenized into `Alice` and `X`, using OR semantics. Since `X` is a very short term, it can match many irrelevant documents. Avoid using very short query terms or single letters.

		> Note: TiDB full-text search does not support exact phrase matching (matching all tokens consecutively in order).


		#### Relevance scoring algorithm

		TiDB full-text search uses the BM25Tanvity algorithm for computing relevance scores. It is a variant of the classic BM25 (Okapi BM25) that uses Count-Min Sketch to approximate document frequency (DF) estimation for improved performance.

Uh oh!

Conversation

zhaoshangzi commented Jul 17, 2026 • edited by lilin90 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

First-time contributors' checklist

What is changed, added or deleted? (Required)

Which TiDB version(s) do your changes apply to? (Required)

What is the related PR or file link(s)?

AI agent involvement

Do your changes match any of the following descriptions?

Uh oh!

pingcap-cla-assistant Bot commented Jul 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coderabbitai Bot commented Jul 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Uh oh!

ti-chi-bot Bot commented Jul 17, 2026

Uh oh!

ti-chi-bot Bot commented Jul 20, 2026

[LGTM Timeline notifier]

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lilin90 commented Jul 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

zhaoshangzi commented Jul 17, 2026 •

edited by lilin90

Loading

pingcap-cla-assistant Bot commented Jul 17, 2026 •

edited

Loading

coderabbitai Bot commented Jul 17, 2026 •

edited

Loading