feat(fts): show tokenized queries in analyze plan

[BubbleCal]

What is the new feature?

analyze_plan now shows both the original FTS query and its execution-time tokenized representation, including token text and normalized positions.

Why do we need this feature?

Users need to see how the configured tokenizer transforms a query when debugging lowercasing, stemming, stop-word removal, phrase positions, and other analyzer behavior.

How does it work?

• Capture the tokens produced by the actual execution tokenizer in a shared runtime snapshot.
• Append tokenized_query to indexed and flat Match/Phrase nodes and posting-backed compound query leaves.
• Preserve the snapshot when DataFusion replaces plan children.
• Keep explain_plan unchanged because it does not execute the tokenizer.

Validation

• cargo test -p lance test_analyze_plan_shows --lib: 3 passed
• Adjacent parts-searched metric tests: 2 passed
• uv run make lint from python/: passed
• cargo fmt --all -- --check: passed
• cargo clippy --all --tests --benches -- -D warnings: passed
• git diff main...HEAD --check: passed

Limitations

Plans that resolve directly to EmptyExec do not execute an FTS tokenizer node, so they have no tokenization snapshot to display.

Linear: OSS-2007

[lance-gatekeeper]

✅ Gate recommendation: approve.

The change captures analyzer output at the execution boundary where tokenizer selection is authoritative, preserves it through plan replacement, and leaves non-executing explain plans unchanged. This is preferable to eager planning-time tokenization, which could observe the wrong tokenizer and blur the explain/analyze contract.