feat(python): expose FTS query tokenization

[BubbleCal]

What is the new feature?

Add public lance.tokenize(...) and FtsToken(text, position) APIs for inspecting FTS query tokenization without creating a dataset or index.

Why do we need this feature?

Applications need to preview and debug the exact query terms and positions produced by Lance tokenizer configuration before running a search or building an index.

How does it work?

• Reuse the Rust FTS tokenizer and query-token collection logic.
• Share tokenizer option parsing with PyLance INVERTED index creation so the two APIs cannot drift.
• Support text/code analyzer profiles, language, custom stop words, n-grams, maximum token length, and code tokenizer flags.
• Add top-level exports, type stubs, API examples, focused tests, and user documentation.

Validation

• CARGO_BUILD_JOBS=2 make build from python/: passed
• New and adjacent Python tests: 8 passed
• Runtime example: the Cats and Dogs -> [("cat", 0), ("dog", 2)]
• uv run make lint from python/: passed
• cargo fmt --all -- --check: passed
• cargo clippy --all --tests --benches -- -D warnings: passed
• git diff main...HEAD --check: passed

The only build warning was the existing macOS linker warning that __eh_frame is too large.

Linear: OSS-2008

[lance-gatekeeper]

✅ Gate recommendation: approve.

This closes a real FTS observability gap at the right boundary: the Python API reuses the production Rust tokenizer configuration and query-token collector, while shared option parsing keeps index creation aligned. The additive API, documentation, and focused coverage preserve the existing query and index contracts.