Support CREATE VECTOR INDEX

[moshap-firebolt]

CREATE VECTOR INDEX creates an index for approximate nearest-neighbor search over an embedding column. It is not specific to one dialect — BigQuery, Oracle, SQL Server, MariaDB and TiDB all have it (so the Generic dialect should too). VECTOR is not a keyword, so parse_create previously failed with Expected: an object type after CREATE, found: VECTOR.

The statement shares a common core across dialects and then diverges into dialect-specific trailers. It is parsed permissively for every dialect.

Common core

CREATE [OR REPLACE] VECTOR INDEX [IF NOT EXISTS] <name> ON <table>(<column | expr>)

VECTOR is treated as a modifier on CREATE INDEX (like UNIQUE), reusing the existing CreateIndex node rather than a new statement variant. It is routed through parse_create_index, so it inherits the standard index trailers — USING <method>, INCLUDE (...), WITH (...), expression targets and index options.

Per-dialect forms

Dialect	Form / trailer	Status
BigQuery	OPTIONS(index_type=…, distance_type=…, ivf_options='{…}'), STORING(col, …)	parsed
SQL Server	WITH (METRIC=…, TYPE=…, MAXDOP=…), bracket-quoted names	parsed
TiDB	expression target ((VEC_COSINE_DISTANCE(col))), trailing USING HNSW	parsed
Oracle	core, expression target, INCLUDE (...)	parsed
Oracle	ORGANIZATION …, DISTANCE …, WITH TARGET ACCURACY …, PARAMETERS(...)	not yet — follow-up
MariaDB	bare DISTANCE = … M = … options	not yet — follow-up

The two follow-up rows need dedicated grammar with real keyword-collision hazards — WITH TARGET ACCURACY overloads the same WITH keyword as SQL Server's WITH (...), and MariaDB's bare single-letter M collides with identifiers — so they are left out of this change.

Changes

• Add vector, or_replace, options and storing fields to CreateIndex. Display renders CREATE [OR REPLACE ]VECTOR INDEX … and the STORING(...) / OPTIONS(...) trailers.
• Accept a WITH (...) clause on a vector index in every dialect (for SQL Server), not only the dialects that enable it for a plain CREATE INDEX.
• Plain CREATE INDEX is unchanged (the new fields default to false/empty).

Tests

All are round-trip (verified_stmt) tests.

• tests/sqlparser_common.rs — the generic core plus the shared trailers (INCLUDE, STORING, WITH, USING, expression targets), across all dialects.
• tests/sqlparser_bigquery.rs — OPTIONS(...) with real index_type / distance_type / JSON keys, and STORING(...).
• tests/sqlparser_mssql.rs — bracket-quoted names with WITH (...).
• tests/sqlparser_mysql.rs — TiDB's distance-function target with USING.
• tests/sqlparser_oracle.rs — the core, an expression target and an INCLUDE list.

Docs: BigQuery · Oracle · SQL Server · MariaDB · TiDB

[LucaCappelletti94]

At this time, this PR edits would suggest that CREATE VECTOR INDEX is a BigQuery-specific syntax, but it is not, as it is supported also by:

• Oracle SQL
• Microsoft SQL
• Therefore also Generic Dialect

Therefore, move the tests to sqlparser common and ensure that all dialects that support the syntax successfully parse it. It may be acceptable to maintainers that dialects that do not support the syntax also parse it given the current preference for permissive parsing. I suggest you also review the possible syntax variations that these other dialects may have on this syntax, so as to cover more of its variants completely.

[moshap-firebolt]

@LucaCappelletti94 - done, added tests demonstrating different syntaxes. Note - didn't implement full blown Oracle's one.

[LucaCappelletti94]

I believe any Oracle support note should be removed at this time, given that its mandatory grammar is still unsupported.

Do recall to rebase on main to avoid this branch getting stale.

[LucaCappelletti94]

None of these inputs are valid Oracle statements, and the common test already covers permissive parsing of the shared prefix across every dialect. Since correct Oracle statements are not currently supported, it does not make sense to test for incorrect ones even though this implementation parses them without erroring out, which I am unsure whether that is desirable.

Suggested change

	#[test]
	fn parse_oracle_create_vector_index() {
	// Oracle's specialized clauses (ORGANIZATION / DISTANCE / WITH TARGET
	// ACCURACY / PARAMETERS) are not yet parsed; the forms it shares with the
	// common grammar — an expression target and an `INCLUDE` list — round-trip.
	oracle().verified_stmt("CREATE VECTOR INDEX g_idx ON galaxies(embedding)");
	oracle().verified_stmt("CREATE VECTOR INDEX g_idx ON galaxies(VEC_DISTANCE(embedding))");
	oracle().verified_stmt("CREATE VECTOR INDEX g_idx ON galaxies(embedding) INCLUDE (id)");
	}

[LucaCappelletti94]

VECTOR likely should be a keyword

Suggested change

	VECTOR,
	VERBOSE,

[LucaCappelletti94]

After adding the keyword, you can now replace this with:

Suggested change

	} else if matches!(
	&self.peek_token_ref().token,
	Token::Word(w) if w.keyword == Keyword::NoKeyword && w.value.eq_ignore_ascii_case("VECTOR")
	) {
	} else if self.parse_keyword(Keyword::VECTOR) {

[LucaCappelletti94]

The ON syntax from SQL Server CREATE VECTOR INDEX grammar still does not parse, here is a red test for it:

Suggested change

	);
	);
	ms().verified_stmt(
	"CREATE VECTOR INDEX vec_idx ON [dbo].[articles]([title_vector]) WITH (METRIC = 'cosine', TYPE = 'DiskANN', MAXDOP = 8) ON [PRIMARY]",
	);