fix(vector): persist database_id in vector index catalog entry

[EnRaiha]

Vector Index Rebuild Fails for Non-Default Databases; Schemaless JSON Embeddings Dropped

Branch: fix/vector-rebuild-strict (base cc4a0a2a6)
Files changed: 6 · +180 / −49
Status: Patch verified in live deployment (partial) — SEARCH returns rows for strict collections in the default database; non-default-database rebuild pending restart after re-embed.

---

TL;DR

Two independent defects prevent HNSW vector indexes from materializing for real workloads:

1. StoredVectorIndexParams carries no database_id. The boot-time seed and the durable-store rebuild both hardcode DatabaseId::DEFAULT (0), so any collection living in a non-default database (e.g. the graph database, id ≥ 1024) is never seeded, never rebuilt, and its SEARCH returns 0 rows — even though CREATE VECTOR INDEX succeeded and the catalog entry exists.
2. The vector put path only accepts Value::Array embeddings. Schemaless collections whose embedding column is stored/transcoded as a JSON string ("[0.1, 0.2, ...]") are silently skipped by apply_point_put_vector_indexes, so the durable-store rebuild indexes nothing for those collections.

Root Cause 1 — database_id is lost between CREATE and boot

Chain of custody

CREATE VECTOR INDEX (SQL, session db = graph)
  → execute_set_vector_params          (uses task.request.database_id — CORRECT)
  → put_vector_index_params            (catalog entry — database_id NOT stored!)
  → StoredVectorIndexParams            (11 fields, no database_id)
        ↓ boot
  load_vector_index_param_seed         (catalog → entries)
  → seed_vector_index_params           (hardcodes DatabaseId::DEFAULT)
  → rebuild_vector_indexes_from_store  (hardcodes DatabaseId::DEFAULT)
  → scan_documents_for_each(db=0, ...) (collection lives in db=graph → 0 docs)
  → no rebuild → SEARCH returns 0 rows

• VShardId::from_collection_in_database mixes database_id into the vshard hash (nodedb-types/src/id/vshard.rs:48), so at runtime the CREATE path keys everything under the real database id.
• But the durable catalog entry — the only thing that survives a restart — never records which database the index belongs to.
• On boot, seed + rebuild default to database 0. schemaless_vector_field_names and strict_vector_fields then look up vector_params under (0, tid, ...), find nothing, and the HNSW stays empty.

Evidence (live)

Collection	Database	Rebuild log	SEARCH
vtest (strict, vector(8))	default (id 0)	INFO vector_index_rebuild: rebuilt vector index from durable store core=0 collection=vtest rebuilt=3	✅ 3 rows
code2g_nodes (37,325 rows, 1024-dim)	graph (id ≥ 1024)	no rebuild log	❌ 0 rows

The same binary, the same boot, the same code path — the only difference is the database id. That isolates the defect beyond doubt.

Root Cause 2 — schemaless embeddings stored as JSON strings are dropped

apply_point_put_vector_indexes (strict and schemaless arms) matched only:

if let Some(nodedb_types::Value::Array(arr)) = obj.get(field_name)

But a schemaless body transcoded from a columnar TEXT/JSON column (or ingested via doc-object UPSERT where the column is projected as JSON) decodes to Value::String containing "[0.1, 0.2, ...]". The pattern match fails, the document is silently skipped, and the rebuild completes with 0 vectors — indistinguishable from "no embeddings exist."

Fix

1. Persist database_id in the catalog entry

nodedb-types/src/vector_index_params.rs:

• Added database_id: u64 as the 12th field of StoredVectorIndexParams.
• zerompk structs serialize as arrays in this crate (the default-as-map feature is not enabled), so field order is part of the on-disk format — the new field is appended last to keep the legacy shape decodable.
• Documented this in the struct doc comment.

2. Legacy decode ladder (backward compatible)

nodedb/src/control/security/catalog/vector_index_params.rs:

• New decode_vector_index_params(): tries the 12-field struct first; on failure falls back to the legacy 11-field tuple and fills database_id: 0.
• Both get_vector_index_params and list_all_vector_index_params route through it, so existing on-disk entries (written by older builds) keep loading — no catalog migration required.

3. Seed uses the real database id

nodedb/src/data/executor/core_loop/vector_index_seed.rs:

• seed_vector_index_params now keys vector_params / index_configs / declared_dims with e.database_id instead of DatabaseId::DEFAULT.

4. Rebuild scans the real database

nodedb/src/data/executor/core_loop/vector_index_rebuild.rs:

• rebuild_vector_indexes_from_store groups targets by (database_id, tenant_id, collection) from the durable entries, and passes the entry's database_id into sparse_body_format, scan_documents_for_each, and apply_point_put_vector_indexes.
• Removed the hardcoded let db = DatabaseId::DEFAULT.as_u64().
• Still skips VectorSidecar encodings (vector-primary collections have no field in the sidecar; their durability is served by replay_direct_upsert in wal_replay_vector_extended.rs).

5. Accept JSON-string embeddings in both put arms

nodedb/src/data/executor/handlers/point/apply_put/vector/put.rs:

• New floats_from_value() helper: accepts Value::Array (native msgpack) or Value::String (JSON array "[0.1,...]", or comma/whitespace-separated list).
• Both the strict-schema arm and the schemaless arm now extract floats through it, so the durable-store rebuild indexes the same vectors a live PUT would.
• Existing width validation (check_vector_width, RejectedConstraint on dim mismatch) unchanged.

Test Plan

Unit (passing):

cargo test -p nodedb-types --lib vector_index_params
  msgpack_roundtrip                              ... ok   (database_id=7 round-trips)
  legacy_11_field_array_decodes_with_database_id_zero ... ok
cargo check -p nodedb --lib                       # clean

Live verification (vector-only build from this branch, deployed):

1. CREATE TABLE vtest (id int primary key, embedding vector(8)) + 3 INSERTs (default db) → restart
2. Boot log: rebuilt vector index from durable store core=0 collection=vtest rebuilt=3
3. SEARCH vtest USING VECTOR(embedding, ARRAY[0.1,...], 3) → 3 rows, distance 0.92
4. SHOW VECTOR INDEX status ON vtest → dimensions=8, metric=cosine, index_type=hnsw
5. Vector checkpoint written (files_written=1) and restored on next boot (loaded=1 vectors=9)

code2g_nodes (db graph) now has a fresh catalog entry carrying database_id=graph; a restart after the re-embed run triggers the rebuild of all 37k+ rows (in progress).

Files

File	Change
nodedb-types/src/vector_index_params.rs	database_id field + roundtrip tests
nodedb/src/control/security/catalog/vector_index_params.rs	legacy 11-field decode ladder
nodedb/src/control/server/shared/ddl/neutral/dsl/vector_index.rs	CREATE passes database_id into stored params
nodedb/src/data/executor/core_loop/vector_index_seed.rs	seed keys on e.database_id
nodedb/src/data/executor/core_loop/vector_index_rebuild.rs	rebuild scans per-entry database_id
nodedb/src/data/executor/handlers/point/apply_put/vector/put.rs	floats_from_value (Array + JSON String) in both arms

Out of scope / notes

• The WAL VectorParams record already carries the correct database_id (it routes through task.request.database_id); only the durable catalog seed was losing it. No WAL format change.
• The #[msgpack(map)] route was considered and rejected: switching the struct's zerompk representation from array to map would break decoding of every existing catalog entry, which is exactly what the ladder avoids.
• checkpoint_durable_lsn may log a vector checkpoint flush failed ... No such file or directory warning for a dropped vector-primary collection's stale checkpoint path — benign (clamps LSN), pre-existing, unrelated to this change.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.