fix(cubestore): rebuild missing secondary index when unique index lookup returns empty

[Xuxiaotuan]

Issue Reference this PR resolves

#11212

Description of Changes Made

Background

DROP TABLE <schema>.<table> via MySQL wire protocol fails with:

Internal: One value expected in SchemaRocksTable for "SEMANTIC_LAYER_XXX" but nothing found

The affected schemas' tables still appear in information_schema.tables (which uses schema ID lookup, not the name index), so they look normal from the client's perspective. But any DROP against them hits this error at the schema name lookup stage. MySQL wire clients are stuck — they can't clean up. Cube's own driver doesn't help either because it uses a different code path (version_entries) that also doesn't see these schemas. Physical tables accumulate indefinitely.

Root Cause

The error comes from get_single_row_by_index() at rocks_table.rs:883. The secondary index entry for the schema name is missing, but the schema row itself still exists (findable by primary key ID).

Normal write paths (do_insert, delete, update) write both the row KV and secondary index entries atomically in the same RocksDB batch, so the issue is not caused by normal operations. The index entry gets lost in less common scenarios:

1. Incomplete index migration: migration_check_indexes() only checks for the SecondaryIndexInfo marker entry, not the actual index entries. If the marker exists but individual entries are missing (e.g. from an interrupted upgrade), no rebuild is triggered.
2. Interrupted schema rename: Process crash between deleting the old name index entry and writing the new one.
3. RocksDB compaction / SST recovery edge cases.

Changes

File: rust/cubestore/cubestore/src/metastore/rocks_table.rs — get_single_row_by_index() (+11 lines)

// Before:
let row = self.get_row_by_index_opt(row_key, secondary_index, false)?;
row.ok_or(CubeError::internal(format!(
    "One value expected in {:?} for {:?} but nothing found",
    self, row_key
)))

// After:
let row = self.get_row_by_index_opt(row_key, secondary_index, false)?;
if let Some(row) = row {
    return Ok(row);
}
// Unique index: index entry missing but row might still exist.
// Rebuild the index so the next lookup succeeds, mirroring the
// existing repair pattern in get_row_by_index_opt.
if RocksSecondaryIndex::is_unique(secondary_index) {
    let index = self.get_index_by_id(BaseRocksSecondaryIndex::get_id(secondary_index));
    self.rebuild_index(&index)?;
}
Err(CubeError::internal(format!(
    "One value expected in {:?} for {:?} but nothing found",
    self, row_key
)))

Why this approach: Mirrors the existing repair pattern at get_row_by_index_opt (line 833), which handles the reverse case — index entry exists but row is missing — by calling rebuild_index() and returning an error.

Scope: Only fires for unique indexes. Non-unique indexes returning empty is a legitimate "no results" result. The affected callers are all name-based lookups on small datasets: SchemaRocksIndex::Name, TableRocksIndex::Name, SourceRocksIndex::Name, MultiIndexRocksIndex::ByName.

Caveat: Since rebuild_index() writes directly to RocksDB but get_single_row_by_index is typically called inside a write_operation with a point-in-time snapshot, the first call returns an error (stale snapshot doesn't see the new entries). The caller retries with a fresh snapshot and the operation succeeds. This is the same behavior as the existing repair path.

Alternative considered: Also considered scanning all rows to find the match and writing just the missing entry — that way the first call would succeed without a retry. The code would be:

if RocksSecondaryIndex::is_unique(secondary_index) {
    let search = secondary_index.key_to_bytes(row_key);
    for row in self.scan_all_rows()? {
        let row = row?;
        if RocksSecondaryIndex::should_index_row(secondary_index, row.get_row())
            && RocksSecondaryIndex::index_key_by(secondary_index, row.get_row()) == search
        {
            let index = self.get_index_by_id(BaseRocksSecondaryIndex::get_id(secondary_index));
            let kv = self.index_key_val(row.get_row(), row.get_id(), &index);
            let mut batch = WriteBatch::default();
            batch.put(kv.key, kv.val);
            self.db().write(batch)?;
            return Ok(Some(row));
        }
    }
}

|                     | Chosen (rebuild + error) | Alternative (scan + Ok) |
|---------------------|--------------------------|--------------------------|
| First attempt       | Fails, user retries      | Succeeds                 |
| Cost                | scan_all_rows + delete ALL + rewrite ALL | scan_all_rows + write 1 entry |
| Codebase precedent  | Mirrors existing pattern | No precedent             |

Went with the current approach because it mirrors the existing rebuild_index pattern in get_row_by_index_opt. The alternative's advantage of first-attempt success doesn't justify introducing a repair mechanism with no precedent in the codebase.

Verification

A separate reproduction binary (not included in this PR) validated the fix:

1. Create a schema via MetaStore API
2. Delete the secondary index entry from RocksDB directly (simulating corruption)
3. Run CREATE TABLE test_schema.t1 (id INT) — first call triggers rebuild_index but returns error (stale snapshot)
4. Run CREATE TABLE test_schema.t1 (id INT) again — succeeds (fresh snapshot sees repaired index)

Step 1: Created schema id=1
Step 2: Deleted index entry for row_id=1
Step 3: First lookup triggers rebuild... Expected error
        Internal: One value expected in SchemaRocksTable for "test_schema" but nothing found
Step 4: Second lookup succeeds... SUCCESS! FIX WORKS

Check List

• Tests have been run in packages where changes have been made if available
• Linter has been run for changed code
• Tests for the changes have been added if not covered yet
• Docs have been added / updated if required