[diskann-garnet] Implement XVCREATE, XVIMPORT

diskann-garnet/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "diskann-garnet"
-version = "6.0.0"
+version = "6.0.1"
 edition = "2024"
 authors.workspace = true
 license.workspace = true
@@ -22,6 +22,7 @@ diskann-providers.workspace = true
 diskann-utils.workspace = true
 diskann-vector.workspace = true
 foldhash = "0.2.0"
+parking_lot = "0.12.5"
 rand.workspace = true
 strum = { version = "0.28.0", features = ["derive"] }
 thiserror.workspace = true

diskann-garnet/README.md

@@ -37,8 +37,27 @@ Redis always uses cosine distance, but many vector data sets use other metrics.
 - `XBIN_U8`: binary quantization of 8-bit unsigned integer (using DiskANN's spherical quantizer based on RaBitQ)
 
 
-Currently there is a limit of `2^32 - 1` vectors in a single instance due to
-internal IDs being `u32`. This restriction will be lifted in the future.
+Currently there is a limit of `2^32 - 2` user vectors in a single instance. Internal
+ID 0 is reserved for the start point, and `u32::MAX` marks ID allocation exhaustion.
+
+### Bulk Import
+
+Existing DiskANN graphs can be loaded without rebuilding them through `VADD`:
+
+1. Create an empty set with `XVCREATE`, supplying `QUANT_STATE` for a quantized index.
+2. Load its terms with `XVIMPORT`. Terms may be imported in any order and in parallel.
+3. Wait for all term imports to complete, then call `XVIMPORT key FINISH` before normal use.
+
+Internal IDs may also arrive in any order. Importing a high ID creates all missing
+free-space-map blocks through that ID; each block covers 65,536 IDs and uses 8 KiB.
+The start point is created during finalization and is excluded from FSM accounting.
+
+At the FFI layer, `can_import` checks eligibility.
+
+Ordinary operations (`card`, search, ID checks) and `FINISH` permanently disable importing.
+
+See the [FFI reference](docs/ffi-design.rs) and [data design](docs/data-design.md)
+for API and term formats.
 
 ## Installing
 

diskann-garnet/diskann-garnet.nuspec

@@ -2,7 +2,7 @@
 <package>
   <metadata>
     <id>diskann-garnet</id>
-    <version>6.0.0</version>
+    <version>6.0.1</version>
     <readme>docs/README.md</readme>
     <authors>Microsoft</authors>
     <projectUrl>https://github.com/microsoft/DiskANN</projectUrl>

diskann-garnet/docs/data-design.md

@@ -30,7 +30,7 @@ The read-modify-write (rmw) method accesses a single key with a callback but all
 
 ### Keys & Contexts
 
-Garnet keys are arbitrary byte strings (e.g. `&[u8]`). The methods described above can use whatever keys they like, with some caveats, to read and write data, however those methods also take a semi-opaque context which gives the operation a scope. For the most part this context is used for internal Garnet bookkeeping and is opaque, but the least significant 3 bits are available for diskann-garnet to use for its own scoping. Keys must be a multiple of 4 bytes in length, except for keys stored under context 3 and context 5 (`Term::Attributes` and `Term::IntMap`) which are special cased in Garnet as they key is the user-provided external ID. There are no alignment requirements on keys.
+Garnet keys are arbitrary byte strings (e.g. `&[u8]`). The methods described above can use whatever keys they like, with some caveats, to read and write data, however those methods also take a semi-opaque context which gives the operation a scope. For the most part this context is used for internal Garnet bookkeeping and is opaque, but the least significant 3 bits are available for diskann-garnet to use for its own scoping. Keys must be a multiple of 4 bytes in length, except for keys stored under context 5 (`Term::IntMap`), which use the user-provided external ID. There are no alignment requirements on keys.
 
 Diskann-garnet uses these bits to distinguish between differnet kinds of index data so that the same key can be used to fetch different kinds of data. For example, vector data might be stored under the same key, the vector ID, as neighbor lists by setting different context bits for the operation.
 
@@ -58,7 +58,7 @@ precision only index, they are the most accessed term.
 *Key*: Internal ID as bytes; this key is always 4 bytes.
 *Value*: `[u32; M + 1]` stored as bytes, where `M` is the same `M` passed to VADD. Every value in the index will have this fixed size of `(M + 1) * 4` bytes.
 
-Neighbor lists are stored as a fixed size of `(M + 1) * mem::size_of::<u32>()`. The final entry is the true length of the neighbor list.
+Neighbor lists are stored as a fixed size of `(M + 1) * mem::size_of::<u32>()`. The final entry is the true length of the neighbor list and must not exceed `M`.
 
 For example, in a graph where the `M` value is given as 16, the size of a neighbor list would be `(16 + 1) * 4 = 68` bytes long. Using fixed size lists this way means that all neighbor list allocations are the same size.
 
@@ -75,7 +75,7 @@ In a quantized index, these vectors are the most often accessed piece of data an
 
 ### Attributes
 
-*Key*: External ID as bytes; this is variable length byte string that the user assigned.
+*Key*: Internal ID as bytes; this key is always 4 bytes.
 *Value*: Attributes given by the user. Variable length.
 
 When vectors are inserted by the Redis Vector Set API, an arbitrary JSON blob of attributes can be attached. These attributes are stored as a utf-8 string and read/written as a whole unit.
@@ -88,13 +88,17 @@ The are several terms in the index used for internal state management of the dis
 
 #### Start Points
 
-Currently only a single start point is supported, and it is given the internal ID of 0. Its vector data is the same as the first vector that was inserted, and it will not be returned by search, and it will not be modified during the lifetime of the index.
+Currently only a single start point is supported, and it is given the internal ID of 0. During normal insertion, its vector data is the same as the first vector that was inserted. It will not be returned by search, and its vector data will not be modified during the lifetime of the index.
+
+For a nonempty import, finalization copies the first used imported ID's full vector, quantized vector (if present), and neighbor list to ID 0. ID 0 cannot be imported directly.
 
 Start points have no associated attributes.
 
+ID 0 is reserved outside the free space map. The FSM never allocates, counts, visits, or reuses it; the provider maintains the start point's validity separately.
+
 #### Metadata
 
-Metadata is currently used for the free space map which manages used and available internal IDs and the quantizer tables.
+Metadata is currently used for the free space map which manages used and available internal IDs, the quantizer tables, and import eligibility.
 
 ##### Free Space Map
 
@@ -103,9 +107,11 @@ Metadata is currently used for the free space map which manages used and availab
 
 The free space map is used to keep track of which internal IDs are allocated and in use. Please see the [ID Mapping](#id-mapping) section for more details on why mapped IDs are used.
 
-The free space map is a series of blocks where each block is a string of 1-bit values representing the state of the corresponding internal ID. Free IDs are represented by `0b0`, used IDs by `0b1`. Blocks are created on demand during insert when they are needed. Since each block contains 64k internal ID states, the total number of FSM blocks in the index will be at least number of active IDs / 2^16, each of which is 8kB in size.
+The free space map is a contiguous sequence of blocks starting at block zero. Each block contains 65,536 one-bit ID states and occupies 8 KiB: `0b0` for free and `0b1` for used. The bit for reserved ID 0 is ignored. Blocks grow on demand during insertion or import, including intermediate blocks.
 
-During startup, the index will scan FSM blocks in sequence to restore state. It will update the correct bits in a FSM block whenever the state of an internal ID changes.
+During startup, the index scans FSM blocks in sequence to restore state. It updates the corresponding bit whenever a user vector's allocation changes. A failed expansion retains its successfully written prefix, allowing subsequent expansion to continue without erasing earlier claims.
+
+Importing a term keyed by an internal ID claims that ID before writing the term and advances the maximum assigned ID as needed. A failed claim writes no term; a failed term write leaves the ID claimed for a retry. Repeated claims do not increase the used count. Gaps below the maximum ID are available for reuse once reuse is enabled; quantization backfill keeps reuse disabled until it finishes.
 
 ##### Quantizer Tables
 
@@ -114,12 +120,27 @@ During startup, the index will scan FSM blocks in sequence to restore state. It
 
 Note that for the BIN quantizer, a 1 byte flag precedes the quantizer table which indicates whether quantization backfill is complete. That byte is accounted for in the value sizing above.
 
+Quantizer state can only be replaced before importing begins, while no user IDs are occupied in the FSM and no start-point vector exists. Setup takes the import-transition mutex before the FSM quantization barrier. Initial setup from `Disabled` requires exclusive caller access before publishing the index: ordinary operations in that state do not take the transition mutex.
+
+##### Import Status
+
+*Key*: b'_imp'; this key is always 4 bytes.
+*Value*: A single byte: 0 for `Disabled`, 1 for `Ready`, or 2 for `Importing`.
+
+When no saved status exists, an empty index starts `Ready` if it is unquantized or has supplied quantizer state, and `Disabled` otherwise. The initial status is persisted before writing Q8's generated default table, so that table cannot enable imports on reopen. Successfully supplying quantizer state during creation publishes `Ready` after setup completes. A saved status is restored rather than inferred from the absence of a start point.
+
+The first ordinary operation transitions `Ready` to `Disabled`; the first accepted imported term transitions `Ready` to `Importing`. More terms can then arrive in parallel, but ordinary operations fail until all verification tasks and finalization succeed. The last successful task transitions `Importing` to `Disabled`. Empty finalization also waits for every task and does not create a start point. Failed verification or finalization leaves an active import `Importing`.
+
+Status transitions persist the new byte before publishing it with an atomic compare-exchange, under a transition-only mutex. A failed metadata write does not publish the new in-memory status. Ordinary operations in `Disabled` only load the atomic status; admitted term writes and verification workers do not hold the transition mutex. Callers must finish all term writes before starting finalization.
+
+The former Boolean format did not distinguish ready indexes from unfinished imports; finish those imports before upgrading.
+
 #### Internal ID Mapping
 
 *Key*: Internal ID as bytes; this key is always 4 bytes in length.
 *Value*: External ID; this is variable length byte string that the user assigned.
 
-Each internal ID corresponds to an external ID, which is a byte string of arbitrary length. The external IDs are stored unmodified and read/written as a whole.
+Each internal ID corresponds to an external ID, which is a byte string of arbitrary length, including zero. The external IDs are stored unmodified and read/written as a whole.
 
 Lookup of an external ID will happen during post processing when we return results to Garnet.
 
@@ -134,7 +155,7 @@ Lookup of an internal ID will happen for things such as delete.
 
 ## ID Mapping
 
-Garnet vector set IDs are arbitrary length byte strings natively. These are quite inefficient for indexing so we map each external ID to a `u32` internal ID. This imposes a maximum on the number of vectors that are indexable of `u32::MAX - 1` (the start point always consumes an ID).
+Garnet vector set IDs are arbitrary length byte strings natively. These are quite inefficient for indexing so we map each external ID to a `u32` internal ID. User vector IDs range from 1 through `u32::MAX - 1`, supporting at most 2^32 - 2 vectors. ID 0 is reserved for the start point, and `u32::MAX` is the exhausted next-ID value rather than an assignable ID.
 
 When the DiskANN algorithm performs searches and other operations it works with internal IDs only. The external IDs are only used when returning data to the user (which has no concept of the internal IDs) or when asked to perform operations like delete on specific vectors which will be identified by their external ID. In order to convert back and forth for these occasions, lookup tables must be kept for the mapping.
 
@@ -175,4 +196,4 @@ Filtered search introduces two problems regarding attributes. The first is that
 It is possible to co-locate the appropriate vector term and the attribute term as a single value which may alleviate much of this problem. Because we can read partial values in the read callback, we can access the correct portion of the data (vector data would be stored first as it has a known, fixed size). However, we can alleviate only one of the two problems in an index with both full precision and quantized vector terms as we must choose whether the search operation should not see skew or whether the final reranking step should not see skew.
 
 
-
+

diskann-garnet/docs/ffi-design.rs

@@ -45,6 +45,16 @@ enum InsertResult {
     SuccessUpdate = 3,
 }
 
+/// Status returned by `finish_import`, encoded as a `u8`.
+///
+/// `Success` means the task succeeded, including finalization if it was the last task.
+/// `TaskFailed` permits retrying that task; `FinishFailed` is a non-retryable finalization failure.
+enum ImportResult {
+    Success = 0,
+    TaskFailed = 1,
+    FinishFailed = 2,
+}
+
 /// Read one or more keys from Garnet.
 ///
 /// `keys` holds `key_count` keys, each 4-byte length prefixed. `value_length_hint` is the
@@ -314,9 +324,11 @@ extern "C" fn remove(
     id_len: usize,
 ) -> bool;
 
-/// Return number of vectors stored in index.
+/// Return number of live vectors in index, excluding the reserved start point.
 ///
 /// Equivalent to VCARD (https://redis.io/docs/latest/commands/vcard/) can be approximate, must be fast.
+///
+/// Returns `u64::MAX` on error.
 #[unsafe(no_mangle)]
 extern "C" fn card(context: u64, index_ptr: *const c_void) -> u64;
 
@@ -381,3 +393,70 @@ extern "C" fn search_neighbors(
     output_distances_len: usize,
     overflow: *mut *mut c_void,
 ) -> i32;
+
+/// Set serialized quantizer state on an empty, quantized index, for XVCREATE.
+///
+/// Pass serialized quantizer bytes. The metric and dimension must match. Importing must not have
+/// begun and the index must contain no data. Initial setup on an import-disabled index requires
+/// exclusive access before publishing it to concurrent commands. Success enables importing;
+/// Q8's automatically generated default state does not.
+///
+/// Returns true on success and false otherwise. On failure, discard the vector set: in-memory
+/// state and persisted metadata may already have changed.
+#[unsafe(no_mangle)]
+extern "C" fn set_quant_state(
+    context: u64,
+    index_ptr: *const c_void,
+    state: *const u8,
+    state_len: usize,
+) -> bool;
+
+/// Return whether the index accepts imported terms, both before and during import.
+/// This query does not begin or disable importing.
+#[unsafe(no_mangle)]
+extern "C" fn can_import(context: u64, index_ptr: *const c_void) -> bool;
+
+/// Import a term directly into storage, for XVIMPORT.
+///
+/// term_type is 0 for vectors, 1 for neighbors, 2 for quantized vectors, 3 for
+/// attributes, 5 for external-to-internal ID mappings, or 6 for
+/// internal-to-external ID mappings. Metadata (4) cannot be imported. IDs and
+/// values must be non-null and non-empty (except for external IDs which may be
+/// empty). Terms may arrive in any order and must use IDs between 1 and
+/// u32::MAX - 1. Importing a high ID creates every missing FSM block through
+/// that ID. ID 0 is reserved for the start point, which finalization creates
+/// outside FSM accounting. External-to-internal mapping values must be a
+/// nonzero native-endian u32. Neighbor terms must have `(max_degree + 1) * 4`
+/// bytes and a trailing native-endian u32 count no greater than max_degree.
+///
+/// The first accepted term begins importing. Ordinary operations fail until FINISH succeeds.
+/// A failed term write keeps the import in progress; callers may repair or retry that term.
+///
+/// Returns true on success and false otherwise.
+#[unsafe(no_mangle)]
+extern "C" fn import_term(
+    context: u64,
+    index_ptr: *const c_void,
+    term_type: u32,
+    id: *const u8,
+    id_len: usize,
+    value: *const u8,
+    value_len: usize,
+) -> bool;
+
+/// Verify imported terms and finalize the index, for XVIMPORT ... FINISH.
+///
+/// Finish all term writes before starting these tasks; no term writes may overlap verification.
+/// Each task receives its own task_idx in 0..task_count. Do not repeat successful tasks. The last
+/// successful task finalizes the index and enables ordinary operations. Until then, including
+/// after a failed verification or finalization, ordinary operations remain blocked. An empty
+/// import also waits for every task and closes without creating a start point.
+///
+/// Returns an `ImportResult` discriminant.
+#[unsafe(no_mangle)]
+extern "C" fn finish_import(
+    context: u64,
+    index_ptr: *const c_void,
+    task_idx: usize,
+    task_count: usize,
+) -> u8;