feat(index): add index format 2 for every index

.clang-tidy

@@ -65,7 +65,7 @@ CheckOptions:
   readability-identifier-naming.StructIgnoredRegexp: '^rebind$'
   readability-identifier-naming.EnumCase: CamelCase
   readability-identifier-naming.EnumConstantCase: CamelCase
-  readability-identifier-naming.EnumConstantIgnoredRegexp: '^(METRIC_L2|METRIC_IP|RECONSTRUCTION|UNBIASED_ESTIMATION|PLAIN)$'
+  readability-identifier-naming.EnumConstantIgnoredRegexp: '^(METRIC_L2|METRIC_IP|METRIC_COSINE|RECONSTRUCTION|UNBIASED_ESTIMATION|PLAIN)$'
   readability-identifier-naming.TypeAliasCase: CamelCase
   readability-identifier-naming.TypeAliasIgnoredRegexp: '^(value_type|pointer|reference|difference_type|iterator_category|is_always_equal|other|maxheap|minheap|ex_ipfunc)$'
   readability-identifier-naming.TypedefCase: CamelCase

CMakeLists.txt

@@ -84,6 +84,7 @@ endfunction()
 
 set(RABITQ_CORE_SOURCES
     src/core/api.cpp
+    src/core/blob.cpp
     src/core/fast_scale_table.cpp
     src/core/provisional.cpp
     src/utils/cpu_features.cpp
@@ -247,6 +248,7 @@ if(RABITQ_BUILD_FASTSCAN)
 endif()
 if(RABITQ_BUILD_INDEX)
     add_library(rabitq_index STATIC
+        src/index/format2.cpp
         src/index/hnsw.cpp
         src/index/ivf.cpp
         src/index/ivf_initializer.cpp
@@ -261,6 +263,10 @@ if(RABITQ_BUILD_INDEX)
     )
     add_library(rabitqlib::index ALIAS rabitq_index)
     target_link_libraries(rabitq_index PUBLIC rabitq_fastscan)
+    # Index files record the library version that wrote them.
+    target_compile_definitions(rabitq_index PRIVATE
+        RABITQ_LIBRARY_VERSION="${PROJECT_VERSION}"
+    )
     set_target_properties(rabitq_index PROPERTIES
         EXPORT_NAME index
         POSITION_INDEPENDENT_CODE ON

README.md

@@ -41,8 +41,8 @@ Earlier versions: [all releases](https://github.com/VectorDB-NTU/RaBitQ-Library/
 ## Vector indexes
 
 Use the included indexes when you want construction, search, and save/load in one
-library. All three support L2 and inner product; normalize vectors for cosine
-search. Their structures, storage costs, and search parameters offer different
+library. All three support L2, inner product and cosine, which normalizes vectors
+for you. Their structures, storage costs, and search parameters offer different
 QPS–recall tradeoffs.
 
 | Index | Search approach | Storage and updates |

benchmarks/hotpaths/load.cpp

@@ -37,18 +37,31 @@ void run(int argc, char** argv) {
         if (!std::filesystem::exists(rabitqlib::io_impl::filesystem_path(path + ".hnsw"))) {
             rabitqlib::ivf::HNSWInitializer index(dim, count);
             index.add_vectors(data.data(), 1);
-            std::ofstream unused;
-            index.save(unused, path.c_str());
+            std::ofstream output(
+                rabitqlib::io_impl::filesystem_path(path + ".hnsw"), std::ios::binary
+            );
+            output.exceptions(std::ios::failbit | std::ios::badbit);
+            index.write(output);
             return;
         }
     }
     bench::metadata(dim, count, 1);
     uint64_t expected = 0;
     for (size_t rep = 0; rep <= reps; ++rep) {
         rabitqlib::ivf::HNSWInitializer index(dim, count);
-        std::ifstream unused;
         const auto start = bench::Clock::now();
-        index.load(unused, path.c_str());
+        // Open the graph file inside the timing, as loading an index does.
+        std::ifstream input(
+            rabitqlib::io_impl::filesystem_path(path + ".hnsw"),
+            std::ios::binary | std::ios::ate
+        );
+        if (!input) {
+            throw std::runtime_error("Cannot open the HNSW centroid graph file");
+        }
+        const auto bytes =
+            static_cast<uint64_t>(static_cast<std::streamoff>(input.tellg()));
+        input.seekg(0);
+        index.read(input, bytes);
         const double seconds =
             std::chrono::duration<double>(bench::Clock::now() - start).count();
         uint64_t h = 14695981039346656037ULL;

docs/development.md

@@ -462,7 +462,9 @@ parity against a saved pre-change build when changing those shared implementatio
 
 Keep `tests/fixtures/core_api_v1/consumer.cpp` unchanged when extending v1; add tests.
 Changes to v1 names, signatures, aggregate fields, enum values, buffer/code semantics
-or allocation guarantees require an explicit API-major decision. The stable header
+or allocation guarantees require an explicit API-major decision. Adding a `MetricType`
+value that v1 rejects, as `METRIC_COSINE` does, is not such a change; keep v1's rejection
+covered by `core_api_test.cpp`. The stable header
 must not expose Eigen, record maps, ANN or ISA-specific types. Workspace APIs must
 never silently allocate on the heap or spawn worker threads.
 
@@ -505,12 +507,13 @@ in place of extra-bit codes.
 
 #### IVF formats and initialization
 
-New IVF files have a magic/version header, an explicit padded dimension, and a resolved initializer type (v2); v1 files infer the historical
-Flat/HNSW choice at 20,000 clusters, independently of current auto selection.
-New IVF auto routing uses Flat below 5,000 clusters, Flat RaBitQ below 60,000,
-and HNSW otherwise. Flat RaBitQ persists its mean and codes in the main file;
-HNSW retains its sidecar. Legacy quantized files and raw v1 files infer
-64-dimension padding.
+New IVF files use index format 2 (`docs/formats.md`): the resolved initializer type
+is in `ivf_params`, and the routing data, including the HNSW graph, is a section of
+the one file. Earlier `RABQIDX1` v2 files store the initializer type; v1 files infer
+the historical Flat/HNSW choice at 20,000 clusters, independently of current auto
+selection. Those layouts keep HNSW routing in the `<file>.hnsw` sidecar. New IVF auto
+routing uses Flat below 5,000 clusters, Flat RaBitQ below 60,000, and HNSW otherwise.
+Legacy quantized files and raw v1 files infer 64-dimension padding.
 
 #### Search defaults
 
@@ -532,7 +535,7 @@ and buffer changes must also preserve the [ID marker](#search-buffer-id-marker)
 
 Filesystem path strings are UTF-8 on Windows and native path bytes on POSIX. Use
 `io_impl::filesystem_path` from `utils/path.hpp` before opening files or querying
-filesystem metadata, including IVF's HNSW sidecar. It converts UTF-8 to native Unicode
+filesystem metadata, including the HNSW sidecar of earlier IVF files. It converts UTF-8 to native Unicode
 on Windows and preserves existing path bytes on Linux/macOS.
 
 For path-only changes, check path conversion and the affected I/O operations; the
@@ -550,15 +553,20 @@ keep their provenance and hashes, and do not regenerate them with the current wr
 Index files are trusted input (see the usage guide): validate structure and sizes so
 that corrupt or mismatched files are reported before allocation, without treating a
 crafted file as a supported input.
+Readers throw `rabitqlib::FormatError` (`include/rabitqlib/index/format_error.hpp`) for
+any truncated, corrupt, mismatched or unsupported content, including errors raised by
+shared validators while loading; only a file that cannot be opened throws a plain
+`std::runtime_error`. Python maps it to `rabitqlib.FormatError`; rejection tests assert it.
 
 Do not silently reinterpret an old file. Add a magic/version discriminator, use fixed-width
-serialized fields for new formats (the current IVF, HNSW and QG bodies still store several
-`size_t` fields; change those only with a format version bump), validate sizes before
-allocation, check every read, and retain a
-compatibility test fixture or an explicit rejection path. Review each affected format;
-changes to shared serialization helpers require auditing every format using them.
-IVF-RaBitQ, HNSW-RaBitQ, and SymphonyQG have separate formats. SymphonyQG includes a versioned quantized format plus a legacy
-raw-format fallback; preserve both unless a breaking change is explicitly requested.
+serialized fields for new formats (earlier layouts store several `size_t` fields; keep their readers unchanged), validate sizes before allocation, check every
+read, and retain a compatibility test fixture or an explicit rejection path. Review each affected
+format; changes to shared serialization helpers require auditing every format using them.
+Every index writes index format 2 ([`docs/formats.md`](formats.md#index-format-2), container in
+`src/index/format2.*`) and still reads its earlier layouts: IVF-RaBitQ's `RABQIDX1` versions and
+unversioned files, HNSW-RaBitQ's unversioned layout, and SymphonyQG's versioned quantized layout
+and raw fallback. Preserve those readers unless a breaking change is explicitly requested. When a writer moves to
+a new format, freeze the last files of the old writer as release-produced fixtures first.
 
 ### Change Python bindings
 
@@ -840,15 +848,18 @@ Centroids, data, and queries compared by the same estimator must be in the same
 
 Rotator state is part of persisted index state. Loading an index must restore the exact rotation
 used at construction; generating a new random rotator produces silently incorrect distances.
-Honor stored padded dimensions instead of recomputing them from current defaults. IVF `RABQIDX1`
-v1 and v2 files explicitly store this field; preserve it for both versions. Only legacy unversioned
+Honor stored padded dimensions instead of recomputing them from current defaults. Index format 2
+and IVF `RABQIDX1` v1 and v2 files explicitly store this field; preserve it. Only legacy unversioned
 quantized IVF files and old raw v1 files omit it and infer historical 64-dimension padding.
 Unreleased pad16 files whose stored dimension is not divisible by 32 are rejected.
 
 ### Distance conventions
 
 The library supports `METRIC_L2` and `METRIC_IP`. Some internal helpers represent inner-product
-distance as `1 - dot_product`, and estimator correction terms differ between metrics. Do not reuse
+distance as `1 - dot_product`, and estimator correction terms differ between metrics.
+`METRIC_COSINE` exists only at the ANN index boundary: an index stores `METRIC_IP` internally,
+normalizes inputs where they enter (`src/index/cosine.hpp`), and records the metric in its file.
+Quantizers, FastScan, clustering and the IVF initializers keep rejecting it. Do not reuse
 an L2 norm/correction formula for IP merely because both paths compile. Test ordering and returned
 distance semantics separately.
 
@@ -875,9 +886,9 @@ updates or persistence. These markers affect live search as well as saved files.
 IVF-RaBitQ `remove` stores a removed point as `f_add = +inf` inside the existing batch data, so it needs no
 discriminator. No index written before `remove` existed has an infinite `f_add`, so nothing in an old
 file is reinterpreted. In the legacy file formats readable by release 0.5.0, that reader also
-excludes removed points. This compatibility of the removal marker does not extend to the current
-writer's `RABQIDX1` v2 format: older releases cannot read current saves, even when resaving a loaded
-legacy index. Keep this convention when touching IVF-RaBitQ code:
+excludes removed points. This compatibility of the removal marker does not extend to index
+format 2, which the current writer uses: releases up to 0.8 cannot read current saves, even when
+resaving a loaded legacy index. Keep this convention when touching IVF-RaBitQ code:
 
 - Every estimator and scan path must turn `f_add = +inf` into an infinite estimated distance and an
   infinite lower bound, and never into NaN. A NaN distance would enter the result buffer.

docs/docs/index/hnsw.md

@@ -70,7 +70,7 @@ see [C++ integration](../cpp_integration.md).
 | `nbits` / C++ `total_bits` | 1–9 total bits including the sign bit. Python's keyword-only `nbits` defaults to 7; C++ requires it. Raw-vector storage is not supported. |
 | `num_clusters` | Python keyword, default 16: centroids `train` or the first `add` learns, or `set_centroids` supplies; the first `add` caps it at its row count. |
 | `M`, `ef_construction` | Graph connectivity and construction search budget; Python defaults are 16 and 200, and the index reports both as properties. `M` must be at least 2 and is capped at 10,000; `ef_construction` is raised to at least `M`. The base layer keeps up to `2 * M` edges per node. |
-| `metric`, `seed` | Python defaults are `"l2"` and 100; `"ip"` is also supported. `seed` was called `random_seed` before 0.8. |
+| `metric`, `seed` | Python defaults are `"l2"` and 100; `"ip"` and [`"cosine"`](usage.md#the-cosine-metric) are also supported. `seed` was called `random_seed` before 0.8. |
 | `max_elements` | Python reports the current capacity, which `add` grows geometrically (at least doubling) and `resize` can preallocate. C++ `add` never grows it. |
 | `k`, `ef` | Positive result count and search window. `k` may exceed the stored point count; Python pads missing results, while C++ returns fewer pairs. Omitted, `None` or zero `ef` uses `max(k, 64)` in Python, positive `ef` is raised to at least `k`, and `ef` above the stored point count is clamped to it. |
 
@@ -82,7 +82,7 @@ available CPU, so **default builds are not reproducible**: pass `num_threads=1`
 still change results. Loading restores the saved rotation.
 
 Results use squared L2 or `1 - dot` estimates, ordered from smallest to largest.
-Python fills missing results with ID `2**32 - 1` and distance `+inf`; C++ returns
+Python returns `int64` IDs and fills missing results with ID `-1` and distance `+inf`; C++ returns
 only the available pairs. See [result handling](../quick_start.md#read-the-results)
 and the shared [input, count and concurrency contracts](usage.md).
 
@@ -271,8 +271,8 @@ removed = index.remove(new_ids[:10])   # IDs in [0, num_stored_points)
 A removed point keeps its codes, its links, and its place in the graph. Search
 still walks through it and `add` may link new points to it, which keeps the graph
 connected, as in hnswlib. Search never returns it, so a query can get fewer than
-`k` results once points are removed: in Python the missing slots hold `kPidMax`
-(`2**32 - 1`) with an infinite distance. Removed points still count in
+`k` results once points are removed: in Python the missing slots hold ID `-1`
+with an infinite distance. Removed points still count in
 `num_stored_points()` and cannot be restored. Removal requires the same
 exclusive access as other updates.
 
@@ -316,6 +316,10 @@ through 0.5.1 check every cluster ID on `load` and reject the file as invalid
 instead of returning removed points. Releases before 0.3.7 do not check cluster
 IDs, so do not open such a file with them.
 
+`save` writes [index format 2](https://github.com/VectorDB-NTU/RaBitQ-Library/blob/main/docs/formats.md#index-format-2), which also records the construction seed, so
+`seed` survives a reload. Releases up to 0.8 cannot read these files; every earlier
+file still loads, without a recorded seed, and saving it writes format 2.
+
 New HNSW-RaBitQ indexes pad dimensions to multiples of 32 for all supported bit widths.
 Loading preserves the saved padded dimension and rotation, so older indexes
 using multiples of 64 remain readable. Older library versions cannot load new