Sync fork with Zilliz main

.gitignore

@@ -10,8 +10,11 @@ build/
 venv/
 .venv/
 .idea/
-results/
 logs/
+vectordb_bench/results/cloudleaderboard/
+
+# Worktrees
+.worktrees/
 
 # AI rules
 CLAUDE.md

Makefile

@@ -1,8 +1,10 @@
-lint:
-	PYTHONPATH=`pwd` python3 -m black vectordb_bench --check
-	PYTHONPATH=`pwd` python3 -m ruff check vectordb_bench
-
 unittest:
 	PYTHONPATH=`pwd` python3 -m pytest tests/test_dataset.py::TestDataSet::test_download_small -svv
 
-.PHONY: lint unittest
+format:
+	PYTHONPATH=`pwd` python3 -m black vectordb_bench
+	PYTHONPATH=`pwd` python3 -m ruff check vectordb_bench --fix
+
+lint:
+	PYTHONPATH=`pwd` python3 -m black vectordb_bench --check
+	PYTHONPATH=`pwd` python3 -m ruff check vectordb_bench

docs/release/2026-05-cloud-leaderboard.md

@@ -0,0 +1,170 @@
+# VectorDBBench Cloud Leaderboard Release Note
+
+May 2026
+
+The VectorDBBench Cloud Leaderboard moves beyond a single raw-throughput ranking. It evaluates managed vector databases around the behaviors production teams have to plan for: ingest readiness, payload-aware search, tenant-shaped workloads, cold latency, and cost at practical QPS targets.
+
+## Why we need a new leaderboard now
+
+The vector database market has moved past the "highest QPS wins" phase. Production teams choosing a managed vector database also care about budget, data freshness, tail latency, recall, metadata payloads, tenant isolation, and operational predictability.
+
+The existing VectorDBBench leaderboard remains useful for comparing baseline search performance across systems. But cloud buyers ask a wider set of questions:
+
+- When does newly inserted data become searchable?
+- When is it fully indexed?
+- What happens when search returns metadata or vectors instead of only IDs?
+- What happens when traffic is split across many tenants?
+- What does each reachable QPS tier cost?
+
+The Cloud Leaderboard is designed around those questions. It keeps performance visible, but puts it next to the readiness, payload, tenant, cold-start, and cost signals that determine what a customer can safely deploy.
+
+## What the Cloud Leaderboard changes
+
+The Cloud Leaderboard is a production cloud decision layer, not a replacement for the original raw-performance board. The main change is that benchmark cases now model cloud operating concerns directly instead of treating all products as simple warm search engines.
+
+The new cases add:
+
+- Insert readiness measurement, including client insert completion, searchable delay, and indexed delay.
+- Explicit response payload profiles: IDs only, scalar label metadata, or vector values.
+- Cloud cold-latency measurement for the first search path after idle or cache-cold conditions.
+- Multi-tenant search, where data is split into deterministic tenant labels or namespaces and queries are routed by tenant.
+- Cost-oriented interpretation, so raw QPS can be read together with monthly cost and readiness constraints.
+
+This matters because a top-line QPS table can hide important tradeoffs. A system can look strong on peak throughput while losing ground on recall, p99 latency, cold-start behavior, payload cost, or sustained cost at the same target QPS.
+
+## Who we tested this round
+
+This round focuses on three popular cloud vector databases:
+
+- Zilliz Cloud, including tiered and fixed-capacity configurations.
+- turbopuffer, including normal, pinned, and backpressure-related configurations where applicable.
+- Pinecone serverless.
+
+The tested matrix is intentionally cloud-oriented. It compares managed products and managed-service modes rather than only local or self-hosted engine behavior.
+
+## The new tests we added
+
+Version 2 adds four cloud-oriented cases in VectorDBBench. Each case is designed to expose a production behavior that a plain QPS benchmark can miss.
+
+### CloudInsertCase
+
+**Purpose.** CloudInsertCase measures write readiness, not just client-side insert speed. This is important for backfills, migrations, daily refreshes, and release workflows where a team needs to know when newly written vectors can safely take traffic.
+
+**How it works.** The case loads the dataset with `ConcurrentInsertRunner`, records insert completion time and rows per second, then polls the database until inserted data is fully searchable and fully indexed. The resulting metric separates:
+
+- `insert_completion_seconds`
+- `insert_rows_per_second`
+- `searchable_after_insert_seconds`
+- `indexed_after_searchable_seconds`
+
+Example: run LAION 100M insert readiness on Zilliz Cloud with a 10k batch size.
+
+```bash
+vectordbbench zillizautoindex \
+  --case-type CloudInsertCase \
+  --uri "$ZILLIZ_URI" \
+  --token "$ZILLIZ_TOKEN" \
+  --collection-name cloud_insert_laion100m_bs10k \
+  --cloud-insert-batch-size 10000 \
+  --load-concurrency 16 \
+  --skip-search-serial \
+  --skip-search-concurrent \
+  --task-label cloud-insert-zilliz-12cu
+```
+
+### CloudPayloadSearchCase
+
+**Purpose.** CloudPayloadSearchCase measures search when the response body resembles production traffic. Many applications return more than vector IDs: they return scalar metadata, labels, or the vector values themselves. That response payload can change throughput, latency, and even product ranking.
+
+**How it works.** The case extends the normal performance case with an explicit `payload_profile`. Supported profiles are:
+
+- `ids_only`
+- `scalar_label`
+- `vector`
+
+The case can also run unfiltered search, integer-filter search through `--cloud-filter-rate`, or scalar-label filter search through `--cloud-label-percentage`. It records QPS, latency, recall where applicable, and estimated response payload bytes per query.
+
+When `payload_profile` is `scalar_label`, VectorDBBench materializes scalar label data even for unfiltered runs. This keeps the loaded schema aligned with the requested response payload instead of only loading labels for scalar-label filter runs.
+
+Example: run vector-payload search on Pinecone with a highly selective integer filter.
+
+```bash
+vectordbbench pinecone \
+  --case-type CloudPayloadSearchCase \
+  --api-key "$PINECONE_API_KEY" \
+  --index-name "$PINECONE_INDEX" \
+  --payload-profile vector \
+  --cloud-filter-rate 0.001 \
+  --k 100 \
+  --num-concurrency 60,80 \
+  --concurrency-duration 30 \
+  --task-label cloud-payload-pinecone-vector-int-filter-0-1p
+```
+
+### CloudMultiTenantSearchCase
+
+**Purpose.** CloudMultiTenantSearchCase models SaaS-shaped traffic. Instead of treating the dataset as one flat global collection, it splits records across many tenants and routes each query to a tenant. This highlights products whose namespace, partition-key, or tenant-filter paths behave differently from single-tenant search.
+
+**How it works.** The case defaults to the Cohere 10M dataset and assigns each row to a deterministic tenant by `row_id % tenant_count`. During search, queries are routed to the corresponding tenant label or namespace. The case supports the same payload profiles and optional filter modes as payload search.
+
+Tenant routing labels and scalar payload labels are separate concepts. A multi-tenant run can route by tenant while still storing and returning scalar-label payload data when `payload_profile` is `scalar_label`, and scalar-label filters continue to use the scalar label field rather than the tenant routing field.
+
+TurboPuffer tenant namespace cache warmup is explicit. By default, `CloudMultiTenantSearchCase` does not warm tenant namespaces during `optimize()`; use `--multitenant-warmup-policy all` only when the benchmark should model proactively warmed tenant namespaces.
+
+Example: run 1,000-tenant IDs-only search on turbopuffer.
+
+```bash
+vectordbbench turbopuffer \
+  --case-type CloudMultiTenantSearchCase \
+  --dataset-with-size-type "Large Cohere (768dim, 10M)" \
+  --api-key "$TURBOPUFFER_API_KEY" \
+  --region aws-us-east-1 \
+  --namespace vdbbench_mt_seed \
+  --multitenant-namespace-prefix vdbbench_mt_ \
+  --tenant-count 1000 \
+  --tenant-prefix tenant_ \
+  --tenant-id-width 4 \
+  --payload-profile ids_only \
+  --num-concurrency 40,60,80 \
+  --concurrency-duration 30 \
+  --task-label cloud-multitenant-turbopuffer-1000t
+```
+
+### CloudColdLatencyCase
+
+**Purpose.** CloudColdLatencyCase measures first-query and cold-path latency that warm benchmark loops can hide. This matters for serverless products, storage-tiered products, idle workloads, and customer-facing applications where the first query after an idle period is visible to users.
+
+**How it works.** The case is intentionally search-only and must run against an existing collection that has already become cold according to the product's cache and storage behavior. It rejects `drop_old` and `load` stages because insert-then-immediately-search runs can leave caches, indexing paths, or vendor warmup APIs in an ambiguous state. `ColdWarmSearchRunner` runs serial searches in cold and warm passes. It records cold-latency details in `additional_parameters["cold_latency"]` and also records payload profile and estimated payload bytes per query.
+
+Example: run a pinned turbopuffer cold-latency test with scalar-label payloads.
+
+```bash
+vectordbbench turbopuffer \
+  --case-type CloudColdLatencyCase \
+  --skip-drop-old \
+  --skip-load \
+  --api-key "$TURBOPUFFER_API_KEY" \
+  --region aws-us-east-1 \
+  --namespace cloud_cold_latency_scalar_label \
+  --pin-namespace \
+  --pin-replicas 2 \
+  --payload-profile scalar_label \
+  --cloud-cold-query-count 1000 \
+  --skip-search-concurrent \
+  --task-label cloud-cold-latency-turbopuffer-pinned-scalar-label
+```
+
+## Caveats
+
+This release note introduces the new Cloud Leaderboard direction; it is not the full benchmark report. Detailed tables, raw JSON artifacts, pricing worksheets, and edge-case analysis should live in the benchmark report or external result artifact repository.
+
+Important caveats:
+
+- Pricing changes over time. Cost charts need a pricing date, region, and configuration assumptions.
+- Managed-service configuration can materially change results, especially for serverless scaling, pinned replicas, capacity units, and storage-tiering modes.
+- "Fully indexed" and "fully searchable" readiness may be exposed differently by each vendor, so the implementation must document how each status is detected or inferred.
+- The current multi-tenant case uses deterministic tenant assignment and uniform tenant routing. It does not represent every SaaS tenant distribution.
+- Multi-tenant routing labels or namespaces are not equivalent to scalar payload labels. Benchmark clients must keep those fields separate when a run combines tenant routing with scalar-label payload or filter behavior.
+- Cold latency depends on cache state, idle window, replica pinning, storage architecture, and service warmup behavior. The idle and warmup rules must stay strict between products.
+- Payload search rankings are workload-specific. IDs-only, scalar-label, vector-return, integer-filter, and label-filter runs can produce different winners.
+- Cost Pareto results must be read together with recall, latency, payload profile, and readiness constraints rather than as a standalone ranking.

docs/release/2026-06-full-text-search.md

@@ -0,0 +1,118 @@
+# VectorDBBench Full Text Search Release Note
+
+June 2026
+
+Full text search support adds BM25-style text retrieval workloads to VectorDBBench. The goal is to compare database full text search paths with the same benchmark harness used for vector workloads: repeatable datasets, explicit load/search stages, structured result JSONs, and frontend result visualization.
+
+## Context
+
+VectorDBBench has historically focused on dense vector search and vector-oriented cloud cases. That leaves a gap for systems that also expose native text retrieval, sparse search, or BM25 ranking. Users evaluating retrieval systems often need to compare text-only retrieval before deciding whether to use dense vector, sparse vector, hybrid, or reranking layers.
+
+The Full Text Search benchmark covers that text-only layer. It measures end-to-end behavior for indexing raw text, optimizing the backend, searching with BM25-style ranking, and validating retrieval quality against semantic relevance labels from `ir_datasets`.
+
+The benchmark also records payload behavior. Some applications return only document IDs, while others return text fields in the search response. VectorDBBench models both paths through explicit payload profiles so throughput and latency can be interpreted together with the response shape.
+
+## Who we tested this round
+
+This round focuses on full text search support across the following backends:
+
+- Milvus, using its full text search BM25 path.
+- Zilliz Cloud, using the cloud full text search path.
+- Elasticsearch, using its BM25 text search path.
+- OSS OpenSearch, using its BM25 text search path.
+- Vespa, using BM25 ranking over indexed text fields.
+- turbopuffer, using its full text search namespace path.
+
+The benchmark is designed to keep the workload shape consistent while still recording backend-specific behavior. BM25 parameters and analyzer settings are controlled through the backend case config or CLI flags, so users can run either product-default comparisons or explicitly tuned comparisons.
+
+## The new tests we added
+
+### FullTextSearchPerformance
+
+**Purpose.** FullTextSearchPerformance measures BM25-style full text search as a first-class benchmark case. It answers the baseline question: after a backend indexes the same text corpus, what QPS, latency, recall, MRR, and NDCG does it deliver for text queries?
+
+**How it works.** The case loads raw text documents, builds the backend text index, runs the backend optimize path, executes optional serial quality checks, and then runs concurrent search. The result metric records load duration, insert duration, optimize duration, QPS, serial latency, concurrent latency, recall, MRR, NDCG, payload profile, inserted count, and explicit backend parameters.
+
+Example: run MS MARCO small on Milvus with IDs-only responses.
+
+```bash
+vectordbbench milvusfts \
+  --case-type FTSBm25Performance \
+  --dataset-with-size-type "MS MARCO Small (100K documents)" \
+  --uri "$MILVUS_URI" \
+  --payload-profile ids_only \
+  --load-concurrency 0 \
+  --num-concurrency 40,80 \
+  --task-label fts-milvus-msmarco-small-ids
+```
+
+### Full text search datasets and semantic ground truth
+
+**Purpose.** Full text search quality should measure whether a backend returns documents that are semantically relevant to the query, not only whether it reproduces one particular BM25 implementation's exact ranking. This aligns the benchmark with common IR evaluation practice and makes recall, MRR, and NDCG comparable across backend analyzers and ranking implementations.
+
+**How it works.** FTS datasets provide raw text for document insertion and query execution through `ir_datasets`. VectorDBBench loads positive qrels from the same source and uses them as semantic ground truth for serial quality checks. BM25 and analyzer settings are not loaded from dataset manifests; use backend config or CLI flags when a run needs explicit `k1`, `b`, analyzer, or backend-specific search parameters.
+
+Example: use a larger dataset while keeping the same FTS case type.
+
+```bash
+vectordbbench elasticcloudhnsw \
+  --case-type FTSBm25Performance \
+  --dataset-with-size-type "HotpotQA Medium (1M documents)" \
+  --host "$ELASTIC_HOST" \
+  --port "$ELASTIC_PORT" \
+  --password "$ELASTIC_PASSWORD" \
+  --payload-profile ids_only \
+  --load-concurrency 0 \
+  --num-concurrency 40,80 \
+  --task-label fts-elastic-hotpotqa-medium-ids
+```
+
+### Payload-aware full text search
+
+**Purpose.** Payload-aware FTS measures the cost of returning text fields, not only document IDs. This matters for applications where search results immediately include snippets, source text, or other fields needed by downstream ranking and display layers.
+
+**How it works.** The FTS case supports `payload_profile` values including `ids_only` and `text`. IDs-only runs are the recall baseline because they can run serial search against the ground truth without paying text-return overhead. Text payload runs measure concurrent search throughput and latency for larger response bodies; they can skip serial recall when the same indexed namespace or collection has already been validated by the IDs-only run.
+
+Example: run Vespa with text payload responses and concurrent search only.
+
+```bash
+vectordbbench vespa \
+  --case-type FTSBm25Performance \
+  --dataset-with-size-type "MS MARCO Medium (1M documents)" \
+  --uri "$VESPA_URI" \
+  --port "$VESPA_PORT" \
+  --payload-profile text \
+  --load-concurrency 0 \
+  --skip-search-serial \
+  --num-concurrency 40,80 \
+  --task-label fts-vespa-msmarco-medium-text
+```
+
+## Result artifacts
+
+Committed FTS result JSONs live under `vectordb_bench/results/FullTextSearch/<Backend>/`. Published artifacts are consolidated so each backend directory contains one backend-level JSON with multiple case results in its `results` list, instead of a pile of one-case files. That keeps the release payload readable while preserving the per-dataset and per-payload records needed by the dashboard.
+
+Use the result collector to consolidate split FTS run outputs before committing published examples:
+
+```bash
+python -m vectordb_bench.backend.result_collector \
+  vectordb_bench/results/FullTextSearch \
+  --merge-by-db \
+  --task-label fts_standard \
+  --replace
+```
+
+The default collector behavior still groups normal benchmark files by `run_id`. The `--merge-by-db` mode is intended for curated FTS result publication, where each backend should present the latest benchmark matrix as a single artifact. The Full Text Search frontend reads the latest backend-level result file from each backend directory and displays the committed matrix by backend, dataset, payload profile, load duration, QPS, recall, and latency.
+
+## Caveats
+
+This release note introduces the FTS benchmark path; it is not a complete benchmark report. Detailed raw outputs, machine setup, backend deployment scripts, and rerun notes should live in separate experiment reports when needed.
+
+Important caveats:
+
+- Semantic relevance labels are not the same as exact BM25 implementation correctness. A high quality score means the backend retrieved judged-relevant documents, not that it matched a specific analyzer or scorer bit-for-bit.
+- Analyzer behavior can materially affect recall and ranking. Tokenization, lowercase filters, stop words, stemming, token length limits, and field normalization should be recorded with the benchmark run when they are configured explicitly.
+- BM25 parameter support differs across products. Some backends expose `k1` and `b`, some expose average field length controls, and some compute or hide those values internally. Use backend config or CLI flags for explicit comparisons; otherwise the run represents product-default behavior.
+- IDs-only and text payload runs answer different questions. IDs-only is the cleanest recall and throughput baseline; text payload runs expose response-size overhead.
+- Load duration includes backend-specific insert and optimize behavior. Products may differ in whether optimize means force merge, compaction, warmup, or index deployment readiness.
+- Result JSONs under `vectordb_bench/results/FullTextSearch` are curated examples for the frontend. They should not be treated as the full historical experiment archive.

install/requirements_py3.11.txt

@@ -5,6 +5,8 @@ pinecone
 weaviate-client
 elasticsearch==8.16.0
 opensearch-py
+boto3
+requests-aws4auth
 pgvector
 pgvecto_rs[psycopg3]>=0.2.1
 sqlalchemy
@@ -14,17 +16,20 @@ pytz
 streamlit-autorefresh
 streamlit>=1.23.0
 streamlit_extras
+tornado>=6.0
 tqdm
 s3fs
 psutil
 polars
 plotly
 environs
-pydantic<v2
+pydantic>=2.0,<3
 scikit-learn
-pymilvus
+pymilvus>=2.6.15,<3.0.0
 clickhouse_connect
 pyvespa
 mysql-connector-python
+PyMySQL
 packaging
-hdrhistogram>=0.10.1
+hdrhistogram>=0.10.1
+ir_datasets