[QST] CAGRA self-search QPS seems low — is this expected on L4 with SINGLE_CTA/search_width=1?

[marcelo-cjl]

What is your question?

I benchmarked CAGRA native cosine search with a small self-search program and measured a QPS that seems lower than I expected from the cuVS ANN benchmarks. Is ~43.5k QPS a reasonable number for these parameters on an L4, or is something off? And which parameters would have the biggest impact on QPS?

Update (2026-08-26): The original reproducer used raw, unnormalized Cohere vectors with L2Expanded. I corrected the benchmark to use native cuVS CosineExpanded for both index construction and search, and updated the measurements below. The script still passes the input vectors to cuVS unchanged and does not pre-normalize them.

Reproduction

Minimal reproducer (self-contained script that builds cuVS from source and runs a tiny C++ benchmark using cuvs::neighbors::cagra::build / cuvs::neighbors::cagra::search):

https://github.com/marcelo-cjl/cuvs/blob/33ac4cfef430934cf4f4c583f942d0b429736133/cpp/bench/cuvs_cagra_self_search/cuvs_cagra_self_search.sh

Environment:

• Machine: AWS g6.4xlarge
• GPU: NVIDIA L4 23 GB
• Driver: 560.35.05
• cuVS: built from source at commit 33ac4cfe (Release, -DBUILD_TESTS=OFF -DBUILD_C_LIBRARY=OFF -DBUILD_CAGRA_HNSWLIB=OFF -DBUILD_MG_ALGOS=OFF -DCUVS_COMPILE_DYNAMIC_ONLY=ON, CMAKE_CUDA_ARCHITECTURES=native)

Dataset: VectorDBBench Cohere Medium, converted to .fbin (1,000,000 rows x 768 dims, float). This is a self-search workload: the same training file is used to build the index and as the query input. self_recall_at_1 is the fraction of queries where the top-1 label equals the query's own row id.

Index / search parameters:

Parameter	Value
dtype	float
metric	CosineExpanded
graph_degree	64
intermediate_graph_degree	128
topk	1
itopk_size	32
search_width	1
max_iterations	0
search_algo	SINGLE_CTA

Observed results from two searches against the same rebuilt cosine index:

run	stage	rows	dim	search_ms	qps	self_recall_at_1
1	search	1,000,000	768	22,993.9	43,489.8	0.995099
2	search	1,000,000	768	22,996.2	43,485.5	0.995099

search_ms is measured with CUDA events around a single cuvs::neighbors::cagra::search call (1M queries in one batch); qps = rows * 1000 / search_ms.

Expected behavior

I was expecting higher throughput based on the cuVS ANN benchmarks. Specifically:

1. Is ~43.5k QPS reasonable for native CosineExpanded, SINGLE_CTA, search_width=1, topk=1, 1M x 768 float on an L4, or does it indicate a configuration problem?
2. Which knobs have the biggest impact on QPS (e.g., multi-CTA search algorithm, larger search_width/itopk_size, half precision, query batch size)?
3. Is there an official way to run this exact native-cosine workload through cuvs-bench so I can compare against a reference number?

[tfeher]

Hi @marcelo-cjl, thank you for the questions! To confirm, the Cohere dataset mentioned here is the one which is used in VectorDBBench from Zilliz?

https://github.com/zilliztech/VectorDBBench/blob/08fabb2df7436bbaffb549f6d1144d66f14b9fb2/vectordb_bench/backend/dataset.py#L180

[sherylll]

Hi @marcelo-cjl,

To answer some of your questions, (1) it is a bit hard to tell if a certain set of parameters can guarantee a certain QPS per se, but I noticed that your recall is very high (0.999338), usually QPS falls off quickly when recall approaches 1, so you might want to trade a bit of recall for thruput; (2) in most cases you can leave cuvs to select multi/single CTA implementation, and your itopk and width is already at the bottom, so I'd suggest to try manually reducing max_iterations; (3) if you build from source with CUVS_CAGRA_ANN_BENCH target on, you should be able to find an executable cuvs/cpp/build/bench/ann/CUVS_CAGRA_ANN_BENCH, which you can use to sweep across parameters, e.g. CUVS_CAGRA_ANN_BENCH --search --override_kv=max_iterations:1:2:3..., or you can fill the JSON config file with multiple sets of configurations; you can also use the python benchmark tool. Please refer to https://docs.nvidia.com/cuvs/user-guide/benchmarking-guide for details.

[marcelo-cjl]

Hi @tfeher , thanks for checking! Yes, I used VectorDBBench’s Cohere Medium dataset (1M vectors, 768 dimensions), converted to .fbin. For this self-search benchmark, the same training vectors are used for both index construction and queries. I also updated the reproducer to use cosine (CosineExpanded) consistently.

[marcelo-cjl]

Hi @marcelo-cjl,

To answer some of your questions, (1) it is a bit hard to tell if a certain set of parameters can guarantee a certain QPS per se, but I noticed that your recall is very high (0.999338), usually QPS falls off quickly when recall approaches 1, so you might want to trade a bit of recall for thruput; (2) in most cases you can leave cuvs to select multi/single CTA implementation, and your itopk and width is already at the bottom, so I'd suggest to try manually reducing max_iterations; (3) if you build from source with CUVS_CAGRA_ANN_BENCH target on, you should be able to find an executable cuvs/cpp/build/bench/ann/CUVS_CAGRA_ANN_BENCH, which you can use to sweep across parameters, e.g. CUVS_CAGRA_ANN_BENCH --search --override_kv=max_iterations:1:2:3..., or you can fill the JSON config file with multiple sets of configurations; you can also use the python benchmark tool. Please refer to https://docs.nvidia.com/cuvs/user-guide/benchmarking-guide for details.

Hi @sherylll, thanks! With native cosine, max_iterations=16/20/24/28/32 gave 103.8k/81.2k/66.6k/56.4k/48.9k QPS and 0.9388/0.9683/0.9819/0.9890/0.9929 self-recall, respectively. This confirms that tuning max_iterations is effective. Beyond this recall-throughput trade-off, is there a way to improve QPS while maintaining roughly the same recall?

[sherylll]

Hi @marcelo-cjl, the shape of the QPS-recall curve depends on the "difficulty" of your dataset as well. For easier problems, you might observe the QPS drops slowly towards recall=1, while for harder problems, there is usually no "free lunch", and the curve might drop quickly, e.g. large batch case in Figure 10 from the original paper: https://arxiv.org/pdf/2308.15136

That being said, I think there is one thing you can try without hurting QPS at search time, which is to improve the graph build. You can use the from_dataset function and provide a high build_quality, and just use the returned parameters for build.

[marcelo-cjl]

Hi @marcelo-cjl, the shape of the QPS-recall curve depends on the "difficulty" of your dataset as well. For easier problems, you might observe the QPS drops slowly towards recall=1, while for harder problems, there is usually no "free lunch", and the curve might drop quickly, e.g. large batch case in Figure 10 from the original paper: https://arxiv.org/pdf/2308.15136

That being said, I think there is one thing you can try without hurting QPS at search time, which is to improve the graph build. You can use the from_dataset function and provide a high build_quality, and just use the returned parameters for build.

Got it. Thanks for answering. I'll try improving the graph quality.