[S-h-u-b-h-1]

Summary

Add opt-in concurrent inference for CPU-only ONNX text embeddings using a shared ONNX Runtime session.

• Keep serial inference as the default.
• Execute independent batches concurrently with bounded in-flight work.
• Preserve document ordering and numerical outputs.
• Reject incompatible execution providers and multiprocessing combinations.
• Cancel queued work during early iterator closure.

Performance

Related to #535.

Measured on Apple Silicon, macOS 27.0.1, FastEmbed 0.9.0, ONNX Runtime 1.31.0, using BAAI/bge-small-en-v1.5.

For batch size 16 with eight concurrent workers and one ONNX thread:

• 512 short documents: 47–58% higher throughput in two cycles.
• 128 medium documents: 9–57% higher throughput.
• 128 long documents: results varied from 19% slower to 11% faster.

Results are workload-dependent. Concurrent inference is not enabled automatically.

Validation

• 19 focused regression and integration tests passed.
• Ruff lint and formatting checks passed.
• Concurrent embeddings match serial embeddings in the tested configurations.
• Existing default inference behavior is unchanged.

This is an optional throughput improvement, not a complete resolution of #535.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

• Source mode: Auto
• Repository: Unavailable
• Branch: Unavailable
• File: No configuration file from this repository applied
• Commit: Unavailable
• Layers: Organization UI
• Review profile: CHILL
• Plan: Advanced
• Run ID: 85a6a9a1-0804-479f-a24b-3d96566b9848

📥 Commits

Reviewing files that changed from the base of the PR and between 6dab7f9 and 50c3dde.

📒 Files selected for processing: 3

• fastembed/text/onnx_text_model.py
• fastembed/text/text_embedding.py
• tests/test_cpu_concurrent_batches.py

Included review availability: This review used your included allowance. Your plan provides up to 8 included reviews per hour; 7 remain after this review.

📝 Walkthrough

OnnxTextModel adds an inference_workers setting, defaulting to 1. Values above 1 require a CPU-only ONNX session and cannot be combined with parallel. For eligible inputs, batches run concurrently with bounded outstanding work and results yield in submission order. Early generator closure cancels pending tasks and waits for running tasks. The changes also update type annotations and document the setting. New tests cover validation, execution behavior, cleanup, and numerical agreement.

Priority: ➖ Normal

Estimated code review effort: 3 (Moderate) | ~25 minutes

Change: Feature

Merge Risk: ⚪ Minimal · up to 50c3d

No actionable issue remains from this review. The opt-in concurrent path is mergeable after normal checks.

Pre-merge checks: 4 · 1

Failed checks: 1 warning

Check name	Status	Explanation	Resolution
Docstring Coverage		Docstring coverage is 10.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 30 functions across 3 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

Passed checks: 4 passed

Check name	Status	Explanation
Title check		The title "feat(text): add opt-in concurrent CPU embedding batches" matches the main change in the changeset. The raw summary shows an opt-in inference_workers setting in OnnxTextModel that runs batch…
Description check		The description covers the changeset. It states that it adds opt-in concurrent inference for CPU-only ONNX text embeddings using a shared session, keeps serial inference as the default, bounds in-flig…
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.

Full details: Title check

Explanation

The title "feat(text): add opt-in concurrent CPU embedding batches" matches the main change in the changeset. The raw summary shows an opt-in inference_workers setting in OnnxTextModel that runs batches on a thread pool over a shared CPU-only ONNX session, with serial behavior kept as the default. The title is short, specific, and describes the primary change without noise.

Full details: Description check

Explanation

The description covers the changeset. It states that it adds opt-in concurrent inference for CPU-only ONNX text embeddings using a shared session, keeps serial inference as the default, bounds in-flight work, preserves order, rejects incompatible execution providers and multiprocessing combinations, and cancels queued work on early iterator closure. These points match the file summaries for onnx_text_model.py, text_embedding.py, and the new concurrent-batch tests. It also includes performance and validation notes.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create a new PR

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}