Repository navigation
Run the hashing embedder off the event loop - #7
Merged
Merged
Conversation
HashingEmbedder.embed_documents was async but its body was a plain list comprehension over the sync self.embed(text), so a document with hundreds of chunks kept the asyncio loop busy for the whole embed. The ingest pipeline runs extract in a thread and awaits embed_documents right after, so the API worker was blocked on the CPU-bound half. Dispatch the inner loop to a worker thread the same way extraction does. embed_query is a single call and stays on the loop. Fixes #6
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What was wrong:
HashingEmbedder.embed_documentsin backend/src/docmind/embeddings/hashing.py:61 wasasync defbut its body was a plain list comprehension over the syncself.embed(text). Hashing a document with hundreds of chunks kept the asyncio loop busy for the whole embed. The ingest pipeline in backend/src/docmind/ingestion/pipeline.py:106-113 runsextractin a worker thread and awaitsembed_documentsright after it, so the API worker was still blocked on the CPU-bound half.The fix: dispatch the inner loop to a worker thread with
asyncio.to_thread, same as extraction.embed_queryis a single call and stays on the loop.How it was tested: new
test_embed_documents_yields_to_the_event_loopin backend/tests/test_retrieval.py; a concurrent ticker thatawait asyncio.sleep(0)in a loop had to advance more than five times whileembed_documentsran on 40 long inputs. It stayed at 0 before the fix and reaches hundreds after. Locally ran what CI runs: ruff check, ruff format --check, mypy src, pytest --cov (116 passed), and on the frontend npm ci, lint, typecheck, test (34 passed), build:pages.Fixes #6