🐛 Fix: kb import deadlock problem

[cj2026-bit]

Fix Celery Queue Starvation During Knowledge Base Ingestion

Problem

Large knowledge-base files could remain stuck in the ingesting state, and the data-processing service could appear to be hung.

The root cause was queue starvation:

1. The forward parent task was executed on forward_q.
2. It submitted multiple forward_part tasks and an aggregate_forward_parts callback.
3. The parent task waited for the chord result.
4. Before this fix, the child tasks and aggregation callback were also routed to forward_q.
5. When all forward_q workers were occupied by waiting parent tasks, the child tasks could not start, causing a deadlock.

Solution

Processing stages are now isolated into dedicated Celery queues and Worker processes:

forward_q
    └── forward_part_q
            └── forward_aggregate_q

##Verification
Multiple files uploading at the same time, 2 types of files, 1 type has single-digit chunks, the other type has 2K chunks.


After a while

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!