Skip to content

fix(tracing): serialize concurrent trace finalization - #6713

Draft
CTWalk wants to merge 1 commit into
crewAIInc:mainfrom
CTWalk:fix/serialize-trace-finalization
Draft

fix(tracing): serialize concurrent trace finalization#6713
CTWalk wants to merge 1 commit into
crewAIInc:mainfrom
CTWalk:fix/serialize-trace-finalization

Conversation

@CTWalk

@CTWalk CTWalk commented Jul 29, 2026

Copy link
Copy Markdown

Summary

TraceBatchManager.finalize_batch() performs an irreversible sequence — snapshot
the event buffer, send it, finalize the backend batch, then clear local state —
but only the backend-finalization step ran under a lock. Two overlapping callers
could therefore send the same event snapshot before either one cleared the
buffer.

This change holds _finalize_lock across the whole public finalize_batch()
transaction. The lock becomes an RLock rather than moving the existing one,
because FirstTimeTraceHandler.handle_execution_completion() calls
_finalize_backend_batch() directly and must keep its current serialization.

The event bus can execute lifecycle handlers concurrently, while the singleton
trace listener calls finalize_batch() from terminal crew handlers and its
signal-flush handler. A second finalizer now serializes behind the first,
including its existing backend I/O, instead of racing it.

This fixes duplicate sends from overlapping local finalizers. It does not claim
exactly-once delivery across an ambiguous network response. There is no public
API or dependency change.

Reproduction

lib/crewai/tests/tracing/test_tracing.py::TestTraceListenerSetup::test_finalize_batch_sends_events_once_when_called_concurrently
races two public finalizers around a successful event send. On main it fails:

AssertionError: Expected 'send_trace_events' to have been called once. Called 2 times.

The test captures worker exceptions, so a thread error cannot let it pass
silently.

Verification

uv run pytest lib/crewai/tests/tracing/test_tracing.py -q
# 40 passed, 1 skipped

uv run pytest lib/crewai/tests/test_flow_conversation.py -q
# 47 passed, 21 skipped

uv run ruff check lib/crewai/src/crewai/events/listeners/tracing/trace_batch_manager.py
uv run mypy lib/crewai/src/crewai/events/listeners/tracing/trace_batch_manager.py
# clean

The new test was also run twelve times in isolation with no flake.

AI assistance disclosure

This PR was prepared with AI coding assistance and reviewed by the contributor
before submission. GitHub does not permit this fork contributor to apply labels
in the upstream repository, so maintainer assistance is needed to add the
required llm-generated label.

@coderabbitai

coderabbitai Bot commented Jul 29, 2026

Copy link
Copy Markdown

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 0aeab608-0587-4ad6-8fbf-b15337601c0a

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@CTWalk

CTWalk commented Jul 29, 2026

Copy link
Copy Markdown
Author

AI assistance is disclosed in the PR body. GitHub does not grant this fork contributor permission to set upstream labels; could a maintainer please add the required llm-generated label?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant