Repository navigation
Conversation
This fixes b/510434185, "Improper Error Handling and Silent Data
Corruption via OP_REQUIRES_OK in Worker Lambdas". Some parsing
functions:(e.g., ParsePrograms, GetProgramsAndNumQubits, GetPauliSums)
used TensorFlow's OP_REQUIRES_OK macro inside worker lambdas dispatched
via ParallelFor. An error such as an unparseable protobuf could result
in OP_REQUIRES_OK emitting an error to the context but returning early
from the worker lambda. This would not stop other threads or signal the
outer function to stop execution. The outer function would return
Status::OK(), and downstream kernels would execute using uninitialized
or partially populated data (e.g., num_qubits remaining at -1), leading
to logic corruption, OOM, or crashes.
In more detail, the problem involved this bit of code in
`tensorflow_quantum/core/ops/tfq_ps_weights_from_symbols_op.cc`:
```cpp
OP_REQUIRES(context, false,
tensorflow::errors::InvalidArgument(
"A circuit contains a sympy.Symbol not found "
"in symbols!"));
```
The call of OP_REQUIRES from inside a worker thread caused 2 failures:
1. Thread-safety bug (data race): context is shared across the whole
operation and is not thread-safe. If two worker threads hit invalid
circuits at the same time, both threads would write to context's
internal error fields simultaneously without a lock.
2. Control-flow bug (`return;` exited the wrong function): Because
OP_REQUIRES uses `return;` to abort, calling it inside DoWork only
returned from that single worker's DoWork call, not from the outer
Compute method. Once the worker pool finished, Compute kept
executing even though the operation had already failed.
The change in this PR follows the recommended remediation in the
vulnerability report, namely to replace OP_REQUIRES_OK in lambdas with a
synchronized status tracker and ensure the outer function returns the
failure status.
mhucka
marked this pull request as ready for review
October 29, 2026 03:35
Contributor
There was a problem hiding this comment.
Code Review
This pull request improves thread safety and error handling during parallel execution by refactoring the 'NESTED_FN_STATUS_SYNC' macro to use a robust 'do-while' block and 'absl::MutexLock'. It updates several operations to use this macro for synchronizing thread-local statuses with a global status, and introduces comprehensive C++ and Python tests to verify correct behavior. The review feedback recommends renaming the macro-local variables in 'NESTED_FN_STATUS_SYNC' to highly unique names to prevent potential name collisions and shadowing issues.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This fixes b/510434185, "Improper Error Handling and Silent Data Corruption via OP_REQUIRES_OK in Worker Lambdas". Some parsing functions:(e.g., ParsePrograms, GetProgramsAndNumQubits, GetPauliSums) used TensorFlow's OP_REQUIRES_OK macro inside worker lambdas dispatched via ParallelFor. An error such as an unparseable protobuf could result in OP_REQUIRES_OK emitting an error to the context but returning early from the worker lambda. This would not stop other threads or signal the outer function to stop execution. The outer function would return Status::OK(), and downstream kernels would execute using uninitialized or partially populated data (e.g., num_qubits remaining at -1), leading to logic corruption, OOM, or crashes.
In more detail, the problem involved a couple of places that used this in a worker:
The call of OP_REQUIRES from inside a worker thread caused 2 failures:
Thread-safety bug (data race): context is shared across the whole operation and is not thread-safe. If two worker threads hit invalid circuits at the same time, both threads would write to context's internal error fields simultaneously without a lock.
Control-flow bug (
return;exited the wrong function): Because OP_REQUIRES usesreturn;to abort, calling it inside DoWork only returned from that single worker's DoWork call, not from the outer Compute method. Once the worker pool finished, Compute kept executing even though the operation had already failed.The change in this PR follows the recommended remediation in the vulnerability report, namely to replace OP_REQUIRES_OK in lambdas with a synchronized status tracker and ensure the outer function returns the failure status. This PR also adds some test cases.
This work was done with the help of Google Gemini.