Skip to content

Pipe: Handle transient resource failures locally - #18381

Open
Caideyipi wants to merge 1 commit into
apache:masterfrom
Caideyipi:fix/pipe-resource-failure-handling
Open

Pipe: Handle transient resource failures locally#18381
Caideyipi wants to merge 1 commit into
apache:masterfrom
Caideyipi:fix/pipe-resource-failure-handling

Conversation

@Caideyipi

Copy link
Copy Markdown
Collaborator

Description

Handle transient resource failures locally

  • Add PipeStopStrategy.accept(Exception, TSStatus) as the shared decision point for sink failures.
  • Treat nested client-pool borrow failures, local Pipe memory allocation failures, and temporary receiver unavailability as transient resource failures.
  • Retry these failures locally without reporting a critical runtime exception or stopping the Pipe.
  • Preserve nested causes in the IoTConsensusV2 sync sink so client-pool failures remain classifiable.

Expose recent resource failures

  • Track failures per Pipe in a fixed one-minute rolling window using bounded one-second buckets.
  • Report per-node snapshots through optional Thrift heartbeat fields and aggregate them on the ConfigNode.
  • Expire stale node snapshots after one minute.
  • Add RecentFailures to tree-model SHOW PIPES and recent_failures to information_schema.pipes.
  • Render values deterministically, for example {memory_timeout=15, network_timeout=10}, and render {} when empty.

Compatibility and verification

The new Thrift fields are optional, so mixed-version heartbeats and show-pipe responses continue to parse without them.

Verified with:

  • 9 focused node-commons tests
  • 10 focused DataNode tests
  • 17 focused ConfigNode tests
  • DataNode, ConfigNode, and node-commons test compilation for both English and Chinese locales
  • clean installation of the modified Thrift modules
  • Spotless, Checkstyle, and git diff --check

The updated IoTDBDatabaseIT assertion was compiled. Its targeted execution was attempted, but the reactor was stopped before reaching the integration test by host Windows page-file exhaustion in an unrelated upstream test module.


This PR has:

  • been self-reviewed.
    • concurrent read
    • concurrent write
    • concurrent read and write
  • added comments explaining the why and the intent of the code wherever it would not be obvious.
  • added unit tests or modified existing tests to cover new code paths.

Key changed/added classes (or packages if there are too many classes) in this PR
  • PipeStopStrategy
  • PipeRecentFailureCounter
  • PipeAbstractSinkSubtask
  • IoTDBDataRegionAsyncSink
  • PipeTemporaryMetaInAgent / PipeTemporaryMetaInCoordinator
  • Pipe heartbeat and SHOW PIPES response paths

Comment on lines +74 to +80
final Map<String, Long> sanitizedFailures = new HashMap<>();
recentFailures.forEach(
(failureType, count) -> {
if (Objects.nonNull(failureType) && Objects.nonNull(count) && count > 0) {
sanitizedFailures.put(failureType, count);
}
});

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the necessity of this step?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants