Skip to content

[To dev/1.3] Pipe: Handle hybrid meta progress indexes (#18331) - #18383

Open
Caideyipi wants to merge 1 commit into
apache:dev/1.3from
Caideyipi:cp-6559aa52-dev-1.3
Open

[To dev/1.3] Pipe: Handle hybrid meta progress indexes (#18331)#18383
Caideyipi wants to merge 1 commit into
apache:dev/1.3from
Caideyipi:cp-6559aa52-dev-1.3

Conversation

@Caideyipi

Copy link
Copy Markdown
Collaborator

Description

Backport of #18331 to dev/1.3.

Problem

IoTDBNonDataRegionSource assumed every non-minimum task progress was a MetaProgressIndex. Pipe task progress can also be merged into a HybridProgressIndex, and may be wrapped by a StateProgressIndex. This caused a ClassCastException while restoring a non-data-region source or calculating its remaining events during DataNode shutdown.

Changes

  • Extract the requested progress-index component from direct, hybrid, and state-wrapped progress indexes.
  • Use the meta component for non-data-region source recovery, remaining-event accounting, and listening-queue cleanup.
  • Preserve aggregate and two-stage processor state when progress indexes are hybrid.
  • Handle time-window state nested in hybrid indexes when selecting historical TsFiles.
  • Conservatively fall back to snapshot recovery or full queue size when no meta component exists.
  • Add unit coverage for hybrid, state-wrapped hybrid, and hybrid-without-meta cases.

Backport adaptations

  • Apply the historical-source fix to PipeHistoricalDataRegionTsFileSource, which is the implementation used by dev/1.3.
  • Keep the implementation compatible with the Java 8 source level and the logging structure on dev/1.3.

Verification

  • mvn spotless:apply -pl iotdb-core/node-commons,iotdb-core/datanode,iotdb-core/confignode
  • mvn test -pl iotdb-core/node-commons -Dtest=ProgressIndexTest,IoTDBNonDataRegionSourceTest
  • mvn test -pl iotdb-core/datanode -Dtest=TwoStageCountProcessorTest,TsFileResourceProgressIndexTest
  • mvn test-compile -pl iotdb-core/confignode -DskipTests

This PR has:

  • been self-reviewed.
  • added unit tests to cover new code paths.

Key changed/added classes
  • ProgressIndex and its implementations
  • IoTDBNonDataRegionSource
  • PipeHistoricalDataRegionTsFileSource
  • AggregateProcessor
  • TwoStageCountProcessor

* Pipe: Handle hybrid meta progress indexes

* Pipe: Handle hybrid meta progress indexes

* Fix getter

* Pipe: Delegate progress index extraction to subclasses
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant