Skip to content

[GLUTEN-12985][VL] Handle mid-page EOS in VeloxRssSortShuffleReaderDeserializer - #12986

Open
ccat3z wants to merge 1 commit into
apache:mainfrom
ccat3z:rss-reader-endless-loop
Open

ccat3z wants to merge 1 commit into
apache:mainfrom
ccat3z:rss-reader-endless-loop

Conversation

@ccat3z

@ccat3z ccat3z commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

What changes are proposed in this pull request?

GlutenByteInputStream::readBytes() drives next(true) in a for(;;) loop. When a page header declares more bytes than the stream actually holds (truncated partition data), the reader hits EOS mid-page — but VeloxInputStream::next() ignores its throwIfPastEnd argument and silently returns on EOS, so the loop never exits.

This PR impl throwIfPassEnd arg of VeloxRssSortShuffleReaderDeserializer::VeloxInputStream::next(), fixed bug1 in #12985

How was this patch tested?

UT

Was this patch authored or co-authored using generative AI tooling?

Co-Authored-By: Claude

@ccat3z
ccat3z marked this pull request as ready for review September 9, 2026 12:27
@github-actions github-actions Bot added the VELOX label Sep 9, 2026
@ccat3z
ccat3z force-pushed the rss-reader-endless-loop branch from 17ee6b4 to 46252b1 Compare September 10, 2026 04:49
Comment thread cpp/velox/tests/VeloxShuffleReaderTest.cc

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Fix the dangling timer reference and correct the negative-read assertion setup.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR fixes mid-page EOS and negative-read handling in the Velox RSS sort shuffle reader and adds regression coverage.

Changes:

  • Honors throwIfPastEnd and rejects negative reads.
  • Adds EOS and invalid-read tests.
  • Registers the new Velox test target.
File summaries
File Summary
cpp/velox/tests/VeloxShuffleReaderTest.cc Adds regression tests. Critical (3 votes): fixes needed for the dangling deserializeTime reference. Moderate (1 vote): the negative-read assertion must wrap deserializer construction.
cpp/velox/tests/CMakeLists.txt Registers the reader test executable.
cpp/velox/shuffle/VeloxShuffleReader.cc Implements EOS and negative-read handling.
Review details

Suppressed comments (1)

cpp/velox/tests/VeloxShuffleReaderTest.cc:210

  • The negative Read() is triggered by VeloxInputStream's constructor (next(false)), so makeDeserializer(...) throws before VELOX_ASSERT_THROW is entered. As written, this test aborts at the helper assignment instead of passing through the assertion; wrap the makeDeserializer call itself in VELOX_ASSERT_THROW (or otherwise defer the negative return until next()).
  auto deserializer =
      makeDeserializer(std::make_shared<FakeInputStream>(std::vector<uint8_t>{}, /*negativeRead=*/true));
  • Files reviewed: 3/3 changed files
  • Comments generated: 1
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread cpp/velox/tests/VeloxShuffleReaderTest.cc
@ccat3z
ccat3z force-pushed the rss-reader-endless-loop branch from 66c8ba7 to 24e9682 Compare September 14, 2026 07:17
@zhouyuan

Copy link
Copy Markdown
Member

@ccat3z @kecookier it seems several Spark UT will hang there with this patch, please have a check

…serializer

GlutenByteInputStream::readBytes() drives next(true) in a for(;;) loop.
When a page header declares more bytes than the stream actually holds
(truncated partition data), the reader hits EOS mid-page — but
VeloxInputStream::next() ignores its throwIfPastEnd argument and silently
returns on EOS, so the loop never exits.

This PR impl throwIfPassEnd arg of VeloxRssSortShuffleReaderDeserializer::VeloxInputStream::next()
@ccat3z
ccat3z force-pushed the rss-reader-endless-loop branch from 24e9682 to e5c015f Compare September 15, 2026 02:19
@ccat3z

ccat3z commented Sep 15, 2026

Copy link
Copy Markdown
Contributor Author
2026-09-14T07:47:23.1750547Z 07:47:23.172 WARN org.apache.spark.sql.execution.GlutenFallbackReporter: Validation failed for plan: Subquery[QueryId=14005], due to: Gluten does not touch it or does not support it
2026-09-14T07:47:23.1751346Z 
2026-09-14T07:47:23.2389322Z 07:47:23.238 WARN org.apache.spark.sql.execution.GlutenFallbackReporter: Validation failed for plan: LocalTableScan[QueryId=14005], due to: Gluten does not touch it or does not support it
2026-09-14T07:47:23.2391116Z 
2026-09-14T07:47:23.2470911Z E20260914 07:47:23.246634 31338 VeloxMemoryManager.cc:446] Failed to release Velox memory manager as there are still outstanding memory resources. 
2026-09-14T07:47:23.2474689Z E20260914 07:47:23.246676 31338 MemoryPool.cpp:480] [MEM] Memory leak (Used memory): Memory Pool[cached_table_4571277_0 LEAF root[root] parent[root] MALLOC track-usage thread-safe]<unlimited max capacity capacity 8.00MB used 4.00KB available 1020.00KB reservation [used 4.00KB, reserved 1.00MB, min 0B] counters [allocs 2, frees 1, reserves 0, releases 0, collisions 0, external-allocs 0, external-frees 0, cumulative-external 0B])>
2026-09-14T07:47:23.2478439Z E20260914 07:47:23.246721 31338 Exceptions.h:87] Line: /work/cpp/velox/memory/VeloxMemoryManager.cc:108, Function:removePool, Expression: pool->reservedBytes() == 0 (1048576 vs. 0), Source: RUNTIME, ErrorCode: INVALID_STATE
2026-09-14T07:47:23.2487494Z terminate called after throwing an instance of 'facebook::velox::VeloxRuntimeError'
2026-09-14T07:47:23.2496608Z #
2026-09-14T07:47:23.2497558Z # A fatal error has been detected by the Java Runtime Environment:
2026-09-14T07:47:23.2498444Z #
2026-09-14T07:47:23.2499016Z #  SIGSEGV (0xb) at pc=0x0000000000000000, pid=2086, tid=31064
2026-09-14T07:47:23.2499692Z #
2026-09-14T07:47:23.2500520Z # JRE version: OpenJDK Runtime Environment (Red_Hat-17.0.6.0.9-0.3.ea.el8) (17.0.6+9) (build 17.0.6-ea+9-LTS)
2026-09-14T07:47:23.2502450Z # Java VM: OpenJDK 64-Bit Server VM (Red_Hat-17.0.6.0.9-0.3.ea.el8) (17.0.6-ea+9-LTS, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64)
2026-09-14T07:47:23.2504064Z # Problematic frame:
2026-09-14T07:47:23.2504579Z # C  0x0000000000000000
2026-09-14T07:47:23.2505068Z #
2026-09-14T07:47:23.2506456Z # Core dump will be written. Default location: Core dumps may be processed with "/lib/systemd/systemd-coredump %P %u %g %s %t 9223372036854775808 %h %d" (or dumping to /__w/gluten/gluten/gluten-ut/spark35/core.2086)
2026-09-14T07:47:23.2508020Z #
2026-09-14T07:47:23.2508582Z # An error report file with more information is saved as:
2026-09-14T07:47:23.2509352Z # /__w/gluten/gluten/gluten-ut/spark35/hs_err_pid2086.log
2026-09-14T07:47:23.2527013Z   what():  Exception: VeloxRuntimeError
2026-09-14T07:47:23.2527621Z Error Source: RUNTIME
2026-09-14T07:47:23.2528102Z Error Code: INVALID_STATE
2026-09-14T07:47:23.2530210Z Reason: (1048576 vs. 0)
2026-09-14T07:47:23.2531147Z Retriable: False
2026-09-14T07:47:23.2531720Z Expression: pool->reservedBytes() == 0
2026-09-14T07:47:23.2532279Z Function: removePool
2026-09-14T07:47:23.2532829Z File: /work/cpp/velox/memory/VeloxMemoryManager.cc
2026-09-14T07:47:23.2533438Z Line: 108
2026-09-14T07:47:23.2534575Z Stack trace:
2026-09-14T07:47:23.2535460Z # 0  _ZN8facebook5velox7process10StackTraceC1Ei
2026-09-14T07:47:23.2536402Z # 1  _ZN8facebook5velox14VeloxExceptionC1EPKcmS3_St17basic_string_viewIcSt11char_traitsIcEES7_S7_S7_bNS0_24CompileTimeStringLiteralENS1_4TypeES7_
2026-09-14T07:47:23.2537733Z # 2  _ZN8facebook5velox6detail14veloxCheckFailINS0_17VeloxRuntimeErrorERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEEvRKNS1_18VeloxCheckFailArgsET0_NS0_24CompileTimeStringLiteralE
2026-09-14T07:47:23.2538912Z # 3  _ZN6gluten12_GLOBAL__N_120ListenableArbitrator10removePoolEPN8facebook5velox6memory10MemoryPoolE
2026-09-14T07:47:23.2539676Z # 4  _ZN8facebook5velox6memory13MemoryManager8dropPoolEPNS1_10MemoryPoolE
2026-09-14T07:47:23.2540883Z # 5  _ZN8facebook5velox6memory14MemoryPoolImplD2Ev
2026-09-14T07:47:23.2541674Z # 6  _ZNSt16_Sp_counted_baseILN9__gnu_cxx12_Lock_policyE2EE24_M_release_last_use_coldEv
2026-09-14T07:47:23.2542259Z # 7  _ZN6gluten18VeloxMemoryManagerD1Ev
2026-09-14T07:47:23.2542669Z # 8  _ZN6gluten18VeloxMemoryManagerD0Ev
2026-09-14T07:47:23.2543089Z # 9  _ZN6gluten13MemoryManager7releaseEPS0_
2026-09-14T07:47:23.2543779Z # 10 Java_org_apache_gluten_memory_NativeMemoryManagerJniWrapper_release
2026-09-14T07:47:23.2544438Z # 11 0x00007f111d423c64
2026-09-14T07:47:23.2544798Z # 12 0x00007f111edc3554
2026-09-14T07:47:23.2545045Z 

The UT failures seems unrelated to this PR. Let me rebase onto the latest main and retrigger UT.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants