Apache Cloudberry version
2.1.0
What happened
The regression test contrib/pax_storage/sql/pax_inject_fault.sql is flaky and fails non-deterministically.
The test deliberately injects a panic fault (orc_writer_write_tuple) and then executes an INSERT statement to trigger it. Because the fault type is panic, it crashes the segment postmaster. When the segment crashes, it asynchronously outputs system crash notices (DETAIL, HINT, WARNING).
The .out expected file only expects a single line: ERROR: fault triggered .... Due to timing variations (race condition), the QD sometimes receives only the ERROR message (test passes), and sometimes it receives the ERROR along with the crash notices (test fails with a diff).
Here is the exact diff of the failure:
insert into t_insert select generate_series(1,10);
+DETAIL: The postmaster has commanded this server process to roll back the current
+ transaction and exit, because another server process exited abnormally
+ and possibly corrupted shared memory.
ERROR: fault triggered, fault name:'orc_writer_write_tuple' fault type:'panic'
+HINT: In a moment you should be able to reconnect to the database and repeat your command.
+WARNING: terminating connection because of crash of another server process
In CI, it often fails non-deterministically in profiles like pax-ic-deb-good-opt-off. Furthermore, this crash can cause a cascading failure where the immediately following test (e.g., transactions) fails because it catches the aftermath of the segment crash.
What you think should happen instead
resulting crash messages should be deterministic
How to reproduce
Run the pax_storage regression tests, specifically the pax_inject_fault test.
Because this is a timing-related race condition, you may need to run the test in a loop or under specific load/CI configurations to catch the moment when the segment manages to send the DETAIL/HINT/WARNING messages before the QD finishes reading the response.
Operating System
rockylinux9
Anything else
on testing
Are you willing to submit PR?
Code of Conduct
Apache Cloudberry version
2.1.0
What happened
The regression test contrib/pax_storage/sql/pax_inject_fault.sql is flaky and fails non-deterministically.
The test deliberately injects a panic fault (orc_writer_write_tuple) and then executes an INSERT statement to trigger it. Because the fault type is panic, it crashes the segment postmaster. When the segment crashes, it asynchronously outputs system crash notices (DETAIL, HINT, WARNING).
The .out expected file only expects a single line: ERROR: fault triggered .... Due to timing variations (race condition), the QD sometimes receives only the ERROR message (test passes), and sometimes it receives the ERROR along with the crash notices (test fails with a diff).
Here is the exact diff of the failure:
In CI, it often fails non-deterministically in profiles like pax-ic-deb-good-opt-off. Furthermore, this crash can cause a cascading failure where the immediately following test (e.g., transactions) fails because it catches the aftermath of the segment crash.
What you think should happen instead
resulting crash messages should be deterministic
How to reproduce
Run the pax_storage regression tests, specifically the pax_inject_fault test.
Because this is a timing-related race condition, you may need to run the test in a loop or under specific load/CI configurations to catch the moment when the segment manages to send the DETAIL/HINT/WARNING messages before the QD finishes reading the response.
Operating System
rockylinux9
Anything else
on testing
Are you willing to submit PR?
Code of Conduct