Hardware Deadlock And RSX Stability Fixes

[Cruslan]

Summary

This pull request introduces critical stability fixes to librsx, cgcomp, and related GCM headers in PSL1GHT. These patches resolve severe hardware deadlocks, memory corruptions, pointer arithmetic overflows, and shader compilation failures that break continuous drawing and streaming workloads.

These issues directly surfaced during the porting and bare-metal deployment of PS3-Moonlight and PS3-Movian.

Important

Bare-Metal Hardware Verification:
Both the reproduced failure states and their corresponding fixes have been thoroughly tested and validated on physical PlayStation 3 retail hardware running HEN as well as RPCS3. These fixes are proven in sustained real-world production workloads.

---

Technical Problem Breakdown & Fix Details

1. librsx: Wrap User Callback in __get_opd32 to Prevent Fatal Crash on Buffer Wrap

• Commit: 03a2c10
• Affected File: ppu/librsx/init.c
• Root Cause:
  In the 64-bit PowerPC ELF ABI (ppc64), C function pointers do not point directly to executable machine code; they point to a 24-byte Official Procedure Descriptor (OPD) consisting of [function_entry_address, toc_pointer, environment_pointer].
  However, in ppu/include/rsx/rsx_function_macros.h, the low-level GCM assembly callback dispatcher (rsxContextCallback) is designed with 32-bit word instructions (lwz 0, 0(%3); lwz 2, 4(%3); mtctr 0; bctrl), expecting a 32-bit OPD layout (opd32).
  Passing a standard 64-bit function pointer directly to rsxSetContextCallback causes the dispatcher to read high 32-bit word halves as function entry and TOC pointers. The moment the RSX command buffer wraps around, the dispatcher executes a jump to corrupted memory, immediately producing a fatal PPU trap / panic.
• Fix:
  rsxSetContextCallback now wraps user callback handlers through __get_opd32(), guaranteeing that a properly formatted 32-bit OPD entry point and TOC descriptor are stored into context->callback.

---

2. librsx & cgcomp: Fix Pointer Arithmetic Overflow, Vertex Constant Offsets, and Texture Sampling Flag

• Commit: b7c7ed9
• Affected Files: ppu/librsx/init.c, ppu/librsx/commands_impl.h, ppu/librsx/buffer_impl.h, ppu/include/rsx/rsx_function_macros.h, tools/cgcomp/source/compilerfp.cpp, ppu/include/codec/vdec.h
• Root Cause & Fix Details:
  1. Pointer Arithmetic 4x Scaling Bug in rsxSetupContextData (init.c):
     The command buffer initialization previously performed pointer arithmetic by adding a byte length directly to a typed uint32_t* pointer. In standard C/C++, adding length to uint32_t* scales the offset by sizeof(uint32_t) == 4. For instance, a 1 MB buffer resulted in setting context->current 4 MB ahead, pointing far outside the allocated buffer boundaries and immediately causing heap/memory corruption.
     Fix: Cast the base address to uintptr_t / raw byte offsets before adding the byte length.
  2. cgcomp Texture Sampler Control Bit (allowTex / 0x8000):
     When compiling Cg fragment shaders that perform texture sampling (such as YUV-to-RGB color space conversion in video playback and streaming), the RSX microcode requires bit 15 (0x8000) to be set in the fragment program instruction word. Without this flag, texture coordinate interpolation and texture unit fetches fail or output black/garbage frames.
     Fix: Explicitly set allowTex (bit 15) in compilerfp.cpp whenever the fragment shader references texture samplers.
  3. Vertex Program Constants Multi-Vector Burst Offset (commands_impl.h):
     In SetVertexProgramConstants, the second burst packet was reusing the base pointer value instead of advancing by 16 floats (&value[16]). When uploading matrix blocks or multi-vector arrays exceeding 16 constants, the first chunk was repeatedly overwritten while subsequent vectors were dropped.
     Fix: Properly advance the source data pointer to &value[16] for the second transfer chunk.
  4. Packet Boundary Out-of-Bounds in SetTimeStamp (commands_impl.h):
     SetTimeStamp was writing to current[1] instead of [2], corrupting the command stream and misaligning subsequent GPU commands.
     Fix: Corrected packet indexing to respect packet payload sizing.
  5. Assembly Calling Convention Register Preservation (rsx_function_macros.h):
     The inline callback macro was clobbering caller-saved registers during 32-bit OPD dispatch, causing mysterious crashes upon returning from user callbacks.
     Fix: Guarded register preservation across the assembly invocation.
  6. Missing Video Decoder Definitions (vdec.h):
     Added missing standard VDEC codec constants (MPEG2, AVC/H.264, VC-1, DivX) required for hardware video decoding integration.

---

3. librsx: Bounded Spin Timeout and Hardware Memory Barrier in ResetCommandBuffer

• Commit: 9378160
• Affected File: ppu/librsx/buffer_impl.h
• Root Cause Analysis:
  ResetCommandBuffer had an unbounded loop waiting for the GPU read pointer to reach the offset:

  while(ctrl->get != offset) usleep(30);

  Under sustained load (such as fast video frame rendering in Movian or 60 FPS streaming in Moonlight), if the RSX FIFO pipeline stalls or an interrupt timing difference occurs, ctrl->get may never match offset. This caused the PPU thread to enter an infinite spinlock, hard-freezing the entire console and requiring a physical power-cord disconnection.
• Fix:
  • Implemented an upper-bounded spin counter (spins < 100000, ~3 second timeout) to gracefully break out of deadlocks.
  • Added an explicit hardware memory barrier (__sync()) before updating ctrl->put = offset to ensure all previous cache flushes and memory transactions are committed to the RSX memory controller.
  • Synchronized host write pointers properly using gcmAddressToOffset(context->begin).

---

Real-World Project Impact

Project	Failure Mode Without Patches	Behavior With Patches
PS3-Moonlight	Instant crash on buffer wrap (corrupted OPD callback), missing texture rendering (Cg sampler bit 0x8000), and hard PS3 freezes on frame burst stalls.	Stable 60 FPS 1080p/720p sustained streaming for hours with zero hangs or dropped frames.
PS3-Movian (Movian Next)	Command buffer allocation overflows (4x scaling bug), corrupted OSD/subtitle vertex matrices, and random hard freezes during seeking/playback.	Flawless hardware-accelerated video decoding, correct subtitle rendering, and crash-free UI transitions.