Skip to content

Fix Swing dialog deadlock on macOS desktop build#9

Open
lxpollitt wants to merge 1 commit into
lanceewing:masterfrom
lxpollitt:fix/desktop-dialog-deadlock
Open

Fix Swing dialog deadlock on macOS desktop build#9
lxpollitt wants to merge 1 commit into
lanceewing:masterfrom
lxpollitt:fix/desktop-dialog-deadlock

Conversation

@lxpollitt

Copy link
Copy Markdown
Contributor

Problem

On macOS, clicking any UI button that opens a Swing dialog (File Open, Help / About, Emulator Reset, etc.) hangs JOric with a spinning beach ball that never recovers. The in-emulator curated program list still works because it's rendered by libGDX itself rather than via a Swing dialog.

The root cause is the collision of two macOS-specific main-thread constraints:

  1. GLFW needs the OS main thread. libGDX's lwjgl3 backend uses GLFW, which on macOS must run its event loop on the OS main thread (thread 0). StartupHelper.startNewJvmIfRequired already arranges this by re-launching the child JVM with -XstartOnFirstThread, so the libGDX render thread is the macOS main thread.
  2. AWT/Swing also needs the OS main thread for AppKit initialisation and dispatch. Swing modal dialogs normally route their event handling through the AWT Event Dispatch Thread (EDT), which in turn round-trips to the main thread for AppKit work.

These two constraints fight: GLFW ends up owning the main thread, leaving no main thread available for AppKit when a Swing dialog needs it. The existing code dispatches Swing dialogs onto the libGDX render thread via Gdx.app.postRunnable, making the dialog call run on the thread AppKit needs to be free. The dialog parks waiting for the EDT to complete its dispatch; the EDT can't complete because it needs the main thread; the main thread is parked waiting for the dialog - which results in a deadlock.

I confirmed this via jstack on a hung run: the main thread parked in Object.wait() inside JFileChooser.showOpenDialogWaitDispatchSupport.enter, with the AWT-EventQueue thread spinning in sun.lwawt.macosx.LWCToolkit.isApplicationActive trying to round-trip to the macOS main thread (which was the very thread parked above).

I think this is a macOS-only manifestation. Windows and Linux don't have the same -XstartOnFirstThread-style main-thread constraint for either GLFW or AWT. However, I think the existing Gdx.app.postRunnable pattern technically also violates the Swing thread-safety contract on those platforms (Swing components are normally only supposed to be accessed from the EDT), but the violation just happens to not deadlock there because there's no contested main thread to serialise on.

Fix

Two changes, both applied unconditionally on all OSes although the bug only manifests on macOS:

  • DesktopDialogHandler: route the five dialog methods (confirm, openFileDialog, promptForTextInput, showAboutDialog, promptForOption) through SwingUtilities.invokeLater instead of Gdx.app.postRunnable. The response handler is then posted back through Gdx.app.postRunnable so it returns to the libGDX render thread for any subsequent libGDX-state mutation. This is contract-compliant with Swing's "components are only accessed from the EDT" rule on every OS, not just macOS.

  • DesktopLauncher: pre-initialise AWT via java.awt.Toolkit.getDefaultToolkit() after StartupHelper.startNewJvmIfRequired returns but before the Lwjgl3Application constructor takes over the main thread. Without this, the first Swing call later in the lifecycle would try to initialise AppKit from a non-main thread (the dispatched EDT runnable) and hang. Pre-initialising allows AppKit to initialise on the main thread before libGDX takes it over. This is required for macOS and should be no-op-ish on Windows / Linux (where AWT doesn't have a main-thread requirement).

Scope

  • All five dialog methods migrated to the same pattern.
  • The Toolkit.getDefaultToolkit() pre-init is a single line in the launcher. Could be platform-gated to macOS only if preferred; I left it un-gated because the cost on other platforms is negligible and thought it better to have a single code path for simplicity.

Testing

I've confirmed the fix behaves as expected for me using the macOS desktop for openFileDialog, promptForOption, showAboutDialog, and confirm dialogs. Before this PR, clicking any of these would beach-ball and never recover. After this PR, all these dialogs open promptly and return the results as expected.

However, I have not tested the promptForTextInput dialog because I couldn't see any place in the UI that used it?

I also haven't been able to verify Windows / Linux (I don't have easy access to either). The pattern change is conservative enough that I'd expect it to work as a no-op behaviour-wise on those platforms - I assume the dialogs were already working there, and SwingUtilities.invokeLater is the documented Swing convention. But it would be good to verify on at least one other OS to be sure before merging.

On macOS, the lwjgl3 desktop build requires -XstartOnFirstThread for GLFW, which means the libGDX render thread is also the AppKit main thread. Swing dialog calls (JFileChooser.showOpenDialog, JOptionPane.showXxxDialog, etc.) were being dispatched onto the libGDX render thread via Gdx.app.postRunnable, which deadlocked because the Swing API normally expects to dispatch through the AWT Event Dispatch Thread, which in turn needs the macOS main thread to be free.

DesktopDialogHandler: route the five dialog methods (confirm, openFileDialog, promptForTextInput, showAboutDialog, promptForOption) through SwingUtilities.invokeLater. The response handler is then posted back via Gdx.app.postRunnable so it runs on the libGDX render thread, keeping libGDX state mutations safe.

DesktopLauncher: pre-initialise AWT (Toolkit.getDefaultToolkit) before the Lwjgl3Application constructor takes the main thread. Without this, the first Swing call later would try to initialise AppKit from a non-main thread and hang. Pre-initialising claims AppKit before libGDX takes the main thread.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant