diff --git a/docs/architecture-migration.md b/docs/architecture-migration.md index 2d4896b7..86119e0b 100644 --- a/docs/architecture-migration.md +++ b/docs/architecture-migration.md @@ -115,6 +115,7 @@ - hook：其餘 30 個 `SFR_HOOK` 都改掉了。NUI 裝置（初始化、骨架追蹤、影像串流、身分辨識）拿輸入鎖；選單的 hook（按鈕、語音、轉盤、對話框、選單管理）拿新的選單鎖（`SFR_MENU_HOOK`），順序固定「選單→輸入→圖形」；只碰 guest 記憶體、參數與 atomic 的（相機影像槽、純虛函式工作、Avatar 零件、深度檢視、淡出、`LockRect` 唯讀旗標）不拿鎖；影片進度表、Avatar 動作片段各有自己的鎖，Avatar 模型的繪製拿圖形鎖。建立／重設裝置與 present 只拿圖形鎖：它們碰到的其他主機狀態（視窗、計時）屬於擁有視窗的那個執行緒。 - GPU 資源同步（`synchronize_resource_memory`：等 GPU 閒置、丟掉 renderer 的副本）拿圖形鎖，不再拿全域許可（原本等閒置時對 queue 送出也沒有和 render thread 互斥）。 - 結果：`all` 模式的整場比賽裡，沒有任何執行緒回到全域許可（`SFR_PARALLEL_HOOK_STATS=1` 的 import／hook 統計一次都沒有印出）。全域許可只剩給非 `all` 模式的排程用。 + - 圖形鎖留著（2026-10-20，`SFR_GRAPHICS_LOCK_DEFER`，預設開）：比賽中只有主執行緒拿圖形鎖、從不競爭（`SFR_GRAPHICS_LOCK_STATS=1`），但每幀拿放約 16,000 次。圖形 hook 返回時主執行緒保留這把鎖，下一個圖形 hook 直接接手；在 hook 之外只在跑生成程式碼時拿著，遇到 import、其他 hook（含 `SFR_CONCURRENT_HOOK`）、許可檢查點、慢速記憶體存取、`traced_host_wait` 之前先放掉，有人在等時每次 hook 返回都放。鎖的順序不變。Thor 上 mutex 的時間每幀 0.52 → 0.18 ms（`docs/performance.md`）。 ### 第 5 階段：重編譯期的 mid-asm hook 與幀率 diff --git a/docs/performance.md b/docs/performance.md index b7eb2f03..cf19ee1e 100644 --- a/docs/performance.md +++ b/docs/performance.md @@ -2383,3 +2383,92 @@ The constant writes themselves (`SFR_CONSTANT_TIMING=1`, one run each): 561 ns and 8192 bytes a draw before, 237 ns and 623 bytes after, at about 820 draws a drawn frame. Windows (i9, Vulkan / D3D12, a machine with other load): 776 / 778 ns a draw before, 199 / 140 ns after. + +## 2026-10-20: The graphics lock on the main thread + +The pass above left the graphics lock alone as "an uncontended mutex pair, +~2%". This pass measured what it protects, who takes it, and what it costs. + +**What it protects.** One thread at a time in the native device: every +Direct3D replacement in `guest_graphics_hooks.cpp` (`SFR_GRAPHICS_HOOK`, +`SFR_GRAPHICS_HOST_HOOK`) takes it at its observed entry and gives it back as +its `HookScope` ends; `HostGraphicsScope` takes it for host code outside those +hooks that changes the device (`synchronize_resource_memory`, the renderer's +`invalidate` after `MmFreePhysicalMemory`, the Avatar model's draw). Order: +menu, input, graphics; graphics before the permits. + +**Who takes it.** `SFR_GRAPHICS_LOCK_STATS=1` prints `GRAPHICS_LOCK` every 120 +frames (per frame, per guest thread: acquisitions, waits and wait time, nested +hooks, `HostGraphicsScope` acquisitions, and how far apart the acquisitions +are). Full Dolphin Resort races on Windows (Vulkan) and the AYN Thor: + +| | menus | race | +| --- | --- | --- | +| main-thread acquisitions per frame | ~700 | 14,000-20,000 (Thor 16,000-20,600) | +| acquisitions that found it held | 0 | 0 | +| other threads' acquisitions | none | none | +| `HostGraphicsScope` | <1 | 0 | +| no import, other hook or permit checkpoint since the last release | 85% | 91% | + +So in a race it is only ever the main thread's, and it is never contended: +the cost is the uncontended pair itself. A loop of try_lock/unlock costs 33 ns +a pair on the Thor (18 ns on the desktop) against 14.5 ns (6 ns) for a bare +atomic pair: bionic's `pthread_mutex_trylock`/`unlock` are calls through the +PLT that check the mutex type and then run the outline atomics +(`__aarch64_cas2_acq`, `__aarch64_swp2_rel`), which simpleperf showed as the +hot part. Hooks are 4-15 guest entries apart for most of them. + +**Change: `SFR_GRAPHICS_LOCK_DEFER`** (on unless 0). The main thread keeps the +lock when a graphics hook returns (`hook_scope_exit`), and the next graphics +hook takes it over without an atomic. Outside a hook it is then held only +while generated code runs, and it is given back before anything that may +wait or take another lock: an import (`dispatch_import`), any other hook +(`HookScope`, `SFR_CONCURRENT_HOOK`, the observed entry of a hook, before its +menu or input lock), the permit's checkpoint (`guest_checkpoint_permit`, at +least every 256 entries and loop labels, before the permit can hand off the +core), a slow memory access (`parallel_slow_access`) and `traced_host_wait`. +`HostGraphicsScope` takes a kept lock over, so nothing inside it gives the +lock back early. While another thread waits for the lock (`graphics_waiters`, +its own cache line), it is given back at every hook's return as before, and +a waiter waits at most until the main thread's next checkpoint. Everything +the generated code calls besides those is a memory, vector or reservation +helper (leaf locks only), `read_time_base`, the job mid-asm hooks (their +waits go through `traced_host_wait`) or `synchronize_resource_memory` +(`HostGraphicsScope`). The register save/restore helpers, which never wait, +are `SFR_REGISTER_HOOK` without the check. `parallel_function_entry` also +returns before the permit's thread_local when the subsystem locks leave it +nothing to do. `GRAPHICS_LOCK_DEFER 1` in the log says it is on. With it, a +race frame takes the lock ~1,100-1,500 times instead of ~16,000. + +**Thor A/B** (simpleperf, 40 s of the moving race from ~frame 13800, same APK +with `SFR_GRAPHICS_LOCK_DEFER=0` and on, main thread): + +| | lock off (defer 0) | kept (default) | +| --- | --- | --- | +| `__aarch64_cas2_acq` + `__aarch64_swp2_rel` | 2.44% (0.35 ms/frame) | 0.87% (0.12 ms/frame) | +| `pthread_mutex_*` + `std::mutex` | 1.17% (0.17 ms/frame) | 0.44% (0.06 ms/frame) | +| mutex total | 0.52 ms/frame | 0.18 ms/frame | +| main-thread CPU in the 40 s | 34.2 s | 32.6 s | + +The 0.18 ms left is other locks (critical sections, the layout's +shared_mutex, events). Frame times (HOST_THREADS, frames 13000-21000, +`!SFR_FRAME_METRICS=0`) did not resolve the 0.34 ms against their noise: the +main thread moves between the X3 and the A715s from run to run, and drawn +frames sit at the 16.7 ms budget, so `skipped` swings 34-176 between runs of +the same APK. Runs (drawn_ms / undrawn_ms / cpu_ms / skipped of 9600): + +| Build | runs | +| --- | --- | +| release-equivalent (0.10.13 code; one with the stats on) | 16.03/8.60/13.85/34, 15.99/9.66/14.33/176 | +| kept off (`SFR_GRAPHICS_LOCK_DEFER=0`, profiled) | 15.89/9.53/14.24/157 | +| kept | 16.23/9.88/14.04/168, 15.62/9.42/14.26/117, 16.18/8.97/13.70/52 (profiled), 15.87/9.33/13.91/87 | + +**Checked:** full races with the lock kept on Windows (Vulkan with the +Khronos validation layer, 0 messages, 455/455 pipelines; D3D12, 465/465) and +four on the Thor (and one with it off), each to the present limit with no +hang, stop or `LOCK_LONG_WAIT`; screenshots render normally. Two fixed-step (`SFR_REALTIME_RACE=0`) Windows races of the same +build already differ at the same presents (AI positions), as much as kept +against not, so pictures are compared by eye rather than bit for bit. + +**Not done:** the observed entry of each graphics hook +(`enter_function_observed` ~1%, its TLS and bit tests) is still per hook. diff --git a/src/diagnostic_hooks.h b/src/diagnostic_hooks.h index 84bb1fe7..ffb57943 100644 --- a/src/diagnostic_hooks.h +++ b/src/diagnostic_hooks.h @@ -121,6 +121,15 @@ inline void guest_checkpoint() { void enter_function_observed(PPCContext&, const char*, uint32_t); // Gives back the permit and graphics lock the outermost hook's entry took. void hook_scope_exit(GuestEntryState& entry) noexcept; +// The main thread keeps the graphics lock from one graphics hook to the next +// (SFR_GRAPHICS_LOCK_DEFER): a race frame enters some 15,000 of them, nearly +// all a few guest calls apart. Kept, it is given back before anything that +// may wait or take another lock: an import, any other hook, the permit's +// checkpoint, a slow memory access (diagnostic_main.cpp). +void give_back_kept_graphics(GuestEntryState& entry) noexcept; +inline void give_back_kept_graphics_lock(GuestEntryState& entry) noexcept { + if (entry.graphics_kept) [[unlikely]] give_back_kept_graphics(entry); +} // The calling thread's state, kept in the context's reservation register, // which the generated code never uses (sfr::load_reserved_word keeps the // reservation in GuestThreadState). On Android libmain is loaded after start, @@ -139,10 +148,16 @@ inline GuestThreadState& entry_thread_state(Context& ctx) { } } // Around every hook, with the state its context holds (thousands of graphics -// hooks a frame paid the TLS resolver twice each on Android). +// hooks a frame paid the TLS resolver twice each on Android). A hook other +// than a graphics one gives back a kept graphics lock first: its host code may +// wait or take a lock of its own (and not every hook names itself with +// enter_function, which would give it back too). struct HookScope { template - explicit HookScope(Context& ctx) : entry_(entry_thread_state(ctx).entry) { ++entry_.hook_depth; } + HookScope(Context& ctx, bool graphics) : entry_(entry_thread_state(ctx).entry) { + if (!graphics) give_back_kept_graphics_lock(entry_); + ++entry_.hook_depth; + } ~HookScope() { if (!--entry_.hook_depth && (entry_.hook_stack_pointer || entry_.graphics_stack_pointer || entry_.input_stack_pointer)) hook_scope_exit(entry_); @@ -396,18 +411,26 @@ inline constexpr sfr::NoFastPath sfr_fast{}; // thread that calls the hook from a loop in the same frame: the Kinect // skeleton thread then held the global permit for good after its first // NuiSkeletonGetNextFrame, stalling the whole game for its time slices.) -#define SFR_SCOPED_HOOK(x, registration) [[maybe_unused]] static const bool x##_hook = registration; \ +#define SFR_SCOPED_HOOK(x, registration, graphics) [[maybe_unused]] static const bool x##_hook = registration; \ static void x##_hooked(PPCContext& __restrict ctx, uint8_t* base); \ - PPC_FUNC(x) { ::sfr::HookScope scope(ctx); x##_hooked(ctx, base); } \ + PPC_FUNC(x) { ::sfr::HookScope scope(ctx, graphics); x##_hooked(ctx, base); } \ static void x##_hooked(PPCContext& __restrict ctx, uint8_t* base) -#define SFR_HOOK(x) SFR_SCOPED_HOOK(x, ::sfr::register_hook(#x)) +#define SFR_HOOK(x) SFR_SCOPED_HOOK(x, ::sfr::register_hook(#x), false) // A Direct3D replacement that touches only the native device and guest memory // (see graphics_hook_bits), and one that also reaches other host state. -#define SFR_GRAPHICS_HOOK(x) SFR_SCOPED_HOOK(x, ::sfr::register_graphics_hook(#x, false)) -#define SFR_GRAPHICS_HOST_HOOK(x) SFR_SCOPED_HOOK(x, ::sfr::register_graphics_hook(#x, true)) -#define SFR_INPUT_HOOK(x) SFR_SCOPED_HOOK(x, ::sfr::register_input_hook(#x)) -#define SFR_MENU_HOOK(x) SFR_SCOPED_HOOK(x, ::sfr::register_menu_hook(#x)) +#define SFR_GRAPHICS_HOOK(x) SFR_SCOPED_HOOK(x, ::sfr::register_graphics_hook(#x, false), true) +#define SFR_GRAPHICS_HOST_HOOK(x) SFR_SCOPED_HOOK(x, ::sfr::register_graphics_hook(#x, true), true) +#define SFR_INPUT_HOOK(x) SFR_SCOPED_HOOK(x, ::sfr::register_input_hook(#x), false) +#define SFR_MENU_HOOK(x) SFR_SCOPED_HOOK(x, ::sfr::register_menu_hook(#x), false) // A replacement that touches only guest memory, its arguments, atomics and // host objects that keep their own lock, so a detached guest runs it without // taking any lock of the runtime's (docs/architecture-migration.md, phase 4). -#define SFR_CONCURRENT_HOOK(x) PPC_FUNC(x) +// Such host code may still wait (the job system's patches) or take a lock of +// its own, so it gives back a kept graphics lock first. +#define SFR_CONCURRENT_HOOK(x) static void x##_concurrent(PPCContext& __restrict ctx, uint8_t* base); \ + PPC_FUNC(x) { ::sfr::give_back_kept_graphics_lock(::sfr::entry_thread_state(ctx).entry); x##_concurrent(ctx, base); } \ + static void x##_concurrent(PPCContext& __restrict ctx, uint8_t* base) +// A replacement that touches only this thread's context and guest memory, +// never waits and takes no lock (the register save and restore helpers, +// called at most function entries). +#define SFR_REGISTER_HOOK(x) PPC_FUNC(x) diff --git a/src/diagnostic_main.cpp b/src/diagnostic_main.cpp index cd3a39b3..16edff1b 100644 --- a/src/diagnostic_main.cpp +++ b/src/diagnostic_main.cpp @@ -83,6 +83,7 @@ #include #include #include +#include #include #include #include @@ -551,6 +552,7 @@ void prepare_worker_self_suspend() { } static void parallel_slow_access(uint64_t address) { + give_back_kept_graphics_lock(guest_thread_state.entry); if (!execution_permit || !execution_permit->detached()) return; static const bool trace = [] { const char* t = std::getenv("SFR_PARALLEL_TRACE"); return t && *t != '0'; }(); if (trace) { @@ -591,19 +593,176 @@ static void lock_with_report(std::mutex& lock, const char* name, std::atomic taken{0}, contended{0}, wait_ns{0}, nested{0}, inner{0}, strict{0}, loose{0}; + std::atomic scope{0}, scope_contended{0}, scope_wait_ns{0}; + std::atomic kept{0}, resumed{0}; // SFR_GRAPHICS_LOCK_DEFER + // Strict acquisitions by the checkpoint countdown spent since the + // release: function entries and loop labels in between (0, 1, 2-3, 4-15, + // 16-63, 64+). + std::atomic gap[6]{}; +}; +static GraphicsLockCounters graphics_lock_counters[65]; // by guest id; 64 = any other +static thread_local uint64_t graphics_hard_epoch = 0, graphics_soft_epoch = 0; +static thread_local uint64_t graphics_released_hard = ~uint64_t(0), graphics_released_soft = ~uint64_t(0); +static thread_local uint32_t graphics_released_countdown = 0; +static GraphicsLockCounters& graphics_counters() { return graphics_lock_counters[current_id < 64 ? current_id : 64]; } +static uint64_t stats_now_ns() { + return uint64_t(std::chrono::duration_cast(std::chrono::steady_clock::now().time_since_epoch()).count()); +} +// What a released graphics lock could not be kept across: imports and other +// hooks (hard), the permit's checkpoints (soft). +static void graphics_lock_interrupted(bool hard) { + if (!graphics_lock_stats) return; + if (hard) ++graphics_hard_epoch; + else ++graphics_soft_epoch; +} +static void graphics_lock_released() { + if (!graphics_lock_stats) return; + graphics_released_hard = graphics_hard_epoch; + graphics_released_soft = graphics_soft_epoch; + graphics_released_countdown = guest_thread_state.entry.checkpoint_countdown; +} +// The uncontended cost of a lock and unlock on this machine, once: the +// graphics lock's std::mutex, and a bare atomic exchange pair. +static std::string graphics_lock_benchmark() { + constexpr int pairs = 200000; + std::mutex mutex; + auto started = std::chrono::steady_clock::now(); + for (int i = 0; i < pairs; ++i) { + if (mutex.try_lock()) mutex.unlock(); + std::atomic_signal_fence(std::memory_order_seq_cst); + } + const double mutex_ns = std::chrono::duration(std::chrono::steady_clock::now() - started).count() / pairs; + std::atomic word{0}; + started = std::chrono::steady_clock::now(); + for (int i = 0; i < pairs; ++i) { + uint32_t expected = 0; + if (word.compare_exchange_strong(expected, 1, std::memory_order_acquire, std::memory_order_relaxed)) + word.store(0, std::memory_order_release); + std::atomic_signal_fence(std::memory_order_seq_cst); + } + const double atomic_ns = std::chrono::duration(std::chrono::steady_clock::now() - started).count() / pairs; + char text[96]; + std::snprintf(text, sizeof text, " bench_mutex_ns=%.1f bench_atomic_ns=%.1f", mutex_ns, atomic_ns); + return text; +} +// The line for the last `frames` frames, or "" with the stats off. +std::string graphics_lock_report(uint64_t present, unsigned frames) { + if (!graphics_lock_stats || !frames) return {}; + static const std::string bench = graphics_lock_benchmark(); + const double n = frames; + std::ostringstream line; + line << std::fixed << std::setprecision(1) << "GRAPHICS_LOCK frame=" << present; + std::ostringstream others; + others << std::fixed << std::setprecision(1); + for (int id = 0; id < 65; ++id) { + auto& c = graphics_lock_counters[id]; + const uint64_t taken = c.taken.exchange(0), contended = c.contended.exchange(0), wait = c.wait_ns.exchange(0); + const uint64_t nested = c.nested.exchange(0), inner = c.inner.exchange(0), strict = c.strict.exchange(0), loose = c.loose.exchange(0); + const uint64_t scope = c.scope.exchange(0), scope_contended = c.scope_contended.exchange(0), scope_wait = c.scope_wait_ns.exchange(0); + const uint64_t kept = c.kept.exchange(0), resumed = c.resumed.exchange(0); + uint64_t gap[6]; + for (int b = 0; b < 6; ++b) gap[b] = c.gap[b].exchange(0); + if (id == 1) { + line << " main_kept=" << kept / n << " main_resumed=" << resumed / n << " main_gap=" << gap[0] / n << ',' << gap[1] / n << ',' << gap[2] / n << ',' << gap[3] / n << ',' + << gap[4] / n << ',' << gap[5] / n; + line << " main_taken=" << taken / n << " main_contended=" << contended / n << " main_wait_us=" << wait / n / 1e3 + << " main_nested=" << nested / n << " main_inner=" << inner / n << " main_strict=" << strict / n + << " main_loose=" << loose / n << " main_scope=" << scope / n << " main_scope_contended=" << scope_contended / n; + } else if (taken || scope) { + others << ' ' << id << ':' << taken << '/' << contended << '/' << wait / 1e3 << "us/" << scope << '/' << scope_contended + << '/' << scope_wait / 1e3 << "us"; + } + } + line << " others(total id:taken/contended/wait/scope/contended/wait)=" << others.str() << bench << '\n'; + return line.str(); +} +// SFR_GRAPHICS_LOCK_DEFER (on unless 0): the main thread keeps the graphics +// lock from one graphics hook to the next instead of taking and giving it back +// around each of the ~15,000 a race frame enters (an AYN Thor spent ~2% of the +// main thread in that mutex's atomics). It is held outside any hook then only +// while the thread runs generated code, which touches guest memory alone, and +// is given back before everything else: an import (dispatch_import), any +// other hook (HookScope, SFR_CONCURRENT_HOOK, the observed entry of a hook), +// the permit's checkpoint (guest_checkpoint_permit, at least every 256 entries +// and loop labels, so a thread waiting for the lock waits that long at most), +// a slow memory access (parallel_slow_access). So the thread never waits, nor +// takes another lock, while it keeps the graphics lock outside a hook, and +// the lock order (menu, input, graphics; graphics before the permits) holds. +// While another thread waits for the lock, it is given back at every hook's +// return as before. +static const bool graphics_lock_defer = [] { const char* t = std::getenv("SFR_GRAPHICS_LOCK_DEFER"); return !t || *t != '0'; }(); +// Read at every graphics hook's return: a cache line of its own, so that no +// other thread's writes to a neighbour make that read miss. +struct alignas(64) GraphicsWaiters { std::atomic count{0}; }; +static GraphicsWaiters graphics_waiters_line; +static std::atomic& graphics_waiters = graphics_waiters_line.count; +void give_back_kept_graphics(GuestEntryState& entry) noexcept { + entry.graphics_kept = false; + graphics_lock.unlock(); + graphics_lock_released(); +} static void graphics_lock_entry(const PPCContext& ctx, uint32_t address, GuestEntryState& entry) { const bool graphics = test_hook_bit(graphics_hook_bits, address); + if (entry.graphics_kept) { + // Kept since the last graphics hook (parallel_function_entry gave it + // back already for any other hook). + if (!graphics) return; + entry.graphics_kept = false; + if (graphics_lock_stats) graphics_counters().resumed.fetch_add(1, std::memory_order_relaxed); + graphics_holder_function.store(address, std::memory_order_relaxed); + entry.graphics_stack_pointer = ctx.r1.u32; + return; + } if (entry.graphics_stack_pointer) { - if (graphics) entry.graphics_stack_pointer = (std::max)(entry.graphics_stack_pointer, ctx.r1.u32); - else if (ctx.r1.u32 > entry.graphics_stack_pointer) { + if (graphics) { + entry.graphics_stack_pointer = (std::max)(entry.graphics_stack_pointer, ctx.r1.u32); + if (graphics_lock_stats) graphics_counters().nested.fetch_add(1, std::memory_order_relaxed); + } else if (ctx.r1.u32 > entry.graphics_stack_pointer) { entry.graphics_stack_pointer = 0; graphics_lock.unlock(); + graphics_lock_released(); + } else if (graphics_lock_stats) { + graphics_counters().inner.fetch_add(1, std::memory_order_relaxed); } return; } if (!graphics) return; - if (!graphics_lock.try_lock()) + if (graphics_lock_stats) { + auto& counters = graphics_counters(); + counters.taken.fetch_add(1, std::memory_order_relaxed); + if (graphics_released_hard == graphics_hard_epoch) { + counters.loose.fetch_add(1, std::memory_order_relaxed); + if (graphics_released_soft == graphics_soft_epoch) { + counters.strict.fetch_add(1, std::memory_order_relaxed); + const uint32_t spent = graphics_released_countdown - entry.checkpoint_countdown; + const int bucket = spent == 0 ? 0 : spent == 1 ? 1 : spent < 4 ? 2 : spent < 16 ? 3 : spent < 64 ? 4 : 5; + counters.gap[bucket].fetch_add(1, std::memory_order_relaxed); + } + } + if (!graphics_lock.try_lock()) { + const uint64_t started = stats_now_ns(); + graphics_waiters.fetch_add(1, std::memory_order_relaxed); + lock_with_report(graphics_lock, "graphics", graphics_holder, graphics_holder_function); + graphics_waiters.fetch_sub(1, std::memory_order_relaxed); + counters.contended.fetch_add(1, std::memory_order_relaxed); + counters.wait_ns.fetch_add(stats_now_ns() - started, std::memory_order_relaxed); + } + } else if (!graphics_lock.try_lock()) { + graphics_waiters.fetch_add(1, std::memory_order_relaxed); lock_with_report(graphics_lock, "graphics", graphics_holder, graphics_holder_function); + graphics_waiters.fetch_sub(1, std::memory_order_relaxed); + } graphics_holder.store(current_id, std::memory_order_relaxed); graphics_holder_function.store(address, std::memory_order_relaxed); entry.graphics_stack_pointer = ctx.r1.u32; @@ -611,7 +770,15 @@ static void graphics_lock_entry(const PPCContext& ctx, uint32_t address, GuestEn void hook_scope_exit(GuestEntryState& entry) noexcept { if (entry.graphics_stack_pointer) { entry.graphics_stack_pointer = 0; - graphics_lock.unlock(); + // Kept for the next graphics hook (graphics_lock_defer), unless a + // thread waits for it. + if (entry.graphics_keep && !graphics_waiters.load(std::memory_order_relaxed)) { + entry.graphics_kept = true; + if (graphics_lock_stats) graphics_counters().kept.fetch_add(1, std::memory_order_relaxed); + } else { + graphics_lock.unlock(); + graphics_lock_released(); + } } if (entry.input_stack_pointer) { entry.input_stack_pointer = 0; @@ -632,12 +799,27 @@ void hook_scope_exit(GuestEntryState& entry) noexcept { // the native device (an import dropping the renderer's copies of freed memory). using HostGraphicsLock = HostGraphicsScope; HostGraphicsScope::HostGraphicsScope() { - if (!parallel_main || guest_thread_state.entry.graphics_stack_pointer) return; + if (!parallel_main || guest_thread_state.entry.graphics_stack_pointer) return; // inside a graphics hook + // Kept since a graphics hook (graphics_lock_defer): the scope takes it + // over and gives it back at its end, so nothing inside gives it back early. + if (guest_thread_state.entry.graphics_kept) { + guest_thread_state.entry.graphics_kept = false; + owned_ = true; + return; + } + if (graphics_lock_stats) graphics_counters().scope.fetch_add(1, std::memory_order_relaxed); if (!graphics_lock.try_lock()) { + const uint64_t started = graphics_lock_stats ? stats_now_ns() : 0; const auto wait = [](std::stop_token) { graphics_lock.lock(); }; + graphics_waiters.fetch_add(1, std::memory_order_relaxed); if (!execution_permit) graphics_lock.lock(); else if (execution_permit->detached()) execution_permit->run_wait(wait); else execution_permit->run_blocking(wait); + graphics_waiters.fetch_sub(1, std::memory_order_relaxed); + if (graphics_lock_stats) { + graphics_counters().scope_contended.fetch_add(1, std::memory_order_relaxed); + graphics_counters().scope_wait_ns.fetch_add(stats_now_ns() - started, std::memory_order_relaxed); + } } owned_ = true; } @@ -670,14 +852,25 @@ static void menu_lock_entry(const PPCContext& ctx, uint32_t address, GuestEntryS hook_lock_entry(ctx, address, menu_hook_bits, menu_lock, entry.menu_stack_pointer, "menu", menu_holder, menu_holder_function); } +static const bool parallel_hook_stats = [] { const char* t = std::getenv("SFR_PARALLEL_HOOK_STATS"); return t && *t == '1'; }(); static void parallel_function_entry(const PPCContext& ctx, uint32_t address, GuestEntryState& entry) { - static const bool hook_stats = [] { const char* t = std::getenv("SFR_PARALLEL_HOOK_STATS"); return t && *t == '1'; }(); + const bool hook_stats = parallel_hook_stats; + if (graphics_lock_stats && is_hook(address) && !test_hook_bit(graphics_hook_bits, address)) graphics_lock_interrupted(true); + // A kept graphics lock goes back before any other hook (and before its + // menu or input lock: those come first in the lock order). + if (entry.graphics_kept && is_hook(address) && !test_hook_bit(graphics_hook_bits, address)) + give_back_kept_graphics(entry); if (parallel_main) { // Menu, then input, then graphics: a menu hook's original reads the // skeletons and draws, never the other way round. menu_lock_entry(ctx, address, entry); input_lock_entry(ctx, address, entry); graphics_lock_entry(ctx, address, entry); + // With the subsystem locks a hook takes no permit, so a thread that + // follows no attached hook and no slow access has nothing more to do + // here, attached or not (the branches below agree): this skips the + // permit's thread_local, which every graphics hook paid for. + if (import_locks && !entry.hook_stack_pointer && !entry.detach_at_entry) return; } if (execution_permit->detached()) { if (!is_hook(address)) return; @@ -805,6 +998,9 @@ void guest_checkpoint_permit() { return value; }(); guest_thread_state.entry.checkpoint_countdown = interval - 1; + graphics_lock_interrupted(false); + // Before the permit can hand this core to another thread. + give_back_kept_graphics_lock(guest_thread_state.entry); if (!execution_permit) throw std::logic_error("guest instruction without execution permit"); // Validates ownership and cancellation before shared memory access, and // hands off only outside a reservation. @@ -1207,6 +1403,8 @@ void wait_without_permit(void (*wait)(void*), void* argument) { traced_host_wait(wait, argument, "host", -1); } void traced_host_wait(void (*wait)(void*), void* argument, const char* kind, int64_t timeout_ns) { + // Host code waits only without a kept graphics lock (graphics_lock_defer). + give_back_kept_graphics_lock(guest_thread_state.entry); if (!execution_permit) { wait(argument); return; } block_guest([&](std::stop_token) { wait(argument); }, {kind, current_context ? uint32_t(current_context->lr) : 0, {guest_thread_state.entry.current_address}, timeout_ns}); @@ -1350,6 +1548,8 @@ struct ImportLockScope { }; void dispatch_import(PPCContext& ctx, const char* name, uint32_t address) { + graphics_lock_interrupted(true); + give_back_kept_graphics_lock(entry_thread_state(ctx).entry); // SFR_MAIN_IMPORT_TIME=1: every five seconds, the main thread's time in // each import (waits included) since the last report. static const bool main_import_time = [] { @@ -5155,6 +5355,9 @@ int main(int argc, char** argv) { sfr::GuestMemory::slow_access_hook = sfr::parallel_slow_access; permit->detach(); } + // The main thread keeps the graphics lock between graphics hooks. + sfr::guest_thread_state.entry.graphics_keep = sfr::parallel_main && sfr::graphics_lock_defer; + std::cerr << "GRAPHICS_LOCK_DEFER " << (sfr::guest_thread_state.entry.graphics_keep ? 1 : 0) << '\n'; sfr::refresh_entry_observation(); std::cerr << "GUEST_HOST_THREAD guest_id=1 tid=" << sfr::host_thread_id() << " worker=0x824d22f0\n"; #ifdef _WIN32 @@ -5171,6 +5374,8 @@ int main(int argc, char** argv) { } catch (...) { execution.fail(std::current_exception()); } + // A stop thrown out of a graphics hook leaves the lock kept. + sfr::give_back_kept_graphics_lock(sfr::guest_thread_state.entry); sfr::flush_main_wait_trace(sfr::present_count.load(), true); // How often a guest beside the permit came back to it, by reason. std::cerr << "PARALLEL_ATTACHES imports=" << sfr::parallel_attaches[0] << " hooks=" << sfr::parallel_attaches[1] diff --git a/src/guest_graphics_hooks.cpp b/src/guest_graphics_hooks.cpp index 5a9c55ff..850ed998 100644 --- a/src/guest_graphics_hooks.cpp +++ b/src/guest_graphics_hooks.cpp @@ -54,6 +54,7 @@ PPC_FUNC_IMPL(__imp__sub_824ED588); PPC_FUNC_IMPL(__imp__sub_82500B38); namespace sfr { +std::string graphics_lock_report(uint64_t present, unsigned frames); // diagnostic_main.cpp bool graphics_trace() { static const bool on=[]{ const char* t=std::getenv("SFR_TRACE_GRAPHICS"); return !t || *t!='0'; }(); return on; @@ -629,6 +630,14 @@ SFR_GRAPHICS_HOST_HOOK(sub_824E65A0) { if(!line.empty()) std::cerr << line; } #endif + { // SFR_GRAPHICS_LOCK_STATS (diagnostic_main.cpp) + static unsigned lock_frames=0; + if(++lock_frames==120) { + const std::string line=sfr::graphics_lock_report(sfr::present_count.load(),lock_frames); + if(!line.empty()) std::cerr << line; + lock_frames=0; + } + } // Consume counters even in quiet mode so observation never changes their // frame boundaries. Only detailed mode formats the large diagnostic line. const auto pipeline_work=graphics().renderer().take_pipeline_work(); diff --git a/src/guest_register_hooks.cpp b/src/guest_register_hooks.cpp index 58e912a0..310851d0 100644 --- a/src/guest_register_hooks.cpp +++ b/src/guest_register_hooks.cpp @@ -6,252 +6,252 @@ // Checkpoint before the fast path; fallback runs the original body. PPC_FUNC_IMPL(__imp____savegprlr_14); -SFR_CONCURRENT_HOOK(__savegprlr_14) { +SFR_REGISTER_HOOK(__savegprlr_14) { sfr::enter_function(ctx, "__savegprlr_14", 0x82A56030); if (!sfr::try_save_gprs<14>(*sfr::active_memory, ctx)) __imp____savegprlr_14(ctx, base); } PPC_FUNC_IMPL(__imp____restgprlr_14); -SFR_CONCURRENT_HOOK(__restgprlr_14) { +SFR_REGISTER_HOOK(__restgprlr_14) { sfr::enter_function(ctx, "__restgprlr_14", 0x82A56080); if (!sfr::try_restore_gprs<14>(*sfr::active_memory, ctx)) __imp____restgprlr_14(ctx, base); } PPC_FUNC_IMPL(__imp____savegprlr_15); -SFR_CONCURRENT_HOOK(__savegprlr_15) { +SFR_REGISTER_HOOK(__savegprlr_15) { sfr::enter_function(ctx, "__savegprlr_15", 0x82A56034); if (!sfr::try_save_gprs<15>(*sfr::active_memory, ctx)) __imp____savegprlr_15(ctx, base); } PPC_FUNC_IMPL(__imp____restgprlr_15); -SFR_CONCURRENT_HOOK(__restgprlr_15) { +SFR_REGISTER_HOOK(__restgprlr_15) { sfr::enter_function(ctx, "__restgprlr_15", 0x82A56084); if (!sfr::try_restore_gprs<15>(*sfr::active_memory, ctx)) __imp____restgprlr_15(ctx, base); } PPC_FUNC_IMPL(__imp____savegprlr_16); -SFR_CONCURRENT_HOOK(__savegprlr_16) { +SFR_REGISTER_HOOK(__savegprlr_16) { sfr::enter_function(ctx, "__savegprlr_16", 0x82A56038); if (!sfr::try_save_gprs<16>(*sfr::active_memory, ctx)) __imp____savegprlr_16(ctx, base); } PPC_FUNC_IMPL(__imp____restgprlr_16); -SFR_CONCURRENT_HOOK(__restgprlr_16) { +SFR_REGISTER_HOOK(__restgprlr_16) { sfr::enter_function(ctx, "__restgprlr_16", 0x82A56088); if (!sfr::try_restore_gprs<16>(*sfr::active_memory, ctx)) __imp____restgprlr_16(ctx, base); } PPC_FUNC_IMPL(__imp____savegprlr_17); -SFR_CONCURRENT_HOOK(__savegprlr_17) { +SFR_REGISTER_HOOK(__savegprlr_17) { sfr::enter_function(ctx, "__savegprlr_17", 0x82A5603C); if (!sfr::try_save_gprs<17>(*sfr::active_memory, ctx)) __imp____savegprlr_17(ctx, base); } PPC_FUNC_IMPL(__imp____restgprlr_17); -SFR_CONCURRENT_HOOK(__restgprlr_17) { +SFR_REGISTER_HOOK(__restgprlr_17) { sfr::enter_function(ctx, "__restgprlr_17", 0x82A5608C); if (!sfr::try_restore_gprs<17>(*sfr::active_memory, ctx)) __imp____restgprlr_17(ctx, base); } PPC_FUNC_IMPL(__imp____savegprlr_18); -SFR_CONCURRENT_HOOK(__savegprlr_18) { +SFR_REGISTER_HOOK(__savegprlr_18) { sfr::enter_function(ctx, "__savegprlr_18", 0x82A56040); if (!sfr::try_save_gprs<18>(*sfr::active_memory, ctx)) __imp____savegprlr_18(ctx, base); } PPC_FUNC_IMPL(__imp____restgprlr_18); -SFR_CONCURRENT_HOOK(__restgprlr_18) { +SFR_REGISTER_HOOK(__restgprlr_18) { sfr::enter_function(ctx, "__restgprlr_18", 0x82A56090); if (!sfr::try_restore_gprs<18>(*sfr::active_memory, ctx)) __imp____restgprlr_18(ctx, base); } PPC_FUNC_IMPL(__imp____savegprlr_19); -SFR_CONCURRENT_HOOK(__savegprlr_19) { +SFR_REGISTER_HOOK(__savegprlr_19) { sfr::enter_function(ctx, "__savegprlr_19", 0x82A56044); if (!sfr::try_save_gprs<19>(*sfr::active_memory, ctx)) __imp____savegprlr_19(ctx, base); } PPC_FUNC_IMPL(__imp____restgprlr_19); -SFR_CONCURRENT_HOOK(__restgprlr_19) { +SFR_REGISTER_HOOK(__restgprlr_19) { sfr::enter_function(ctx, "__restgprlr_19", 0x82A56094); if (!sfr::try_restore_gprs<19>(*sfr::active_memory, ctx)) __imp____restgprlr_19(ctx, base); } PPC_FUNC_IMPL(__imp____savegprlr_20); -SFR_CONCURRENT_HOOK(__savegprlr_20) { +SFR_REGISTER_HOOK(__savegprlr_20) { sfr::enter_function(ctx, "__savegprlr_20", 0x82A56048); if (!sfr::try_save_gprs<20>(*sfr::active_memory, ctx)) __imp____savegprlr_20(ctx, base); } PPC_FUNC_IMPL(__imp____restgprlr_20); -SFR_CONCURRENT_HOOK(__restgprlr_20) { +SFR_REGISTER_HOOK(__restgprlr_20) { sfr::enter_function(ctx, "__restgprlr_20", 0x82A56098); if (!sfr::try_restore_gprs<20>(*sfr::active_memory, ctx)) __imp____restgprlr_20(ctx, base); } PPC_FUNC_IMPL(__imp____savegprlr_21); -SFR_CONCURRENT_HOOK(__savegprlr_21) { +SFR_REGISTER_HOOK(__savegprlr_21) { sfr::enter_function(ctx, "__savegprlr_21", 0x82A5604C); if (!sfr::try_save_gprs<21>(*sfr::active_memory, ctx)) __imp____savegprlr_21(ctx, base); } PPC_FUNC_IMPL(__imp____restgprlr_21); -SFR_CONCURRENT_HOOK(__restgprlr_21) { +SFR_REGISTER_HOOK(__restgprlr_21) { sfr::enter_function(ctx, "__restgprlr_21", 0x82A5609C); if (!sfr::try_restore_gprs<21>(*sfr::active_memory, ctx)) __imp____restgprlr_21(ctx, base); } PPC_FUNC_IMPL(__imp____savegprlr_22); -SFR_CONCURRENT_HOOK(__savegprlr_22) { +SFR_REGISTER_HOOK(__savegprlr_22) { sfr::enter_function(ctx, "__savegprlr_22", 0x82A56050); if (!sfr::try_save_gprs<22>(*sfr::active_memory, ctx)) __imp____savegprlr_22(ctx, base); } PPC_FUNC_IMPL(__imp____restgprlr_22); -SFR_CONCURRENT_HOOK(__restgprlr_22) { +SFR_REGISTER_HOOK(__restgprlr_22) { sfr::enter_function(ctx, "__restgprlr_22", 0x82A560A0); if (!sfr::try_restore_gprs<22>(*sfr::active_memory, ctx)) __imp____restgprlr_22(ctx, base); } PPC_FUNC_IMPL(__imp____savegprlr_23); -SFR_CONCURRENT_HOOK(__savegprlr_23) { +SFR_REGISTER_HOOK(__savegprlr_23) { sfr::enter_function(ctx, "__savegprlr_23", 0x82A56054); if (!sfr::try_save_gprs<23>(*sfr::active_memory, ctx)) __imp____savegprlr_23(ctx, base); } PPC_FUNC_IMPL(__imp____restgprlr_23); -SFR_CONCURRENT_HOOK(__restgprlr_23) { +SFR_REGISTER_HOOK(__restgprlr_23) { sfr::enter_function(ctx, "__restgprlr_23", 0x82A560A4); if (!sfr::try_restore_gprs<23>(*sfr::active_memory, ctx)) __imp____restgprlr_23(ctx, base); } PPC_FUNC_IMPL(__imp____savegprlr_24); -SFR_CONCURRENT_HOOK(__savegprlr_24) { +SFR_REGISTER_HOOK(__savegprlr_24) { sfr::enter_function(ctx, "__savegprlr_24", 0x82A56058); if (!sfr::try_save_gprs<24>(*sfr::active_memory, ctx)) __imp____savegprlr_24(ctx, base); } PPC_FUNC_IMPL(__imp____restgprlr_24); -SFR_CONCURRENT_HOOK(__restgprlr_24) { +SFR_REGISTER_HOOK(__restgprlr_24) { sfr::enter_function(ctx, "__restgprlr_24", 0x82A560A8); if (!sfr::try_restore_gprs<24>(*sfr::active_memory, ctx)) __imp____restgprlr_24(ctx, base); } PPC_FUNC_IMPL(__imp____savegprlr_25); -SFR_CONCURRENT_HOOK(__savegprlr_25) { +SFR_REGISTER_HOOK(__savegprlr_25) { sfr::enter_function(ctx, "__savegprlr_25", 0x82A5605C); if (!sfr::try_save_gprs<25>(*sfr::active_memory, ctx)) __imp____savegprlr_25(ctx, base); } PPC_FUNC_IMPL(__imp____restgprlr_25); -SFR_CONCURRENT_HOOK(__restgprlr_25) { +SFR_REGISTER_HOOK(__restgprlr_25) { sfr::enter_function(ctx, "__restgprlr_25", 0x82A560AC); if (!sfr::try_restore_gprs<25>(*sfr::active_memory, ctx)) __imp____restgprlr_25(ctx, base); } PPC_FUNC_IMPL(__imp____savegprlr_26); -SFR_CONCURRENT_HOOK(__savegprlr_26) { +SFR_REGISTER_HOOK(__savegprlr_26) { sfr::enter_function(ctx, "__savegprlr_26", 0x82A56060); if (!sfr::try_save_gprs<26>(*sfr::active_memory, ctx)) __imp____savegprlr_26(ctx, base); } PPC_FUNC_IMPL(__imp____restgprlr_26); -SFR_CONCURRENT_HOOK(__restgprlr_26) { +SFR_REGISTER_HOOK(__restgprlr_26) { sfr::enter_function(ctx, "__restgprlr_26", 0x82A560B0); if (!sfr::try_restore_gprs<26>(*sfr::active_memory, ctx)) __imp____restgprlr_26(ctx, base); } PPC_FUNC_IMPL(__imp____savegprlr_27); -SFR_CONCURRENT_HOOK(__savegprlr_27) { +SFR_REGISTER_HOOK(__savegprlr_27) { sfr::enter_function(ctx, "__savegprlr_27", 0x82A56064); if (!sfr::try_save_gprs<27>(*sfr::active_memory, ctx)) __imp____savegprlr_27(ctx, base); } PPC_FUNC_IMPL(__imp____restgprlr_27); -SFR_CONCURRENT_HOOK(__restgprlr_27) { +SFR_REGISTER_HOOK(__restgprlr_27) { sfr::enter_function(ctx, "__restgprlr_27", 0x82A560B4); if (!sfr::try_restore_gprs<27>(*sfr::active_memory, ctx)) __imp____restgprlr_27(ctx, base); } PPC_FUNC_IMPL(__imp____savegprlr_28); -SFR_CONCURRENT_HOOK(__savegprlr_28) { +SFR_REGISTER_HOOK(__savegprlr_28) { sfr::enter_function(ctx, "__savegprlr_28", 0x82A56068); if (!sfr::try_save_gprs<28>(*sfr::active_memory, ctx)) __imp____savegprlr_28(ctx, base); } PPC_FUNC_IMPL(__imp____restgprlr_28); -SFR_CONCURRENT_HOOK(__restgprlr_28) { +SFR_REGISTER_HOOK(__restgprlr_28) { sfr::enter_function(ctx, "__restgprlr_28", 0x82A560B8); if (!sfr::try_restore_gprs<28>(*sfr::active_memory, ctx)) __imp____restgprlr_28(ctx, base); } PPC_FUNC_IMPL(__imp____savegprlr_29); -SFR_CONCURRENT_HOOK(__savegprlr_29) { +SFR_REGISTER_HOOK(__savegprlr_29) { sfr::enter_function(ctx, "__savegprlr_29", 0x82A5606C); if (!sfr::try_save_gprs<29>(*sfr::active_memory, ctx)) __imp____savegprlr_29(ctx, base); } PPC_FUNC_IMPL(__imp____restgprlr_29); -SFR_CONCURRENT_HOOK(__restgprlr_29) { +SFR_REGISTER_HOOK(__restgprlr_29) { sfr::enter_function(ctx, "__restgprlr_29", 0x82A560BC); if (!sfr::try_restore_gprs<29>(*sfr::active_memory, ctx)) __imp____restgprlr_29(ctx, base); } PPC_FUNC_IMPL(__imp____savegprlr_30); -SFR_CONCURRENT_HOOK(__savegprlr_30) { +SFR_REGISTER_HOOK(__savegprlr_30) { sfr::enter_function(ctx, "__savegprlr_30", 0x82A56070); if (!sfr::try_save_gprs<30>(*sfr::active_memory, ctx)) __imp____savegprlr_30(ctx, base); } PPC_FUNC_IMPL(__imp____restgprlr_30); -SFR_CONCURRENT_HOOK(__restgprlr_30) { +SFR_REGISTER_HOOK(__restgprlr_30) { sfr::enter_function(ctx, "__restgprlr_30", 0x82A560C0); if (!sfr::try_restore_gprs<30>(*sfr::active_memory, ctx)) __imp____restgprlr_30(ctx, base); } PPC_FUNC_IMPL(__imp____savegprlr_31); -SFR_CONCURRENT_HOOK(__savegprlr_31) { +SFR_REGISTER_HOOK(__savegprlr_31) { sfr::enter_function(ctx, "__savegprlr_31", 0x82A56074); if (!sfr::try_save_gprs<31>(*sfr::active_memory, ctx)) __imp____savegprlr_31(ctx, base); } PPC_FUNC_IMPL(__imp____restgprlr_31); -SFR_CONCURRENT_HOOK(__restgprlr_31) { +SFR_REGISTER_HOOK(__restgprlr_31) { sfr::enter_function(ctx, "__restgprlr_31", 0x82A560C4); if (!sfr::try_restore_gprs<31>(*sfr::active_memory, ctx)) __imp____restgprlr_31(ctx, base); diff --git a/src/guest_thread_state.h b/src/guest_thread_state.h index cc55f8b3..9b70e9b9 100644 --- a/src/guest_thread_state.h +++ b/src/guest_thread_state.h @@ -45,7 +45,16 @@ struct GuestEntryState { // Hooks this thread is inside (HookScope): the outermost one's return // gives back what its entry took. uint32_t hook_depth = 0; + // The graphics lock kept from one graphics hook to the next + // (SFR_GRAPHICS_LOCK_DEFER, diagnostic_main.cpp's graphics_lock_defer): + // held outside every hook until the thread reaches anything that may + // wait or take another lock. graphics_keep: this thread may keep it. + // In the padding after hook_depth, so the fields the generated code + // reads keep their offsets. + bool graphics_kept = false; + bool graphics_keep = false; }; +static_assert(sizeof(GuestEntryState) == 48, "GuestEntryState grew: check the generated code's offsets"); // A function entry and its scalar stores repeatedly consult both records. // One TLS object lets ARM64 keep a single resolved address across those