fix(runtime): install the crash handler from the executable's copy only

CHANGELOG.md

@@ -78,6 +78,14 @@ pull request.
   allocated, and each load of the DLL built another one: 488 bytes per load and unload in the
   measurement of #227, whose DLL decoded two names. The copy now frees the cache when `FreeLibrary`
   unloads the DLL, after the DLL's destructors, whose boxes still name types (#227).
+- **The crash handler of an unloaded DLL, on Windows.** With `CT_BACKTRACE`, the copy of the
+  runtime in a DLL that links it installed its crash handler over the filter installed before it,
+  and nothing put that filter back. Once `FreeLibrary` had unloaded the DLL, an exception that
+  nothing handled ended the process without a backtrace: neither the executable's handler nor a
+  filter that the program had installed before the DLL ran. Only the executable's copy installs the
+  crash handler now: it reports the crashes in an instrumented DLL's code too, and a DLL's copy
+  leaves the process's filter alone. An instrumented DLL that a program without the runtime loads
+  reports no crash any more (README, "Crash handler on Windows") (#236).
 
 ### Tests and CI
 

README.md

@@ -284,6 +284,25 @@ runtime as follows:
   have unloaded. Once that copy has stopped following the loader, when the module that holds it
   detaches, the handler prints every frame bare.
 
+## Crash handler on Windows
+
+With `CT_BACKTRACE` set, only the executable's copy of the runtime installs a crash handler for the
+whole process, with `SetUnhandledExceptionFilter`, when the executable initializes. The copy in a
+DLL that links the runtime installs none and leaves the process's filter alone, while the DLL is
+loaded and when it unloads; `FreeLibrary` unloads the DLL as any other (#236).
+
+- In an executable that links the runtime, its handler reports the crashes of the whole process,
+  those in the code of an instrumented DLL included, while the DLL is loaded and once it has
+  unloaded.
+- A filter that the program installs, before or after it loads such a DLL, stays the process's.
+- **Limit.** In a program that does not link the runtime, an instrumented DLL reports no crash,
+  even with `CT_BACKTRACE` set: the program's own filter, if any, or Windows handles it.
+
+Windows has no call that sets the process's filter only if it still is the one expected. A DLL's
+copy that installed its handler could not take it out when the DLL unloads without overwriting a
+filter that the program or another thread installs meanwhile, and a filter installed over the
+handler would call it once its code is gone.
+
 ## LLVM versions
 
 CoreTrace Compiler builds and instruments with LLVM 16 to 23. Most behaviour is the same on

src/runtime/windows/ct_runtime_backtrace.cpp

@@ -1,4 +1,12 @@
 // SPDX-License-Identifier: Apache-2.0
+//
+// The crash handler of the runtime, which CT_BACKTRACE installs for the whole process from the
+// executable's copy of the runtime only. A DLL that links the runtime has a copy of its own, which
+// installs none: Windows has no call that puts the process's filter back only if it still is the
+// one expected, so a DLL's copy could not take its handler out when FreeLibrary unloads the DLL
+// without overwriting a filter that the program installs meanwhile, and a filter installed over the
+// handler would call it once its code is gone. A crash in the DLL is reported by the executable's
+// handler; an instrumented DLL that a program without the runtime loads reports none (#236).
 #include "ct_runtime_internal.h"
 #include "ct_runtime_helpers.h"
 
@@ -14,6 +22,9 @@
 
 #pragma comment(lib, "Dbghelp.lib")
 
+// The module that holds this copy of the runtime.
+extern "C" IMAGE_DOS_HEADER __ImageBase;
+
 namespace
 {
     std::atomic<int> ct_backtrace_installed{0};
@@ -84,7 +95,9 @@ namespace
 
 CT_NOINSTR void ct_maybe_install_backtrace(void)
 {
-    if (std::getenv("CT_BACKTRACE") == nullptr)
+    // A DLL's copy leaves the process's filter alone: see the top of this file.
+    if (std::getenv("CT_BACKTRACE") == nullptr ||
+        reinterpret_cast<HMODULE>(&__ImageBase) != GetModuleHandleW(nullptr))
     {
         return;
     }

test/unit/reload/ct_runtime_library.cpp

@@ -3,9 +3,14 @@
 // A DLL that links the runtime, as an instrumented DLL does, and with it the runtime's watch of
 // the loader, its DbgHelp session and the DbgHelp lock of the process:
 // runtime_vtable_replace_test.cpp makes its copy of the runtime name a virtual call's target, or
-// not, then unloads it, when its destructors name one, and loads other DLLs.
+// not, then unloads it, when its destructors name one, and loads other DLLs, or has it fault.
 #define CT_RUNTIME_LIBRARY_EXPORT extern "C" __declspec(dllexport)
 
+#ifndef NOMINMAX
+#define NOMINMAX
+#endif
+#include <windows.h>
+
 extern "C" void __ct_vcall_trace(void* this_ptr, void* target, const char* site,
                                  const char* static_type);
 
@@ -49,3 +54,17 @@ CT_RUNTIME_LIBRARY_EXPORT void ct_runtime_library_trace()
     __ct_vcall_trace(&gadget, reinterpret_cast<void*>(&ct_runtime_library_measure),
                      "runtime_library:1", "<unknown>");
 }
+
+namespace
+{
+    volatile int faulted = 0;
+} // namespace
+
+// Raises an exception that nothing handles, in the DLL's own code: the handler of the executable's
+// copy of the runtime reports it, the DLL's copy installing none (#236). The store after the call
+// keeps this frame on the stack, which a tail call would leave out.
+CT_RUNTIME_LIBRARY_EXPORT void ct_runtime_library_fault()
+{
+    RaiseException(0xE0000225, EXCEPTION_NONCONTINUABLE, 0, nullptr);
+    faulted = 1;
+}