lua: stop dt.control.execute from flashing a console window on Windows

[radialmonster]

Summary

dt.control.execute() uses system(). On Windows, that launches a console
subprocess from darktable's console-less GUI process and causes a visible
console-window flash.

This PR adds dt_exec_command_sync() and routes dt.control.execute() through
it. On Windows, the helper invokes the configured COMSPEC interpreter with
CREATE_NO_WINDOW; if COMSPEC is unavailable, it uses the cmd.exe in the
Windows system directory. Other platforms continue to use system().

The Windows helper resolves the interpreter to an explicit full path and passes
it as CreateProcessW's application name. This avoids the application-directory
and current-directory executable search that occurs when the application name
is NULL, while preserving shell builtins, batch files, redirection, chaining,
Unicode commands, and configured COMSPEC behavior.

Validation

• Fresh MSYS2/UCRT64 Release build of lib_darktable and test_process.
• test_process: 6/6 passing, covering success, shell builtins, nonzero status,
  NULL, a controlled current-directory cmd.exe decoy, and COMSPEC
  selection.
• The current-directory regression fails against the original implementation,
  where the decoy is launched, and passes with the explicit interpreter path.
• Earlier end-to-end testing with a built darktable.exe showed the same exit
  status with the flash removed for a persistent dt.control.execute() child.

Scope

This addresses only dt.control.execute(). Lua's os.execute() and
io.popen() use separate implementations and remain follow-up work, so issue
#17193 should remain open after this PR.

AI disclosure

OpenAI Codex assisted with implementation, adversarial review, and test
development. The changes were rebuilt and exercised locally as described above.

Related: #17193

[wpferguson]

If the whole point is to stop windows popping up, then changing the windows installer to start darktable using the work around #17193 makes the most sense in terms of the amount of code change required and the amount of code to be maintained.

If we are looking for more functionality, because the current tools don't provide something we need, then that's a different story. But, since all the titles involve "popping windows" I don't think that's the case.

If I take @gerritsangel's PR and this one together we're looking 1200+ lines of code to solve what can be solved with a 1 line shortcut entry.

I'll look at this, but it may take awhile since the gtk4 transition is taking up a lot of my time.

[wpferguson]

darktable.exe is a console-subsystem binary that calls FreeConsole() on startup

So is there a way to build darktable on windows so it doesn't call FreeConsole()?

EDIT:

How to Build Without FreeConsole()

Check Source Code: Search your project files for FreeConsole and remove or comment out the function call.
Configure Compiler Flags:
    If using GCC/MinGW, do not pass -mwindows if you want to keep or manage the console window. Using -mconsole (or omitting the subsystem flag entirely) keeps the console attached.

[radialmonster]

@wpferguson

We tested this directly rather than reasoning about it (fresh clone, real builds). Quick answer to the compiler-flag question first: darktable already links with -mconsole -mwindows -Wl,-subsystem,console; -mwindows is present but overridden by the explicit -subsystem,console after it, so the shipped binary is already IMAGE_SUBSYSTEM_WINDOWS_CUI (confirmed via objdump -p). We also rebuilt a true GUI-subsystem variant to check: it still flashes. The subsystem isn't the cause — only whether the process holds an attached console when it spawns a child is.

That leaves four real approaches, as far as we can tell:

1. This PR's approach (CreateProcessW + CREATE_NO_WINDOW, dt_exec_command_sync()), plus an in-progress follow-up for os.execute()/io.popen()

• Pros: no console ever appears (visible or minimized), no log pollution, works regardless of how darktable is launched.
• Cons: needs C code changes (small and isolated); this PR alone only covers dt.control.execute() — full coverage needs the follow-up, which we're actively working on.

2. Remove the FreeConsole() call in src/main.c

• Pros: one-line change, fixes the flash everywhere regardless of launch method.
• Cons: darktable's own console stays fully visible for the entire session, not a brief blip. Child stdout also leaks into darktable-log.txt — fixable separately (redirect around each system() call), but that doesn't address the window.

3. Same, plus ShowWindow(GetConsoleWindow(), SW_HIDE)

• Pros: same flash fix as Filtering #2, and the window is at least minimized rather than fully shown (on Windows Terminal).
• Cons: under Windows Terminal (the Win11 default), GetConsoleWindow() returns a stub, not the real window — so SW_HIDE only minimizes it, leaving a taskbar/Alt-Tab entry. Detecting "safe to hide" (console-process-count heuristic) has a real false positive: it can minimize a terminal tab the user deliberately opened to run darktable in — we reproduced this. Same log-pollution issue as Filtering #2.

4. Installer/shortcut-level hidden launcher + output redirect (the jeanrenaud VBS trick from earlier in this thread)

• Pros: no code changes at all; works for the whole session once launched that way, not just startup.
• Cons: only fixes whichever specific shortcut uses it — taskbar pins, "Open with", other shortcuts, and CLI launches all keep flashing. Also relocates/pollutes darktable-log.txt, and is fragile (dropping the redirect, or the log directory not existing, silently breaks it).

Happy to help pursue whichever direction you'd prefer, or do more testing/validation on any of these — whatever's most useful.

Edit - fyi Moved to draft, still working on optimizing this.