obs-qsv11: Close MFX session and unload loader

[lazyboson]

Description

Initialize() in the Windows backend creates an mfxLoader and an MFX session, but never releases either. The data out-parameter was unused (UNUSED_PARAMETER(data)) and ReleaseSessionData() was an empty stub, so the loader was dropped on the floor. ClearData() also only called MFXVideoENCODE_Close(), never MFXClose(), so the session itself was leaked too.

This stores the loader in the data pointer that Initialize() already receives, and closes the session and unloads the loader in ClearData(). MFXClose() is moved above the Release() block because the session's scheduler still holds the D3D11 device handle installed by MFXVideoCORE_SetHandle() — releasing the device first and closing the session afterwards [describe what you saw: hang on shutdown / access violation in libmfx / etc].

The Linux backend already tracked per-session state, so this just adds the loader to linux_data and unloads it alongside the VADisplay. Both backends also now clean up the loader on the MFXCreateSession() failure path, which previously returned early via MSDK_CHECK_RESULT and leaked.

Motivation and Context

I have been developing a cross-platform screen-recording application on top of libobs, during soak testing of application over 24 hours when i was using the intel encoding driver, memory was growing continously, i have taken dump of memory and analyzed, initially i though may be intel driver is culprit but on further analysis this dll seems have issue due to following reason - libobs creates a fresh encoder instance every time the output is activated, so one session and one loader leak per recording. Each leaked session keeps roughly [N] threads alive and holds [N] MB.Thread count grows the same way — [N] at baseline, [N] after 8 recordings on stock, flat on the patched build.

This matters most for long-running or automated setups. I hit it building a recording application on top of libobs, where OBS stays up across many start/stop cycles.

How Has This Been Tested?

I have tested this on Windows 11, with Intel Iris Xe, driver 32.0.101.7088. OBS Studio 32.2.1 portable was installed having advanced output with obs_qsv11_v2 (H.264), I have used video configuration 1280x720, 30fps and CBR 5000. I have started and stopped recording eight times, watching memory and thread count after each stop, on both the shipped obs-qsv11.dll and a build from this branch. Recordings play back fine and I saw no shutdown crashes or hangs.

I have not tested the Linux path — I don't have a VA-API machine set up. That change mirrors the existing vaTerminate/close(fd) teardown, but it would be good to have someone confirm it.

Types of changes

• Code cleanup (non-breaking change which makes code smaller or more readable)

Checklist:

• I have read the contributing document.
• My code has been run through clang-format.
• My code follows the project's style guidelines
• My code is not on the master branch.
• My code has been tested.
• All commit messages are properly formatted and commits squashed where appropriate.
• I have included updates to all appropriate documentation.

[Fenrirthviti]

Please properly fill out the PR template.

This smells AI-generated, which is not permitted per our contribution policy.

[lazyboson]

Please properly fill out the PR template.

This smells AI-generated, which is not permitted per our contribution policy.

Sorry — the description was badly written and I mangled the template. Will redo the description, but there is a genuine thread and memory leak, when we use the intel gpu. I am building a screen recording application on top of libobs.

The code is mine and I can walk through the session/loader teardown ordering or
the non-Windows path if that's useful.

[Fenrirthviti]

Thanks, please update the description to properly fill out the template, and we will review.