Skip to content

obs-ffmpeg: Fix AMD AV1 CBR filler data bug - #13780

Open
Shakes4cc wants to merge 1 commit into
obsproject:masterfrom
Shakes4cc:fix/amd-av1-cbr-filler-data
Open

obs-ffmpeg: Fix AMD AV1 CBR filler data bug#13780
Shakes4cc wants to merge 1 commit into
obsproject:masterfrom
Shakes4cc:fix/amd-av1-cbr-filler-data

Conversation

@Shakes4cc

Copy link
Copy Markdown

Description

The AV1 rate control function uses H.264 enum constants instead of AV1 ones, and the enum values differ (H.264 CBR=1, AV1 CBR=3), so the CBR filler data check never matches (the value 3 is used by H.264 for LATENCY_CONSTRAINED_VBR).
Also, HQCBR doesn't enable filler data on any codec (AVC/HEVC/AV1), only plain CBR does.

This PR simply swaps the enum constants with the correct ones, and adds HQCBR to the filler data check on all three codecs.

Motivation and Context

Without filler data, AV1 CBR can't maintain constant bitrate and the encoder is liable to overshoots, leading to frame drops on YouTube.
Fixes #12013.

How Has This Been Tested?

Testing environment:

  • Windows 10 22H2 x64 (with extended security updates)
  • AMD Ryzen 5 3600X, AMD Radeon RX 9060 XT, 16GB DDR4, Kingston KC3000 SSD

Steps:

  • Built obs-ffmpeg.dll from master with the change, swapped into OBS 32.2.1
  • Recorded a static scene for 32 seconds with AV1 CBR 6000 Kbps, B-frames disabled
  • With fix: steady ~6300 kbps in Stats, file 19.1 MB
  • Counter-test with original DLL: ~5345 kbps in Stats, file 20.7 MB
  • Checked both video files with ffprobe:
    the fixed version produced 1220 sub-100-byte frames (filler data padding) vs only 116 in the original version
  • Also tested HQCBR, it behaves identically to CBR with the fix

Types of changes

  • Bug fix (non-breaking change which fixes an issue)

Checklist:

  • I have read the contributing document.
  • My code has been run through clang-format.
  • My code follows the project's style guidelines
  • My code is not on the master branch.
  • My code has been tested.
  • All commit messages are properly formatted and commits squashed where appropriate.
  • I have included updates to all appropriate documentation.
  • I have used AI tooling in the creation of this PR.

The AV1 rate control function compared the AV1 rate control value
against H.264 enum constants instead of AV1 enum constants, which
have different numeric values. This caused filler data to never be
enabled for AV1 CBR, resulting in unstable bitrate.

Also enable filler data for HIGH_QUALITY_CBR on all codecs (AVC,
HEVC, AV1), which was only enabled for plain CBR before.

Fixes obsproject#12013.
@derrod

derrod commented Aug 12, 2026

Copy link
Copy Markdown
Member

To what extent was AI tooling used in the creation of this pull request?

@Shakes4cc

Copy link
Copy Markdown
Author

To what extent was AI tooling used in the creation of this pull request?

It was used to help in the identification of the cause by parsing the codebase and the AMF SDK headers.
I verified the results, wrote the fix and the PR message, and did the testing manually to confirm that the issue had been fixed (and was again present after reverting the change).

For completeness, the headers can be found here:
https://github.com/GPUOpen-LibrariesAndSDKs/AMF/blob/c35f613aea2e5057a688c979e75b1cf24253297e/amf/public/include/components/VideoEncoderAV1.h#L88
https://github.com/GPUOpen-LibrariesAndSDKs/AMF/blob/c35f613aea2e5057a688c979e75b1cf24253297e/amf/public/include/components/VideoEncoderVCE.h#L98

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

AMD HW AV1 exceeds configured bitrate in CBR

2 participants