obs-ffmpeg: Fix AMD AV1 CBR filler data bug - #13780
Conversation
The AV1 rate control function compared the AV1 rate control value against H.264 enum constants instead of AV1 enum constants, which have different numeric values. This caused filler data to never be enabled for AV1 CBR, resulting in unstable bitrate. Also enable filler data for HIGH_QUALITY_CBR on all codecs (AVC, HEVC, AV1), which was only enabled for plain CBR before. Fixes obsproject#12013.
|
To what extent was AI tooling used in the creation of this pull request? |
It was used to help in the identification of the cause by parsing the codebase and the AMF SDK headers. For completeness, the headers can be found here: |
Description
The AV1 rate control function uses H.264 enum constants instead of AV1 ones, and the enum values differ (H.264 CBR=1, AV1 CBR=3), so the CBR filler data check never matches (the value 3 is used by H.264 for LATENCY_CONSTRAINED_VBR).
Also, HQCBR doesn't enable filler data on any codec (AVC/HEVC/AV1), only plain CBR does.
This PR simply swaps the enum constants with the correct ones, and adds HQCBR to the filler data check on all three codecs.
Motivation and Context
Without filler data, AV1 CBR can't maintain constant bitrate and the encoder is liable to overshoots, leading to frame drops on YouTube.
Fixes #12013.
How Has This Been Tested?
Testing environment:
Steps:
the fixed version produced 1220 sub-100-byte frames (filler data padding) vs only 116 in the original version
Types of changes
Checklist: