feat: add H.264 hardware decoding

[DatCaptainHorse]

PR changes

Adds decoding support, scoped to H.264 for now so there's not too much to review at once.. though I feel it already is 😅

LLM's summary:

Adds a stream-driven H.264 decoder alongside the existing encoder, plus a
shared Vulkan Video layer (src/video.rs, src/decoder/codec.rs) used by both
directions.

- Full reference management: IDR, sliding-window, and MMCO (B-pyramid).
- Display-order output by default via an internal reorder buffer, with a
  decode-order opt-out for lowest latency; DecodeConfig gains OutputOrder
  and Decoder gains flush().
- Readback and reorder copies run on the transfer queue; fixes transfer
  queue-family selection to avoid the video engine.
- Encoder B-frame guards return errors instead of panicking.
- Rejects unsupported streams (interlaced, scaling matrices) with clear errors.

Verified byte-identical to ffmpeg across AMD (RADV) and NVIDIA, including
B-pyramid, mid-stream resolution changes, and encode->decode round trips.

Comment

Whew.. this has been atleast a few weeks worth of work, I ended up getting a claude subscription since I was losing my mind over H.264 codec parsing and dealing with all the reference frame management, it's just not fun and goes over my brain's capacity to handle 😓

This is very much a draft, meaning WIP, RFC and all. API could be better IMO, and I'd like some feedback and guidance on what you wish for @hgaiser !

Currently codec parsing and actual decode are done separately for API user:

for au in access_units(&stream) {
    let decoded = match decoder.decode(au, frame_count as u64) {
...

incase someone wants to do their own parsing or do some magic by intercepting parsed data. When adding AV1 decoding in future, the parsing API needs changing to be more codec-agnostic though.

I've verified it works though in a real use-case, here's a picture showing pixelforge decoding integrated into a native app for playing Nestri streams, H.264 is decoded and shown perfectly fine

It's a lot to unpack, despite trying to limit the scope to H.264 decoding for now, there's a lot of boilerplate and logic needed for even just that, I apologize for the size of this PR 😅

Certain code parts are somewhat oriented towards H.26X codecs, so when adding AV1 decoding in future, it will need adjusting to be more clean and sane.

Aside from decoder changes, the encoder pub fn new.. was poked a bit to change the "b-frames unsupported" assertion into an Err return instead, allowing to gracefully handle that. Though I can revert that one since it's overreaching a bit here.

[hgaiser]

I laughed when I read ... scoped to H.264 for now so there's not too much to review at once.. and looked at the amount of diff lines :p

Thanks for submitting it though, cool stuff. I had a rough implementation of decoding working a long time ago but decided to keep the scope relatively smaller by focusing only on encoding. That said, I'm completely open to adding decoding support. It will probably take some time to go over it.

For full transparency: I'll probably review the public API and use LLMs to review the internals.

[DatCaptainHorse]

Take your time @hgaiser - and feel free to be strict on demands/wants!

Also a random 🎉 - pixelforge decoding works under Windows as well :P
(excuse my cursed setup of running Windows in VM with GPU passthrough, connecting to it via Parsec.. yep)

[DatCaptainHorse]

@hgaiser Sorry for the bump, I'd just like to know when you believe you'll have the time and energy to look atleast into surface-level changes (i.e. the public/dev-facing API) 🤔

It doesn't have to be reviewed all at once, see what immediately "sticks out" or even irks you and I'll make the changes 🫡

I can also rebase against current main if it'd help, some conflicts seem to exist now.

[hgaiser]

No worries, fair question. I started looking into this yesterday, it's high on my to-do list :)

I'll try to get to it "soon" 🙄

[hgaiser]

I was wondering: would it make sense to have a similar asynchronous behavior here, comparable to the encode pipeline? It could look something like:

// Producer thread: feeds packets, never blocks on output
let producer = std::thread::spawn(move || {
    for au in access_units(&stream) {
        decoder.decode(au, pts).unwrap();   // just accepts the packet
    }
    decoder.flush().unwrap();               // signal end of stream
});

// Main/consumer thread: awaits decoded frames
loop {
    match decoder.download().await {
        Ok(Some(frame)) => render(frame),
        Ok(None)        => break,
        Err(e)          => return Err(e.into()),
    }
}

[DatCaptainHorse]

WHEW, okay I made sure to go through and pretty much refactor the PR to be more like the encoding-side.. thank heck for LLM help on this one 😅

Not to mention, the encode-API-like pipelining does give benefits, since I got no RTX 2060 anymore I temporarily rented a cloud server with L4 for testing and verifying, also found out that Intel Arc has Vulkan Video support, but behind sneaky ANV_DEBUG=video-decode,video-encode ENV var.

Results on decode speeds with pipelining API'fication:

GPU	320x240 display	320x240 decode	1080p display	1080p decode
AMD RX 9060 XT (RADV)	+19.0%	+6.3%	+57.1%	+14.8%
NVIDIA L4 (610.43.02)	+18.3%	+8.2%	+9.9%	+2.8%
Intel Arc A310 (ANV)	+90.7%	+185.5%	+42.6%	+20.6%

(Arc sees almost triple speeds, whew)

So basically, H.264-only since adding other codecs would bloat this way too much, but I've verified with all 3 major GPU brands and different hardware configs. Decode API is now more codec-agnostic as well to make future codec-support work less painful.

@hgaiser - it's a lot, but I tried my best to test various cases and hardware 👍

[hgaiser]

Those results look promising :o

Regarding the new changes, they look much better, and closer to what the encode pipeline currently does. I have my questions about the Decoder::split functionality though. I guess it came from the access_units(..) function in my example above? It seems with this API like you must have the entire buffer if you want to be able to split it correctly. Live decoding wouldn't work then, right?

I think we might need to split the decoder in a sink and a source, since a packet from the network could, in theory, lead to N frames (where N >= 0). I'm thinking something like:

// Split is optional, but necessary for multi threaded applications.
let (mut sink, mut source) = Decoder::new(context, DecodeConfig::h264())?.split();

// Producer (optionally a thread): feed only, blocks only on backpressure
for chunk in network_or_file {
    sink.decode(chunk, pts)?;            // owns framing; buffers partial pictures
}
sink.finish()?;                          // EOF: drains reorder buffer, unblocks consumer

// Consumer (optionally a thread) : pull frames as they're ready
while let Some(frame) = source.next_frame().await? {
    render(&frame);                      // zero-copy GPU image
    // drop(frame) -> storage returned
}

There are currently three ways to get decoded frames (download, copy_frame_to_planes, OutputOrder::Decode). I'd like only the zero-copy method: a next_frame() that returns the decoder's own GPU image. In the zero-copy path the frame occupies a DPB slot until it's dropped, so the user is responsible for reading it (via download) and dropping it ASAP; this is what provides backpressure. That behavior is documented. copy_frame_to_planes and OutputOrder::Decode become redundant.

I would like decoder.decode(..) to only return an error when there is actually an error. Currently it can also return NeedsKeyframe which is not an error per se. I see two options: 1) don't return a "frame" if no key frame has been received yet (maybe with a decoder.needs_keyframe() -> bool method) or 2) make Decoder::decode return an enum (with variants Frames, Idle and NeedKeyFrame).

[DatCaptainHorse]

Those results look promising :o

Regarding the new changes, they look much better, and closer to what the encode pipeline currently does. I have my questions about the Decoder::split functionality though. I guess it came from the access_units(..) function in my example above? It seems with this API like you must have the entire buffer if you want to be able to split it correctly. Live decoding wouldn't work then, right?

I think we might need to split the decoder in a sink and a source, since a packet from the network could, in theory, lead to N frames (where N >= 0). I'm thinking something like:

// Split is optional, but necessary for multi threaded applications.
let (mut sink, mut source) = Decoder::new(context, DecodeConfig::h264())?.split();

// Producer (optionally a thread): feed only, blocks only on backpressure
for chunk in network_or_file {
    sink.decode(chunk, pts)?;            // owns framing; buffers partial pictures
}
sink.finish()?;                          // EOF: drains reorder buffer, unblocks consumer

// Consumer (optionally a thread) : pull frames as they're ready
while let Some(frame) = source.next_frame().await? {
    render(&frame);                      // zero-copy GPU image
    // drop(frame) -> storage returned
}

There are currently three ways to get decoded frames (download, copy_frame_to_planes, OutputOrder::Decode). I'd like only the zero-copy method: a next_frame() that returns the decoder's own GPU image. In the zero-copy path the frame occupies a DPB slot until it's dropped, so the user is responsible for reading it (via download) and dropping it ASAP; this is what provides backpressure. That behavior is documented. copy_frame_to_planes and OutputOrder::Decode become redundant.

I would like decoder.decode(..) to only return an error when there is actually an error. Currently it can also return NeedsKeyframe which is not an error per se. I see two options: 1) don't return a "frame" if no key frame has been received yet (maybe with a decoder.needs_keyframe() -> bool method) or 2) make Decoder::decode return an enum (with variants Frames, Idle and NeedKeyFrame).

Good to hear it's more to what you want 🙂

I didn't even think of that stream-split syntax, that'd actually work out super-well for decoding here, will get to that ASAP! As for the There are currently three ways to get decoded frames, yeah I thought convenience methods would be good, but then again they also add more "weight" on pixelforge which I reckon you want quite "minimal" ? It's simple to pull those out and I'll do that.

As for NeedsKeyframe, it's useful (atleast until intra-frame encoding feature is added) for when decoding fails and the keyframe-interval is set to "infinite", the consumer can easily request a new keyframe from producer (using some custom connection, akin to WebRTC "PLI" - Picture Loss Indication). Of course that's less useful with periodic keyframes.. with the split-stream approach I could add a "Result" value to sink.decode(chunk, pts)?;, which would then tell if anything is missing without a separate method-call, which is bit less.. janky?

Alas, good input, will get to work 👍

[DatCaptainHorse]

Ah looks like there's a new Rust version out already, that's why I don't get those clippy issues locally 😅

[hgaiser]

Ah looks like there's a new Rust version out already, that's why I don't get those clippy issues locally 😅

Yay, CI :)

[DatCaptainHorse]

Other than just making the changes requested, there was also room for improvements with the new stream syntax.

1 internal (in-GPU) copy is needed without VK_KHR_unified_image_layouts support.. well.. yes but also no 😅
It's basically a "guarantee" that GENERAL layout is usable without separate transitions, Intel doesn't advertise VK_KHR_unified_image_layouts but works with plain GENERAL layouts for decoding.. but that's ANV-specific so I'd say checking extension first is better approach than assuming 🤔

So, with VK_KHR_unified_image_layouts support (most modern GPUs from RDNA and GTX 16-series and above..) there's no copies anywhere, and without extension a single in-GPU one is done to make sure there's no issues from passing decoded images to user.

E: Also the decode ordering logic for frames (since B-frames are fun, getting in-between P and I frames in any order), is now done with direct slot juggling rather than holding separate copies. There may be more optimizations that are possible here, but I'd leave them for future PRs when support for other codec decoding is added?

E2: Hm.. since we now return single decoded image from pixelforge, any application needing to split it into "multiple per-planar resources" wouldn't have it easy.. gonna see how doable adding MUTABLE_FORMAT to the output flags would be 🤔

[DatCaptainHorse]

Alright, verified with Nestri on H.264 1080p60 stream (9060 XT encoding using pixelforge, A310 with debug-env vars enabled so pixelforge Vulkan decoding works on it's side)

I'm glad you pushed back on the earlier API @hgaiser - because holy heck I can't even begin to explain how good the latency feels now 😄

If you can try this PR in some experimental way with moonlight, give it a go 👀

[hgaiser]

The API looks good 👍

Was there anything you still wanted to do?

[DatCaptainHorse]

The API looks good 👍

Was there anything you still wanted to do?

Just the rebase against new upstream and a small extra variable that helps with tracking changes to decoding state (i.e. resolution changes in decoded input, without re-creating decoder), working on that now so won't take too long 🙂

After those are commited, feel free to merge when free 👍

E: Found a possible issue for downstream use, dropping decoded frame too early or holding onto it too long could get problematic so I'll defer the cleanup when no longer referenced, at worst it will add ~200 MB of memory usage on 4K decode if the consumer can't keep up, but it's better to be safe rather than have an use-after-free or similar issue later.

[DatCaptainHorse]

That should be all 🎉

[hgaiser]

Awesome 👍