Repository navigation
Native decoding on Raspberry Pi 4 #129
Description
Activity
I was wondering if the 1.7 build has had any advancements on the stability of playing back with hardware support. The Raspberry page hasn't been updated with any tests there. Is it assumed now that it will work better, or possible no issues on the Pi 4 because it has more power?
Has anyone had any success testing on the Pi4?
S
Hello,
the 1.7 doesn't contain any pi4 improvements as far as I know. However, I've actually been working on the Pi4 hw. decoding / hw. presenting. It mostly already works in my development branch, and hopefully I'll be able to merge it into the master branch by the end of the year.Thats awesome. When you say mostly. What's missing?
I pushed the rpi4 acceleration support to the master branch. It's also now available in the latest arm AppImage build. Note that only video with 4:2:0 subsampling is supported and there are a few limitations like not being able to resize or move the video output once it is on the screen.
To use it launch ultragrid like this
uv -d rpi4 --param use-hw-accel(there is also short documentation in the wiki.Reacted by DeeJayMXReacted by DeeJayMXReacted by DeeJayMXThats awesome. Looking forward to test that out.
Its common that only 420 is hardware accelerated so I am sure it will work just fine.
AWESOME !
Sorry for the dumb question, but when I try to decode I get the following error. I'm using 420 h.264, do I have to use the omx h264 to be able to work?
Unable to find decoder for input codec "H.264"!!!
It looks like this acceleration method supports only hevc (H.265).
For H.264 you can try running the parameter--param force-lavd-decoder=h264_mmalwith the gl or sdl displays, but this does not work very well right now.When trying to run Ultragrid on a Raspberry pi4 I am receiving the error
[lavd] Selected pixel format: rpi [lavc hevc @ 0xeb36e3b0] can't open /dev/rpivid-hevcmem [lavc hevc @ 0xeb36e3b0] Failed to open rpivid devices [lavc hevc @ 0xeb36e3b0] Failed setup for format rpi: hwaccel initialisation returned error.I am using the nightly build if UltraGrid and have tried on both the desktop and console version of Raspbian.
root@raspberrypi:/home/sogorman# ./UltraGrid-latest-armhf.AppImage -d rpi4 --param use-hw-accel UltraGrid 1.8+ (master rev e3026d3 built Sep 12 2023 04:15:58) Display device : rpi4 Capture device : none Audio capture : none Audio playback : none MTU : 9000 B Video compression: none Audio codec : PCM Network protocol : UltraGrid RTP Audio FEC : none Video FEC : none Control socket listening on port 43557 [lavd] Setting thread count to 0, type: slice [lavd] Using decoder: hevc [lavc hevc @ 0xeb304cb0] can't open /dev/rpivid-hevcmem [lavc hevc @ 0xeb304cb0] Failed to open rpivid devices [lavc hevc @ 0xeb304cb0] Failed setup for format rpi: hwaccel initialisation returned error. [lavc hevc @ 0xeb304cb0] A hardware frames or device context is required for hardware accelerated decoding. [lavc hevc @ 0xeb304cb0] Failed setup for format vdpau: hwaccel initialisation returned error. [video dec.] Detected compression properties: YCbCr 4:2:0 8 bit Last message repeated 4 times [display] Successfully reconfigured display to 1920x1080 @30.00p, codec RPI4_8 [lavd] Setting thread count to 0, type: slice [lavd] Using decoder: hevc [lavd] Waiting for first SPS/VPS NALU... Last message repeated 8 times [lavd] Selected pixel format: rpi [lavc hevc @ 0xeb36e3b0] can't open /dev/rpivid-hevcmem [lavc hevc @ 0xeb36e3b0] Failed to open rpivid devices [lavc hevc @ 0xeb36e3b0] Failed setup for format rpi: hwaccel initialisation returned error. [lavc AVHWDeviceContext @ 0xec7fa9c0] libva: vaGetDriverNameByIndex() failed with unknown libva error, driver_name = (null) Failed to initialise VAAPI connection: -1 (unknown libva error). [lavc AVHWDeviceContext @ 0xec7489c0] libva: vaGetDriverNameByIndex() failed with unknown libva error, driver_name = (null) Failed to initialise VAAPI connection: -1 (unknown libva error). [hw accel] Unable to create hwdevice!! [lavc AVHWDeviceContext @ 0xec7fab30] Cannot open the X11 display . [lavc AVHWDeviceContext @ 0xec73f850] Cannot open the X11 display /dev/dri/renderD128. [hw accel] Unable to create hwdevice!! [lavd] Falling back to software decoding! [lavd] Selected pixel format: yuv420p [lavc hevc @ 0xeb36e3b0] Could not find ref with POC 19 [lavc hevc @ 0xeb36e3b0] Could not find ref with POC 18 [lavd] Using swscale to convert from yuv420p to rpi. [lavc swscaler @ 0xec75f9f0] 0bpp not supported by yuv2rgb [lavc swscaler @ 0xec75f9f0] rpi is not supported as output pixel format [lavd] Unable to init sws context. [lavd] Using swscale to convert from yuv420p to rpi. [lavc swscaler @ 0xec75f9f0] 0bpp not supported by yuv2rgb [lavc swscaler @ 0xec75f9f0] rpi is not supported as output pixel format [lavd] Unable to init sws context. [lavd] Using swscale to convert from yuv420p to rpi. [lavc swscaler @ 0xec75f9f0] 0bpp not supported by yuv2rgb [lavc swscaler @ 0xec75f9f0] rpi is not supported as output pixel format [lavd] Unable to init sws context. [lavd] Using swscale to convert from yuv420p to rpi. [lavc swscaler @ 0xec764210] 0bpp not supported by yuv2rgb [lavc swscaler @ 0xec764210] rpi is not supported as output pixel format [lavd] Unable to init sws context. [RPi display] 1 frames in 6.90896 seconds = 0.14474 FPS [lavd] Setting thread count to 0, type: slice [lavd] Using decoder: hevc Unable to find decoder for input codec "H.265"!!! Compression internal codec is "(undefined)". Native codecs are: (none) Could not find neither line conversion nor decompress from H.265 to display supported formats ((none)). [video dec.] Reconfiguration failed!!! [lavd] Setting thread count to 0, type: slice [lavd] Using decoder: hevc Unable to find decoder for input codec "H.265"!!!The missing "/dev/rpivid-hevcmem" was resolved with rolling back to an older Raspberry pi firmware.
Have been able to get video to decode on RPI4 and it's rock solid. That being said have been challanged with getting HDMI audio to pass. Getting ALSA errors on the HDMI port audio.
root@WOPR-110:/home/admin# ./UltraGrid-latest-armhf.AppImage -d rpi4 -r portaudio --param use-hw-accel UltraGrid 1.8+ (master rev e3026d3 built Sep 12 2023 04:15:58) Display device : rpi4 Capture device : none Audio capture : none Audio playback : portaudio MTU : 9000 B Video compression: none Audio codec : PCM Network protocol : UltraGrid RTP Audio FEC : none Video FEC : none Expression 'alsa_snd_pcm_hw_params_set_period_size_near( pcm, hwParams, &alsaPeriodFrames, &dir )' failed in 'src/hostapi/alsa/pa_linux_alsa.c', line: 924 Expression 'alsa_snd_pcm_hw_params_set_period_size_near( pcm, hwParams, &alsaPeriodFrames, &dir )' failed in 'src/hostapi/alsa/pa_linux_alsa.c', line: 924 ALSA lib pcm.c:2660:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.front ALSA lib pcm.c:2660:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.rear ALSA lib pcm.c:2660:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.center_lfe ALSA lib pcm.c:2660:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.side ALSA lib pcm.c:2660:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.surround21 ALSA lib pcm.c:2660:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.surround21 ALSA lib pcm.c:2660:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.surround40 ALSA lib pcm.c:2660:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.surround41 ALSA lib pcm.c:2660:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.surround50 ALSA lib pcm.c:2660:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.surround51 ALSA lib pcm.c:2660:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.surround71 ALSA lib pcm.c:2660:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.iec958 ALSA lib pcm.c:2660:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.iec958 ALSA lib pcm.c:2660:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.iec958 ALSA lib pcm.c:2660:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.hdmi ALSA lib pcm.c:2660:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.hdmi ALSA lib pcm.c:2660:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.modem ALSA lib pcm.c:2660:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.modem ALSA lib pcm.c:2660:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.phoneline ALSA lib pcm.c:2660:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.phoneline Expression 'alsa_snd_pcm_hw_params_set_period_size_near( pcm, hwParams, &alsaPeriodFrames, &dir )' failed in 'src/hostapi/alsa/pa_linux_alsa.c', line: 924 Cannot connect to server socket err = No such file or directory Cannot connect to server request channel jack server is not running or cannot be started JackShmReadWritePtr::~JackShmReadWritePtr - Init not done for -1, skipping unlock JackShmReadWritePtr::~JackShmReadWritePtr - Init not done for -1, skipping unlock [PortAudio] Using PortAudio V19.6.0-devel, revision 396fe4b6699ae929d3a685b3ef8a7e97396139a4 [Portaudio playback] (Re)initializing portaudio playback. Using PortAudio version: PortAudio V19.6.0-devel, revision 396fe4b6699ae929d3a685b3ef8a7e97396139a4 Using default output audio device: bcm2835 ALSA: IEC958/HDMI (hw:0,1) Audio receiving started. Control socket listening on port 39715 New incoming audio format detected: 48000 Hz, 2 channels, 32 bits per sample, codec MP3 [lavd] Setting thread count to 0, type: slice [lavd] Using decoder: hevc [lavd] Waiting for first SPS/VPS NALU... Last message repeated 1 times [lavc hevc @ 0xa880c6f0] RPI HEVC h/w accel init OK [video dec.] Detected compression properties: (undefined) Last message repeated 4 times [display] Successfully reconfigured display to 1920x1080 @29.97p, codec RPI4_8 [lavd] Setting thread count to 0, type: slice [lavd] Using decoder: hevc [lavd] Waiting for first SPS/VPS NALU... Last message repeated 9 times [lavd] Selected pixel format: rpi [lavc hevc @ 0xa8813450] RPI HEVC h/w accel init OK [from_lavc_vid_conv] conversion is reducing subsampling from 4440 to 4200 Expression 'ret' failed in 'src/hostapi/alsa/pa_linux_alsa.c', line: 1736 Expression 'AlsaOpen( hostApi, parameters, streamDir, &pcm )' failed in 'src/hostapi/alsa/pa_linux_alsa.c', line: 1768 SSRC 0x71787918: 128/128 packets received (100.0000%), 0 lost, max loss 0 Expression 'ret' failed in 'src/hostapi/alsa/pa_linux_alsa.c', line: 1736 Expression 'AlsaOpen( hostApi, parameters, streamDir, &pcm )' failed in 'src/hostapi/alsa/pa_linux_alsa.c', line: 1768Did you try alsa or pulse?
speaker-test does produce audio on the HDMI output without error
admin@WOPR-110:~ $ sudo speaker-test -c2 -twav -l7 speaker-test 1.2.4 Playback device is default Stream parameters are 48000Hz, S16_LE, 2 channels WAV file(s) Rate set to 48000Hz (requested 48000Hz) Buffer size range from 480 to 32768 Period size range from 480 to 32768 Using max buffer size 32768 Periods = 4 was set period_size = 8192 was set buffer_size = 32768 0 - Front Left 1 - Front Right Time per period = 2.408387 0 - Front Left ^CTransfer failed: Bad addressStill playing around with this but it looks like in recent versions of raspberry pi firmware they have moved /dev/rpivid-hevcmem to /dev/video19. Is there a way to specify a different rpivid device
I looked into it and it appears that it's not as simple as changing the path. The whole API changed. We might look into supporting the new API sometime in the future, but I can't promise anything right now.
Also, for the sake of documentation, could you please share which version of the firmware works for you with the current implementation?
- added a commit that references this issue
on Sep 25, 2023 I have been having good success with UG on RPi4 with this firmware / raspbian release.
admin@WOPR-RX-111:/dev $ vcgencmd version Mar 17 2023 10:50:39 Copyright (c) 2012 Broadcom version 82f3750a65fadae9a38077e3c2e217ad158c8d54 (clean) (release) (start) admin@WOPR-RX-111:/dev $ cat /etc/os-release PRETTY_NAME="Raspbian GNU/Linux 10 (buster)" NAME="Raspbian GNU/Linux" VERSION_ID="10" VERSION="10 (buster)" VERSION_CODENAME=buster ID=raspbian ID_LIKE=debian HOME_URL="http://www.raspbian.org/" SUPPORT_URL="http://www.raspbian.org/RaspbianForums" BUG_REPORT_URL="http://www.raspbian.org/RaspbianBugs" admin@WOPR-RX-111:/dev $Just a heads up, I was looking into getting this to work again and I was able to implement partial support. Decoding HEVC works, but only in copy mode right now, so the performance is not completely there yet for perfectly smooth playback.
As a part of this I switched the armhf continuous builds over to raspbian bullseye and removed the old hwaccel implementation, since there is no point in maintaining it if it no longer works. As such the 1.9 stable branch will be the last one to support rpi hwaccel on raspbian buster.
Reacted by sogormanI am going to grab a pi5 and can play around with UG on a fresh bullseye install (debian 11).
@MartinPulec I cloned the rpi_bookworm branch and built it on a rpi5 clean os install.
knct@raspberrypi:~/rpi_bookworm/UltraGrid/bin $ uname -a Linux raspberrypi 6.6.20+rpt-rpi-2712 #1 SMP PREEMPT Debian 1:6.6.20-1+rpt1 (2024-03-07) aarch64 GNU/LinuxHere are a list of features that were discovered during the build process
UltraGrid configuration summary =============================== Target: aarch64-unknown-linux-gnu Features: Crypto .......................... no CUDA support .................... no Debug output .................... no iHDTV support ................... no IPv6 support .................... yes Library live555 ................. no Manual pages .................... no OpenCV .......................... no Profiling support ............... no Qt GUI .......................... no RT priority ..................... no Soxr ............................ no SpeexDSP ........................ no Standalone modules .............. no zfec ............................ no Audio: ALSA ............................ no CoreAudio ....................... no JACK ............................ no JACK transport .................. no SDL_mixer ....................... no Pipewire ........................ no Portaudio ....................... no WASAPI .......................... no Video: AJA ............................. no AV Foundation ................... no BitFlow ......................... no Bluefish444 ..................... no caca ............................ no DeckLink ........................ yes DELTACAST ....................... no DirectShow ...................... no DVS ............................. no File ............................ no Gpustitch ....................... no NDI ............................. no OpenGL .......................... no OpenXR VR Display ............... no Panorama Gl Display ............. no RTSP capture client ............. no SAGE ............................ no Screen capture (X11) ............ yes SDL ............................. no SW video mix .................... no V4L2 ............................ yes VULKAN_SDL2 ..................... no XIMEA ........................... no Pipewire ........................ no Compressions: Cineform ........................ no Comprimato J2K .................. no CUDA DXT ........................ no GPUJPEG ......................... no GPUJPEG transcode to DXT ........ no Lavc (VDP no, VA no) ............ no Realtime DXT .................... no UYVY dummy compression .......... no Others: Blank capture filter ............ no GPU accelerated LDGM ............ no Hole punching ................... no iHDTV support ................... no MCU-like video mixer ............ no NAT-PMP traversal ............... no PCP NAT traversal ............... no Resize capture filter ........... no RTSP server ..................... no Scale postprocessor ............. no SDP over HTTP ................... no Spout ........................... no Swscale support ................. no Syphon .......................... no Testcard extras ................. yes Text postprocess and filter ..... noknct@raspberrypi:~/rpi_bookworm/UltraGrid/bin $ ./uv -d fullhelp UltraGrid 1.9+ (master rev 0557730fc built May 16 2024 03:06:22) Available display devices: aggregate decklink dummy dump multiplier none pipe preview unix_sock v4l2Does it make sense to investigate the rpi5 as it it is more full featured and affords vulkan sdl2 support out of the box?
The master branch now contains a special
drmdisplay module, which can display HW decoded frames without any copying and should offer the best possible performance.That said, it is also quite inflexible. It needs exclusive access to the connected display, which means that you can't have any other display server running (no X11, Wayland, desktop environment, etc.). At the moment it also doesn't support any scaling, so the connected display device needs to support a video mode that is appropriate for the decoded video (the decoded video can be larger than the display resolution, in which case it is cropped, but it cannot be smaller). Scaling could be probably implemented using overlay planes in the future, but right now I don't know if the Raspberry hardware supports this or if it would be performant enough.
Here are a list of features that were discovered during the build process
...You need to install the appropriate development packages for the features that you want. See this section of the wiki for more info.
Does it make sense to investigate the rpi5 as it it is more full featured and affords vulkan sdl2 support out of the box?
We currently do not have any rpi5 on hand to test or develop on. That said, the current implementation of the acceleration probably could work on the rpi5 as well and we would be happy if you let us know if that's indeed the case.
Also this issue is quite old, grew quite long and even branched out into unrelated topics. Now that the display module is merged it seems like a good time to close this. For any bugs, feature requests, etc. please create new issues or discussions.
- added a commit that references this issue
on May 1, 2026
The Raspberry Pi 4 has hardware HEVC decoding so it could make a very nice display device for UltraGrid. With VLC and ffmpeg I'm able to get smooth playback of 4K 10-bit (limited to 4:2:0) with about 30% CPU usage.
From what I've read, the HEVC decoder uses a custom pixel format known as "sand". Converting this to a standard pixel format is too slow for realtime, so I think the only option is to pass the opaque sand buffers to the HDMI output which knows how to display it.
Example ffmpeg command (built from https://github.com/popcornmix/FFmpeg/tree/kodi_v4l2_4_2_2):