Repository navigation
Description
Summary
None of the three Publish Docker* workflows have succeeded since April. opendronemap/nodeodm:gpu has therefore not been rebuilt since 2025-10-22, while its own base image opendronemap/odm:gpu was rebuilt 2026-08-04. Users pulling the current tag get a ~10-month-old image, and on it GPU SIFT cannot work at all — silently, with no error in the task log.
CI status
| Workflow | File | Last success | Latest runs |
|---|---|---|---|
| Publish Docker | .github/workflows/publish-docker.yaml |
2026-03-25 | failure 2026-04-30, 2026-04-28 |
| Publish Docker GPU | .github/workflows/publish-docker-gpu.yaml |
(before 2026-03-13) | failure 2026-04-30, 04-28, 03-25, 03-17, 03-17, 03-13 |
| Publish Docker Intel GPU | .github/workflows/publish-docker-gpu-intel.yaml |
2026-03-17 | failure 2026-04-30, 2026-04-28 |
Two distinct failure modes on the GPU workflow:
- 2026-03-13 → 2026-03-25 — failed at step
Build and push Docker image(includes two manualworkflow_dispatchretries) - 2026-04-28, 2026-04-30 — failed at step
Login to DockerHub, which looks like expired/missing Docker Hub credentials
There have been no workflow runs at all since 2026-04-30.
- https://github.com/OpenDroneMap/NodeODM/actions/runs/25143350546 (GPU, 2026-04-30)
- https://github.com/OpenDroneMap/NodeODM/actions/runs/25031580143 (GPU, 2026-04-28)
- https://github.com/OpenDroneMap/NodeODM/actions/runs/25143350540 (CPU, 2026-04-30)
Docker Hub tag dates confirm the freeze:
| tag | last_updated |
|---|---|
opendronemap/odm:gpu (base image) |
2026-08-04 |
opendronemap/nodeodm:latest / master |
2026-03-25 |
opendronemap/nodeodm:gpu.intel |
2026-03-17 |
opendronemap/nodeodm:gpu |
2025-10-22 |
The last successful gpu publish was the hotfix for #252. Since then every attempt has failed. #174 raised a related rebuild-cadence problem.
Impact: GPU SIFT is silently disabled in the published image
In the current opendronemap/nodeodm:gpu (sha256:214fe6a4421fe5283648400e9ce455457bb866c87a20a325fddfd494e4ad482a), the CUDA 11 runtime is missing, so opensfm's pypopsift can never load:
$ ls -la /usr/local/cuda-11.8/targets/x86_64-linux/lib/ | grep cudart
lrwxrwxrwx 1 root root 17 Sep 21 2022 libcudart.so -> libcudart.so.11.0 # dangling
-rw-r--r-- 1 root root 1198880 Sep 21 2022 libcudart_static.a
$ ldd /code/SuperBuild/install/bin/opensfm/opensfm/pypopsift.cpython-38-x86_64-linux-gnu.so | grep cuda
libcudart.so.11.0 => not found
$ dpkg -l | grep -i cuda | awk '{print $2, $3}'
cuda-toolkit-11-8-config-common 11.8.89-1
cuda-toolkit-11-config-common 11.8.89-1
cuda-toolkit-config-common 12.3.52-1
libcufile-11-8 1.4.0.31-1
libcusolver-11-8 11.4.1.48-1
libnccl2 2.16.2-1+cuda11.8
# no cuda-cudart package at allResult, from inside the running container:
$ docker exec -w /code webodm-node-odm-1 python3 -c \
"from opendm.gpu import get_cuda_compute_version, has_popsift_and_can_handle_texsize as f; \
print(get_cuda_compute_version(0)); print(f(4032,3024))"
(8, 9)
[WARNING] libcudart.so.11.0: cannot open shared object file: No such file or directory
Falsehas_popsift_and_can_handle_texsize() swallows this as a warning, so opendm/osfm.py quietly falls back to CPU feature extraction. Nothing in the task log indicates a problem — the job simply runs several times slower. Searching the org for libcudart returns 0 issues, which I suspect is because the failure is invisible rather than because it is rare.
This appears to be collateral from the fix for the CUDA 11.2 conflict discussed in
https://community.opendronemap.org/t/opendronemap-nodeodm-gpu-nvidia-smi-detected-cannot-initialize-cuda/21124
— cuda-cudart-11-2 was removed and nothing installed cuda-cudart-11-8 in its place. A rebuild
on the current opendronemap/odm:gpu (CUDA 12.9 / Ubuntu 24.04) should resolve it, together with
ODM PR #2060 ("Restore DEPENDS opensfm for PyPopsift GPU module install order", merged 2026-08-04).
Minor, related
Dockerfile.gpu on master still contains:
RUN echo /usr/local/cuda-11.2/compat >> /etc/ld.so.conf.d/989_cuda-11.conf && ldconfig/usr/local/cuda-11.2/compat does not exist in the built image — it leaves a dangling entry in
/etc/ld.so.conf.d/. Probably obsolete now that the base is on CUDA 12.9.
Suggested actions
- Refresh the Docker Hub credentials used by the
Publish Docker*workflows. - Re-run
publish-docker-gpu.yamland confirm the MarchBuild and push Docker imagefailure is resolved by the neweropendronemap/odm:gpubase. - Consider a scheduled rebuild, or a trigger on base-image updates, so
nodeodm:gpucannot drift ~10 months behindodm:gpuagain. - Drop the
cuda-11.2/compatline fromDockerfile.gpu.
Environment
- Host: Ubuntu, NVIDIA RTX 4060 Ti 16 GB, driver 580.173.02 (CUDA 13.0), compute capability 8.9
- WebODM 3.0.1,
opendronemap/nodeodm:gpu@sha256:214fe6a4…(ODM 3.5.6) nvidia-container-toolkit1.19.1; GPU is correctly passed into the container (nvidia-smiworks inside, OpenMVS densification does use CUDA)
Activity
erickeller commentedon Aug 11, 2026
Dockerfile.gpu-nodeodm-fixed.txt
Three follow-ups since filing
1. The March
Build and push Docker imagefailuresDockerfile.gpuon master runs the dependency script before the source is copied into the image:Building from a clean checkout fails immediately:
That matches the
Build and push Docker imagestep failing on 2026-03-13, 03-17 (x2) and 03-25.The fix is to move the
mkdir/WORKDIR/useradd/COPYblock ahead of it:There is a second failure immediately after it:
node.shwrapsnodebut notnpm, and nvm isnot on
PATHin a fresh layer, so the following bareRUN npm install --productionalso exits127. Sourcing nvm explicitly fixes it:
2. With those two changes the GPU image builds and PopSift works
docker build -f Dockerfile.gpu -t nodeodm:gpu-local .then succeeds against the currentopendronemap/odm:gpubase (ODM 3.6.1, CUDA 12.9), tested on an RTX 5070 Ti (sm_120):True, againstFalse+libcudart.so.11.0: cannot open shared object fileon the publishedgpuimage. So a rebuild on the current base resolves the silent CPU fallback in the originalreport by itself — no separate
cuda-cudartpatch needed, because the CUDA 12.9 base ships areal
libcudart.so.12.While there: the
RUN echo /usr/local/cuda-11.2/compat >> /etc/ld.so.conf.d/989_cuda-11.confline can be dropped. That directory does not exist in the current base and only leaves a dangling
entry in
/etc/ld.so.conf.d/.3. The April failures look like credentials, not code
The two most recent runs failed at the
Login to DockerHubstep (2026-04-28,2026-04-30) rather than at
build. Both postdate the WebODM fork announced on 6 April 2026
(webodm.org/blog/announcement), which stated that "If
you use ODM or NodeODM, you can find updated forks at ODX and NodeODX, for which stable docker
images will continue to receive updates and improvements at webodm/odx and webodm/nodeodx."
Given that timing, it seems worth checking whether the
DOCKERHUB_*secret in this repo stillbelongs to an account holding push rights to
opendronemap/nodeodm. I have no visibility intothe org's credentials — this is only an inference from the failure step and the dates.
4. For reference, the fork already carries both fixes
WebODM/NodeODX/Dockerfile.gpu(forked from this repo per the announcement above) already has
COPY . /var/wwwahead ofinstall_deps.shand has removed thecuda-11.2/compatline, so the changes in §1 can be portedrather than written from scratch.
Correction to my original report
I implied the
gputag had simply been neglected. Having since learned of the April 2026 fork, Iwant to be explicit that I am not reporting this as a deprecation:
opendronemap/nodeodm:gpuis still what
--gpus-enabled installs on this stack pull today, and on it GPU feature extractioncannot work at all. That is the whole of my report — I have no view on the fork itself and am
raising this purely as a packaging/CI issue.
maurerle commentedon Aug 19, 2026
this issue is fixed by #272
maurerle commentedon Aug 21, 2026
This issue should still be open - as the pr which fixes it is not merged yet.
Drop the CUDA 11.2 compat ldconfig entry