Skip to content

Docker Hub publishing has been failing since March — nodeodm:gpu frozen at 2025-10-22, GPU SIFT silently broken #271

Closed

Description

@erickeller

Summary

None of the three Publish Docker* workflows have succeeded since April. opendronemap/nodeodm:gpu has therefore not been rebuilt since 2025-10-22, while its own base image opendronemap/odm:gpu was rebuilt 2026-08-04. Users pulling the current tag get a ~10-month-old image, and on it GPU SIFT cannot work at all — silently, with no error in the task log.

CI status

Workflow File Last success Latest runs
Publish Docker .github/workflows/publish-docker.yaml 2026-03-25 failure 2026-04-30, 2026-04-28
Publish Docker GPU .github/workflows/publish-docker-gpu.yaml (before 2026-03-13) failure 2026-04-30, 04-28, 03-25, 03-17, 03-17, 03-13
Publish Docker Intel GPU .github/workflows/publish-docker-gpu-intel.yaml 2026-03-17 failure 2026-04-30, 2026-04-28

Two distinct failure modes on the GPU workflow:

  • 2026-03-13 → 2026-03-25 — failed at step Build and push Docker image (includes two manual workflow_dispatch retries)
  • 2026-04-28, 2026-04-30 — failed at step Login to DockerHub, which looks like expired/missing Docker Hub credentials

There have been no workflow runs at all since 2026-04-30.

Docker Hub tag dates confirm the freeze:

tag last_updated
opendronemap/odm:gpu (base image) 2026-08-04
opendronemap/nodeodm:latest / master 2026-03-25
opendronemap/nodeodm:gpu.intel 2026-03-17
opendronemap/nodeodm:gpu 2025-10-22

The last successful gpu publish was the hotfix for #252. Since then every attempt has failed. #174 raised a related rebuild-cadence problem.

Impact: GPU SIFT is silently disabled in the published image

In the current opendronemap/nodeodm:gpu (sha256:214fe6a4421fe5283648400e9ce455457bb866c87a20a325fddfd494e4ad482a), the CUDA 11 runtime is missing, so opensfm's pypopsift can never load:

$ ls -la /usr/local/cuda-11.8/targets/x86_64-linux/lib/ | grep cudart
lrwxrwxrwx 1 root root      17 Sep 21  2022 libcudart.so -> libcudart.so.11.0   # dangling
-rw-r--r-- 1 root root 1198880 Sep 21  2022 libcudart_static.a

$ ldd /code/SuperBuild/install/bin/opensfm/opensfm/pypopsift.cpython-38-x86_64-linux-gnu.so | grep cuda
        libcudart.so.11.0 => not found

$ dpkg -l | grep -i cuda | awk '{print $2, $3}'
cuda-toolkit-11-8-config-common 11.8.89-1
cuda-toolkit-11-config-common 11.8.89-1
cuda-toolkit-config-common 12.3.52-1
libcufile-11-8 1.4.0.31-1
libcusolver-11-8 11.4.1.48-1
libnccl2 2.16.2-1+cuda11.8
# no cuda-cudart package at all

Result, from inside the running container:

$ docker exec -w /code webodm-node-odm-1 python3 -c \
    "from opendm.gpu import get_cuda_compute_version, has_popsift_and_can_handle_texsize as f; \
     print(get_cuda_compute_version(0)); print(f(4032,3024))"
(8, 9)
[WARNING] libcudart.so.11.0: cannot open shared object file: No such file or directory
False

has_popsift_and_can_handle_texsize() swallows this as a warning, so opendm/osfm.py quietly falls back to CPU feature extraction. Nothing in the task log indicates a problem — the job simply runs several times slower. Searching the org for libcudart returns 0 issues, which I suspect is because the failure is invisible rather than because it is rare.

This appears to be collateral from the fix for the CUDA 11.2 conflict discussed in
https://community.opendronemap.org/t/opendronemap-nodeodm-gpu-nvidia-smi-detected-cannot-initialize-cuda/21124
— cuda-cudart-11-2 was removed and nothing installed cuda-cudart-11-8 in its place. A rebuild
on the current opendronemap/odm:gpu (CUDA 12.9 / Ubuntu 24.04) should resolve it, together with
ODM PR #2060 ("Restore DEPENDS opensfm for PyPopsift GPU module install order", merged 2026-08-04).

Minor, related

Dockerfile.gpu on master still contains:

RUN echo /usr/local/cuda-11.2/compat >> /etc/ld.so.conf.d/989_cuda-11.conf && ldconfig

/usr/local/cuda-11.2/compat does not exist in the built image — it leaves a dangling entry in
/etc/ld.so.conf.d/. Probably obsolete now that the base is on CUDA 12.9.

Suggested actions

  1. Refresh the Docker Hub credentials used by the Publish Docker* workflows.
  2. Re-run publish-docker-gpu.yaml and confirm the March Build and push Docker image failure is resolved by the newer opendronemap/odm:gpu base.
  3. Consider a scheduled rebuild, or a trigger on base-image updates, so nodeodm:gpu cannot drift ~10 months behind odm:gpu again.
  4. Drop the cuda-11.2/compat line from Dockerfile.gpu.

Environment

  • Host: Ubuntu, NVIDIA RTX 4060 Ti 16 GB, driver 580.173.02 (CUDA 13.0), compute capability 8.9
  • WebODM 3.0.1, opendronemap/nodeodm:gpu @ sha256:214fe6a4… (ODM 3.5.6)
  • nvidia-container-toolkit 1.19.1; GPU is correctly passed into the container (nvidia-smi works inside, OpenMVS densification does use CUDA)

Activity

  1. erickeller commented on Aug 11, 2026

    @erickeller
    Author

    Dockerfile.gpu-nodeodm-fixed.txt

    Three follow-ups since filing

    1. The March Build and push Docker image failures

    Dockerfile.gpu on master runs the dependency script before the source is copied into the image:

    RUN bash install_deps.sh && \        # <- runs here
        ln -s /code/SuperBuild/install/bin/untwine /usr/bin/untwine && \
        ...
    RUN mkdir /var/www
    WORKDIR "/var/www"
    RUN useradd -m -d "/home/odm" -s /bin/bash odm
    COPY --chown=odm:odm . /var/www      # <- but the file only arrives here

    Building from a clean checkout fails immediately:

    #6 0.176 bash: install_deps.sh: No such file or directory
    ERROR: failed to solve: process "/bin/sh -c bash install_deps.sh && ..."
           did not complete successfully: exit code: 127
    

    That matches the Build and push Docker image step failing on 2026-03-13, 03-17 (x2) and 03-25.

    The fix is to move the mkdir / WORKDIR / useradd / COPY block ahead of it:

    -RUN bash install_deps.sh && \
    -    ln -s /code/SuperBuild/install/bin/untwine /usr/bin/untwine && \
    -    ln -s /code/SuperBuild/install/bin/entwine /usr/bin/entwine && \
    -    ln -s /code/SuperBuild/install/bin/pdal /usr/bin/pdal && \
    -    ln -s /var/www/node.sh /usr/bin/node
    -
     RUN mkdir /var/www
     WORKDIR "/var/www"
     RUN useradd -m -d "/home/odm" -s /bin/bash odm
     COPY --chown=odm:odm . /var/www
    +
    +RUN bash install_deps.sh && \
    +    ln -s /code/SuperBuild/install/bin/untwine /usr/bin/untwine && \
    +    ln -s /code/SuperBuild/install/bin/entwine /usr/bin/entwine && \
    +    ln -s /code/SuperBuild/install/bin/pdal /usr/bin/pdal && \
    +    ln -s /var/www/node.sh /usr/bin/node

    There is a second failure immediately after it: node.sh wraps node but not npm, and nvm is
    not on PATH in a fresh layer, so the following bare RUN npm install --production also exits
    127. Sourcing nvm explicitly fixes it:

    RUN bash -c 'source $NVM_DIR/nvm.sh && nvm use $NODE_VERSION && \
                 npm install --production && mkdir -p tmp && node index.js --powercycle'

    2. With those two changes the GPU image builds and PopSift works

    docker build -f Dockerfile.gpu -t nodeodm:gpu-local . then succeeds against the current
    opendronemap/odm:gpu base (ODM 3.6.1, CUDA 12.9), tested on an RTX 5070 Ti (sm_120):

    $ docker run --rm --gpus all -w /code --entrypoint python3 nodeodm:gpu-local -c \
        "from opendm.gpu import has_popsift_and_can_handle_texsize as f; print(f(4032,3024))"
    [INFO]    popsift can handle texture size 4032x3024
    True

    True, against False + libcudart.so.11.0: cannot open shared object file on the published
    gpu image. So a rebuild on the current base resolves the silent CPU fallback in the original
    report by itself — no separate cuda-cudart patch needed, because the CUDA 12.9 base ships a
    real libcudart.so.12.

    While there: the RUN echo /usr/local/cuda-11.2/compat >> /etc/ld.so.conf.d/989_cuda-11.conf
    line can be dropped. That directory does not exist in the current base and only leaves a dangling
    entry in /etc/ld.so.conf.d/.

    3. The April failures look like credentials, not code

    The two most recent runs failed at the Login to DockerHub step (2026-04-28,
    2026-04-30) rather than at
    build. Both postdate the WebODM fork announced on 6 April 2026
    (webodm.org/blog/announcement), which stated that "If
    you use ODM or NodeODM, you can find updated forks at ODX and NodeODX, for which stable docker
    images will continue to receive updates and improvements at webodm/odx and webodm/nodeodx."

    Given that timing, it seems worth checking whether the DOCKERHUB_* secret in this repo still
    belongs to an account holding push rights to opendronemap/nodeodm. I have no visibility into
    the org's credentials — this is only an inference from the failure step and the dates.

    4. For reference, the fork already carries both fixes

    WebODM/NodeODX/Dockerfile.gpu
    (forked from this repo per the announcement above) already has COPY . /var/www ahead of
    install_deps.sh and has removed the cuda-11.2/compat line, so the changes in §1 can be ported
    rather than written from scratch.

    Correction to my original report

    I implied the gpu tag had simply been neglected. Having since learned of the April 2026 fork, I
    want to be explicit that I am not reporting this as a deprecation: opendronemap/nodeodm:gpu
    is still what --gpus-enabled installs on this stack pull today, and on it GPU feature extraction
    cannot work at all. That is the whole of my report — I have no view on the fork itself and am
    raising this purely as a packaging/CI issue.

  2. maurerle commented on Aug 19, 2026

    @maurerle

    this issue is fixed by #272

  3. maurerle commented on Aug 21, 2026

    @maurerle

    This issue should still be open - as the pr which fixes it is not merged yet.

  4. linked a pull request that will close this issuePrep for release 3.6.2 #273on Sep 16, 2026
  5. added a commit that references this issue on Sep 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions