Repository navigation
socket not available #70
Description
I'm not sure if it's a bug or if I'm just incompetent :)
I have a Quadlet for Podman rootless containers here. I'm trying to create a proxied socket for Podman for Dozzle, Uptime-Kuma, Homepage, etc. for my containers. All these are at a rootless proxy network. These containers should communicate via /run/user/1000/podman/podman-proxy.sock. Unfortunately, I can't get it to work. I'm not sure if I'll end up doing the whole thing via TCP later, but there's no need for that at the moment.
I'm running Fedora CoreOS with recent Podman, and hence I want to stick with SELinux.
BTW, am I right in thinking that Homepage only works via socket?
~/.config/containers/systemd/podman-socket-proxy/podman-socket-proxy.container:
[Unit]
Description=A lightweight and secure unix socket proxy for Podman
Documentation=https://github.com/wollomatic/socket-proxy
After=network-online.target
Requires=podman.socket
[Container]
Image=ghcr.io/wollomatic/socket-proxy:1
ContainerName=podman-socket-proxy
AutoUpdate=registry
Network=proxy.network
DropCapability=all
NoNewPrivileges=true
ReadOnly=true
EnvironmentFile=%h/.config/containers/systemd/podman-socket-proxy/podman-socket-proxy.env
SecurityLabelDisable=true
Volume=%t/podman/podman.sock:/var/run/docker.sock:ro,z
Volume=/etc/localtime:/etc/localtime:ro
[Service]
Restart=always
[Install]
WantedBy=default.target
~/.config/containers/systemd/podman-socket-proxy/podman-socket-proxy.env:
SP_ALLOWBINDMOUNTFROM=/run/user/1000/podman
#SP_ALLOW_GET=/v1\..{2}/(containers/.*|images/.*)
#SP_ALLOW_POST=/v1\..{2}/(containers/.*|images/.*|networks/.*)
#SP_ALLOW_DELETE=/v1\..{2}/(containers/.*|images/.*)
SP_LOGJSON=true
SP_LOGLEVEL=DEBUG
SP_ALLOW_GET=.*
SP_ALLOW_HEAD=.*
SP_ALLOW_POST=.*
SP_ALLOW_PUT=.*
SP_ALLOW_PATCH=.*
SP_ALLOW_DELETE=.*
SP_ALLOW_CONNECT=.*
SP_ALLOW_TRACE=.*
SP_ALLOW_OPTIONS=.*
#SP_ALLOWFROM=dazzle,homepage,uptime-kuma
#SP_LISTENIP=127.0.0.1
#SP_PROXYPORT=2375
SP_ALLOWHEALTHCHECK=yes # 127.0.0.1:55555/health
SP_PROXYSOCKETENDPOINT=yes
SP_PROXYSOCKETENDPOINTFILEMODE=0600
SP_SOCKETPATH=/run/user/1000/podman/podman-proxy.sock
SP_STOPONWATCHDOG=yes # halt program on error and let restart it
SP_WATCHDOGINTERVAL=10 # check for socket availability
SP_SHUTDOWNGRACETIME=10 # wait before shutting down
and here is my socket:
$ ls -laZ /run/user/1000/podman
total 0
drwxr-xr-x. 2 core core unconfined_u:object_r:user_tmp_t:s0 60 Oct 25 11:53 .
drwx------. 9 core core unconfined_u:object_r:user_tmp_t:s0 480 Oct 29 16:58 ..
srw-rw----. 1 core core system_u:object_r:container_file_t:s0 0 Oct 25 11:53 podman.sockFrom journal I get:
Oct 29 17:19:27 coreos-test podman-socket-proxy[1097782]: {"time":"2025-10-29T17:19:27.310131188+01:00","level":"INFO","msg":"configuration info","socketpath":"/run/user/1000/podman/podman-proxy.sock","proxysocketendpoint":"yes","proxysocketendpointfilemode":384,"loglevel":"DEBUG","logjson":true,"shutdowngracetime":10}
Oct 29 17:19:27 coreos-test podman-socket-proxy[1097782]: {"time":"2025-10-29T17:19:27.310311392+01:00","level":"INFO","msg":"proxysocketendpoint is set, so the TCP listener is deactivated"}
Oct 29 17:19:27 coreos-test podman-socket-proxy[1097782]: {"time":"2025-10-29T17:19:27.310331055+01:00","level":"INFO","msg":"watchdog disabled"}
Oct 29 17:19:27 coreos-test podman-socket-proxy[1097782]: {"time":"2025-10-29T17:19:27.31035028+01:00","level":"INFO","msg":"Docker bind mount restrictions enabled","allowbindmountfrom":["/run/user/1000/podman"]}
Oct 29 17:19:27 coreos-test podman-socket-proxy[1097782]: {"time":"2025-10-29T17:19:27.310431882+01:00","level":"INFO","msg":"configured allowed request","method":"GET","regex":"^.*$"}
Oct 29 17:19:27 coreos-test podman-socket-proxy[1097782]: {"time":"2025-10-29T17:19:27.310538609+01:00","level":"INFO","msg":"configured allowed request","method":"POST","regex":"^.*$"}
Oct 29 17:19:27 coreos-test podman-socket-proxy[1097782]: {"time":"2025-10-29T17:19:27.310563922+01:00","level":"INFO","msg":"configured allowed request","method":"PUT","regex":"^.*$"}
Oct 29 17:19:27 coreos-test podman-socket-proxy[1097782]: {"time":"2025-10-29T17:19:27.310588772+01:00","level":"INFO","msg":"configured allowed request","method":"PATCH","regex":"^.*$"}
Oct 29 17:19:27 coreos-test podman-socket-proxy[1097782]: {"time":"2025-10-29T17:19:27.31120126+01:00","level":"INFO","msg":"configured allowed request","method":"DELETE","regex":"^.*$"}
Oct 29 17:19:27 coreos-test podman-socket-proxy[1097782]: {"time":"2025-10-29T17:19:27.311352613+01:00","level":"INFO","msg":"configured allowed request","method":"CONNECT","regex":"^.*$"}
Oct 29 17:19:27 coreos-test podman-socket-proxy[1097782]: {"time":"2025-10-29T17:19:27.311480529+01:00","level":"INFO","msg":"configured allowed request","method":"TRACE","regex":"^.*$"}
Oct 29 17:19:27 coreos-test podman[930674]: @ - - [29/Oct/2025:17:19:27 +0100] "GET /v1.41/containers/6d66c7d955b2/json HTTP/1.1" 200 6120 "" "Docker-Client/Dozzle"
Oct 29 17:19:27 coreos-test podman-socket-proxy[1097782]: {"time":"2025-10-29T17:19:27.312698054+01:00","level":"INFO","msg":"configured allowed request","method":"OPTIONS","regex":"^.*$"}
Oct 29 17:19:27 coreos-test podman-socket-proxy[1097782]: {"time":"2025-10-29T17:19:27.312775843+01:00","level":"INFO","msg":"configured allowed request","method":"HEAD","regex":"^.*$"}
Oct 29 17:19:27 coreos-test podman-socket-proxy[1097782]: {"time":"2025-10-29T17:19:27.312805219+01:00","level":"DEBUG","msg":"checking socket availability","origin":"checkSocketAvailability"}
Oct 29 17:19:27 coreos-test podman-socket-proxy[1097782]: {"time":"2025-10-29T17:19:27.313186815+01:00","level":"ERROR","msg":"socket not available","error":"dial unix /run/user/1000/podman/podman-proxy.sock: connect: no such file or directory"}
Oct 29 17:19:27 coreos-test podman[930674]: 2025-10-29 17:19:27.416152459 +0100 CET m=+78481.122679349 container died 6d66c7d955b273a17c8e15029dfe2d6fd3a4a4311151744f5c94a0dd57888886 (image=ghcr.io/wollomatic/socket-proxy:1, name=podman-socket-proxy, org.opencontainers.image.source=https://github.com/wollomatic/socket-proxy, io.containers.autoupdate=registry, org.opencontainers.image.description=A lightweight and secure unix socket proxy, org.opencontainers.image.licenses=MIT)
Metadata
Metadata
Assignees
Labels
No labels
Activity
homelan-git commentedon Oct 29, 2025
Some test, meanwhile. It doesn't seem to be an SELinux related problem:
~$ sestatus SELinux status: enabled SELinuxfs mount: /sys/fs/selinux SELinux root directory: /etc/selinux Loaded policy name: targeted Current mode: permissive Mode from config file: enforcing Policy MLS status: enabled Policy deny_unknown status: allowed Memory protection checking: actual (secure) Max kernel policy version: 34and
wollomatic commentedon Oct 29, 2025
Hi @homelan-git, thanks for the detailed report!
Your socket-proxy configuration looks very good for testing.
There’s just one thing that might cause an issue: I’d recommend commenting out the
SP_ALLOWBINDMOUNTFROM=/run/user/1000/podmanline in your podman-socket-proxy.env. That’s a very specific feature that only affects container bind mount API calls — but I don’t think it’s related to the socket connection problem.We can see the actual issue here:
Looks like the socket-proxy can’t find the socket file.
I don’t have much personal experience with Podman, but it seems the bind mount name differs between your
podman-socket-proxy.containerfile and the socket-proxy configuration:I think you should set
SP_PROXYSOCKETENDPOINT=/run/user/1000/podman/podman-proxy.sockinstead of
SP_SOCKETPATH, becauseSP_SOCKETPATHdefines the source socket, which defaults to /var/run/docker.sock.homelan-git commentedon Oct 29, 2025
Hi @wollomatic, thank you for your fast answer. Unfortunately, your suggestions doesn't seem to fix the issue.
The container file is mainly the same, I didn't track it in git yet:
and journal's logged error is still the same ;(
Volume=%t/podman/podman.sock:/var/run/docker.sock:ro,zbind-mounts host's Podman socket into container, where is mimics the default docker socket/path. This works with other containers very well.SP_PROXYSOCKETENDPOINT=/run/user/1000/podman/podman-proxy.sockis the to be created filtered socket, isn't it? Hence,"socket not available","error":"dial unix /run/user/1000/podman/podman-proxy.sock: connect: no such file or directory"is confusing me.wollomatic commentedon Oct 29, 2025
Hi @homelan-git,
I can't reproduce it with my test system here (everything works as expected), but the log output in your first post is clear:
This shows that the configuration of the source socket (socketpath) is set to
/run/user/1000/podman/podman-proxy.sock.What happens if you explicitly set
SP_SOCKETPATH=/var/run/docker.sock?homelan-git commentedon Oct 29, 2025
Hi @wollomatic ,
still doesn't start: "ERROR","msg":"socket not available","error":"dial unix /var/run/docker.sock: connect: permission denied"
Is there a way to start a shell, or ls /var/run instead of starting the app?
wollomatic commentedon Oct 29, 2025
Hi @homelan-git,
finally, another error message. Seems the issue with the wrong bind mount is solved.
Now it seems the socket proxy can find the socket, but does not have the right to access it. Because of the security by default policy, socket-proxy runs as the most unprivileged user:group 65534:65534 ("nobody" in some linux distros). You need to set the rights to access the socket explicitly. With Docker, I'd recommend to set the gid to the Docker gid. With Podman, I would guess there is a similar way.
There's no shell in the socket-proxy container image, just the Entrypoint
/socket-proxy.You could build your own debug container image with a Containerfile / Dockerfile like this:
This should create an fedora image with the socket proxy executable inside, so you can run a shell.
homelan-git commentedon Oct 30, 2025
I'm slowly reaching the limits of my practical knowledge. I think I understand the concept of user mapping using Linux namespaces, at least...
Dockerfile, as suggested:
Commandline copy&paste from genberated unit, replaced some systemd specifier:
So, Podman's socket is available and functional inside the container. There is no path or mount to
/run/user/1000/podman/. Using additionally--uidmap=65534:$(id -u):1 --gidmap=65534:$(id -g):1also failed withError: container uses ID mappings ([]specs.LinuxIDMapping{specs.LinuxIDMapping{ContainerID:0xfffe, HostID:0x3e8, Size:0x1}}), but doesn't map UID 0Some minutes later ;-)
I changed at the
.envfileSP_PROXYSOCKETENDPOINT=/var/run/docker-proxy.socknow it seems to start:which gives me the socket
bash-5.3# ls -la /var/run/ total 0 drwxr-xr-x. 3 root root 120 Oct 30 17:09 . dr-xr-xr-x. 1 root root 17 Oct 30 17:06 .. -rw-r--r--. 1 root root 0 Oct 30 17:06 .containerenv srw-------. 1 root root 0 Oct 30 17:09 docker-proxy.sock srw-rw----. 1 root root 0 Oct 30 02:45 docker.sock drwxr-xr-x. 2 root root 40 Oct 30 17:06 secretsI would say, the binary isn't able to create the new socket path '/run/user/1000/podman/'. And, how to get this socket out of the container?