Welcome to Incremental Social! Learn more about this project here!
Check out lemmyverse to find more communities to join from here!

[SOLVED] Podman won't start Pihole with an error saying that it can't bind to port 53, as it is already in use, but nothing is using port 53.

Solution

It was found (here, and here) that Podman uses its own DNS server, aardvark-dns which is bound to port 53 (this explains why I was able to bind to 53 with nc on the host while the container would still fail). So the solution is to bridge the network for that port. So, in the compose file, the ports section would become:

ports:
  - "<host-ip>:53:53/tcp"
  - "<host-ip>:53:53/udp"
  - "80:80/tcp"

where <host-ip> is the ip of the machine running the container — e.g. 192.168.1.141.


Original Post

I so desperately want to bash my head into a hard surface. I cannot figure out what is causing this issue. The full error is as follows:

Error: cannot listen on the UDP port: listen udp4 :53: bind: address already in use

This is my compose file:

version: "3"
services:
  pihole:
    container_name: pihole
    image: docker.io/pihole/pihole:latest
    ports:
      - "53:53/tcp"
      - "53:53/udp"
      - "80:80/tcp"
    environment:
      TZ: '<redacted>'
    volumes:
      - './etc-pihole:/etc/pihole'
      - './etc-dnsmasq.d:/etc/dnsmasq.d'
    restart: unless-stopped

and the result of # ss -tulpn:

Netid       State        Recv-Q       Send-Q                             Local Address:Port               Peer Address:Port       Process                                         
udp         UNCONN       0            0                    [fe80::e877:8420:5869:dbd9]:546                           *:*           users:(("NetworkManager",pid=377,fd=28))       
tcp         LISTEN       0            128                                      0.0.0.0:22                      0.0.0.0:*           users:(("sshd",pid=429,fd=3))                  
tcp         LISTEN       0            128                                         [::]:22                         [::]:*           users:(("sshd",pid=429,fd=4))        

I have looked for possible culprit services like systemd-resolved. I have tried disabling Avahi. I have looked for other potential DNS services. I have rebooted the device. I am running the container as sudo (so it has access to all ports). I am quite at a loss.

  • Raspberry Pi Model 1 B Rev 2
  • Raspbian (bookworm)
  • Kernel v6.6.20+rpt-rpi-v6
  • Podman v4.3.1
  • Podman Compose v1.0.3

EDIT (2024-03-14T22:13Z)

For the sake of clarity, # netstat -pna | grep 53 shows nothing on 53, and # lsof -i -P -n | grep LISTEN shows nothing listening to port 53 — the only listening service is SSH on 22, as expected.

Also, as suggested here, I tried manually binding to port 53, and I was able to without issue.

Decronym Bot , (edited )

Acronyms, initialisms, abbreviations, contractions, and other phrases which expand to something larger, that I've seen in this thread:

Fewer Letters More Letters
DNS Domain Name Service/System
IP Internet Protocol
PiHole Network-wide ad-blocker (DNS sinkhole)

[Thread for this sub, first seen 15th Mar 2024, 07:25]
[FAQ] [Full list] [Contact] [Source code]

backgroundcow ,

After checking that you can open port 53 udp yourself with, say, nc (which you tried), strace the binary that tries to open port 53 and fails, and find the system call that fails. You can compare it with an strace on nc to see how it differs.

If this doesn't clue you in (e.g., you see two attempts to listen to the same port...) Next step would be to find in the source code where it fails (look for the error message printout) and start adding diagnostic printouts before the failing system call and compile and run your edited version.

Kalcifer OP ,
@Kalcifer@sh.itjust.works avatar

See the solution in the post.

possiblylinux127 ,
@possiblylinux127@lemmy.zip avatar

Have you made sure Podman has access to 53? It runs as a local user

Kalcifer OP ,
@Kalcifer@sh.itjust.works avatar

See the solution in the post.

scott ,
@scott@lem.free.as avatar

This post from Stack Exchange might help you, switching 80 for 53, of course.

Kalcifer OP ,
@Kalcifer@sh.itjust.works avatar

See the solution in the post.

scott ,
@scott@lem.free.as avatar

Ahhh... very good. I avoided all this by running Pihole on its own IP on the LAN using a bridged interface from the host.

PoliticallyIncorrect ,
@PoliticallyIncorrect@lemmy.world avatar

What about using Docker image instead?

FrostyCaveman ,

[Thread, post or comment was deleted by the author]

  • Loading...
  • Kalcifer OP ,
    @Kalcifer@sh.itjust.works avatar

    systemd-resolved is not running ­— it isn't even installed on the device. I also already mentioned that I have looked into this fact within the body of the post.

    dandroid ,

    I used this guide and it worked perfect for me:

    https://b-woody.com/posts/2022-05-12-pihole-on-a-rootless-podman-container/

    Kalcifer OP ,
    @Kalcifer@sh.itjust.works avatar

    I am running the container in priveleged mode, so it has access to those ports. That being said, I already tried in unpriveleged mode by giving access to ports above 53 in /etc/sysctl.conf and applying it with sysctl -p. All to no avail, of course.

    catloaf ,

    Can you use nc to bind to udp 53 yourself?

    Kalcifer OP ,
    @Kalcifer@sh.itjust.works avatar

    Yup. I ran # nc -u -l 0.0.0.0 53 to listen on port 53. Then I ran # drill @127.0.0.1 53 archlinux.org in another shell. I saw the request in the listening shell.

    catloaf ,

    Hmm. Can you run another service like https://github.com/vhiribarren/docker-echo-server on udp 53? If it fails, can you bind another port, both under and over 1024, with success? That would tell us if it's something with that image or a system problem.

    Edit: I remember when I was running pihole under docker, maybe this? Not sure if anyone has mentioned it: https://github.com/pi-hole/docker-pi-hole/issues/968 (the NET_ADMIN capability is different from running as root).

    Maybe also try starting the container without the port maps in the compose file to see if that works?

    Also what network are you using? I think I had to put mine in host networking.

    Kalcifer OP ,
    @Kalcifer@sh.itjust.works avatar

    See the post for the solution.

    Rayzor ,
    @Rayzor@lemmy.ml avatar

    You still running into trouble? Are you able to run ss -alnp as root?

    Kalcifer OP ,
    @Kalcifer@sh.itjust.works avatar

    You still running into trouble?

    Yes.

    Are you able to run ss -alnp as root?

    I have already tried checking if something is listening on 53 in about 10 different ways. That command yields the same outcome as before — nothing appears to be listening on 53.

    possiblylinux127 ,
    @possiblylinux127@lemmy.zip avatar

    Then nothing is listening on 53. Check permissions

    Rayzor ,
    @Rayzor@lemmy.ml avatar

    Ever get this resolved?

    Kalcifer OP ,
    @Kalcifer@sh.itjust.works avatar

    Yeah, take a look at the solution at the top of the post.

    Climatic2469 ,

    In the /etc/systemd/resolved.conf

    Set the DNSStubListener=no like below then restart the network resolvd service this should allow you to run an alternate service on port 53

    #  This file is part of systemd.
    #
    #  systemd is free software; you can redistribute it and/or modify it under the
    #  terms of the GNU Lesser General Public License as published by the Free
    #  Software Foundation; either version 2.1 of the License, or (at your option)
    #  any later version.
    #
    # Entries in this file show the compile time defaults. Local configuration
    # should be created by either modifying this file, or by creating "drop-ins" in
    # the resolved.conf.d/ subdirectory. The latter is generally recommended.
    # Defaults can be restored by simply deleting this file and all drop-ins.
    #
    # Use 'systemd-analyze cat-config systemd/resolved.conf' to display the full config.
    #
    # See resolved.conf(5) for details.
    
    [Resolve]
    # Some examples of DNS servers which may be used for DNS= and FallbackDNS=:
    # Cloudflare: 1.1.1.1#cloudflare-dns.com 1.0.0.1#cloudflare-dns.com 2606:4700:4700::1111#cloudflare-dns.com 2606:4700:4700::1001#cloudflare-dns.com
    # Google:     8.8.8.8#dns.google 8.8.4.4#dns.google 2001:4860:4860::8888#dns.google 2001:4860:4860::8844#dns.google
    # Quad9:      9.9.9.9#dns.quad9.net 149.112.112.112#dns.quad9.net 2620:fe::fe#dns.quad9.net 2620:fe::9#dns.quad9.net
    DNS=127.0.0.1
    #FallbackDNS=
    #Domains=
    #DNSSEC=no
    #DNSOverTLS=no
    #MulticastDNS=no
    #LLMNR=no
    #Cache=no-negative
    #CacheFromLocalhost=no
    DNSStubListener=no
    #DNSStubListenerExtra=
    #ReadEtcHosts=yes
    #ResolveUnicastSingleLabel=no
    
    Telodzrum ,

    This is the correct solution. OP's issue is a common one which shows up on the PiHole discussion forums regularly. I frequently forget to perform this step on new installs, too.

    Kalcifer OP , (edited )
    @Kalcifer@sh.itjust.works avatar

    This is the correct solution.

    No it isn't. systemd-resolved is not running ­— it isn't even installed on the device. I also already mentioned that I have looked into this fact within the body of the post.

    Kalcifer OP ,
    @Kalcifer@sh.itjust.works avatar

    systemd-resolved is not running ­— it isn't even installed on the device. I also already mentioned that I have looked into this fact within the body of the post.

    Climatic2469 ,

    That wasn't clear from the body, as it says you investigated systemd but hadn't resolved the issue. I'm glad you found a solution though and have a good day.

    zikal ,

    Check with "sudo ss -tupan | grep 53" to see if there is a process already using port 53. It might be systemd's resolver or something like that.

    Kalcifer OP ,
    @Kalcifer@sh.itjust.works avatar

    If you read the post, I already did that. It shows no device using port 53.

    talentedkiwi ,

    This may sound silly, but have you tried restarting? I feel like that worked for me when I had a similar issue in the past. Something was holding onto the port, but it wasn't showing up anywhere. Restart got it to let go and it worked afterwards.

    Kalcifer OP ,
    @Kalcifer@sh.itjust.works avatar

    Yeah, I have already tried rebooting the device. To no avail, unfortunately.

    talentedkiwi ,

    Darn.

    I suppose you could also try using the lsof command to see if it shows anything different?

    Some examples: https://www.cyberciti.biz/faq/unix-linux-check-if-port-is-in-use-command/

    Kalcifer OP ,
    @Kalcifer@sh.itjust.works avatar

    I appreciate the suggestion, but I have already tried essentially all alternative network commands to see if one might yield a different result. They, of course, all show the same things — nothing is listening on 53. That command specifically only shows that sshd is listening on 22, which is expected.

    talentedkiwi ,

    Hopefully someone smarter than me can help. You can always do what I do, and just blow up the install and start fresh. 😂

    Kalcifer OP ,
    @Kalcifer@sh.itjust.works avatar

    You can always do what I do, and just blow up the install and start fresh.

    This may be what I'll have to do. I just don't understand what's going wrong here. It's so strange.

    Kalcifer OP ,
    @Kalcifer@sh.itjust.works avatar

    If you are interested, a solution was found. See the post for the update.

    kylian0087 ,

    Ports below 1024 are by default reserved for root. So unless you use sudo or change this you wont be able to use port 80 and 53 without root

    oxomoxo ,

    This article covers the solution https://access.redhat.com/solutions/7044059

    psmgx ,

    Huh doesn't require enterprise subscription to see that solution

    Kalcifer OP ,
    @Kalcifer@sh.itjust.works avatar

    That is not the solution. As I have already mentioned a number of times, I am running the container in priveleged mode — I am running the container as root. It has access to all ports.

    possiblylinux127 ,
    @possiblylinux127@lemmy.zip avatar

    Have you tried docker?

    tvcvt ,

    I’ve not done much with podman, but my first thought is that port 53 is privileged and usually podman runs as a non-privileged user, right? Do you have some mechanism in place that would allow podman to use port 53?

    Kalcifer OP ,
    @Kalcifer@sh.itjust.works avatar

    I'm currently running it in privileged mode (as sudo) so it has access to all the ports.

  • All
  • Subscribed
  • Moderated
  • Favorites
  • random
  • selfhosted@lemmy.world
  • incremental_games
  • meta
  • All magazines