Docker's Embedded DNS: Inside 127.0.0.11

Your container can ping the database by name, and you have no idea why

You type ping db inside a container and it works. No DNS server in your compose file, no /etc/hosts edits, no service registry. Then it stops working at 2 AM on a different network and nobody knows where to look.

The answer sits at 127.0.0.11. Every container on a user-defined network talks to a tiny DNS server that dockerd runs for it. That server does two jobs: it answers for container names, service names and aliases, and it forwards every other query to real upstream resolvers. Learn both jobs and most “Docker DNS is broken” tickets become five-minute fixes.

Full example: Clone the working files at github.com/KingPin/sumguy-examples/docker/docker-embedded-dns

Everything below was run on Docker 29.8.2. If you only need the basics (default bridge has no name resolution, so make your own network), Docker Networking Demystified covers that. This post goes into the resolver itself.

The test rig

One compose file, four services, three networks. Save it and keep it handy, because every experiment below uses it.

services:
  web:
    image: nginx:alpine
    hostname: webhost
    networks: [front]

  api:
    image: nginx:alpine
    container_name: demo-api
    networks:
      front:
        aliases: [backend]

  other:
    image: nginx:alpine
    networks: [back]

  lone:
    image: nginx:alpine
    networks: [isolated]

networks:
  front: {}
  back: {}
  isolated:
    internal: true

docker compose up -d --scale web=3

What Docker writes into /etc/resolv.conf

Start with the file the container actually reads.

docker exec demo-api cat /etc/resolv.conf

# Generated by Docker Engine.
# This file can be edited; Docker Engine will not make further changes once it
# has been modified.

nameserver 127.0.0.11
search example.lan
options timeout:2 attempts:2 ndots:0

# Based on host file: '/etc/resolv.conf' (internal resolver)
# ExtServers: [1.1.1.1 8.8.8.8]
# Overrides: [nameservers options]
# Option ndots from: internal

Line by line:

1. nameserver 127.0.0.11 is the embedded resolver. It is the only nameserver the container knows about.
2. search example.lan is copied from the host’s resolv.conf. Keep an eye on it, because it comes back later as a gotcha.
3. options ... ndots:0 means every name is tried as an absolute name first, so db goes to the resolver as db, not db.example.lan. The timeout:2 attempts:2 pair on my machine comes from dns-opts in daemon.json.
4. The comment lines are a report. ExtServers lists the upstream servers the resolver forwards to. Overrides says which fields Docker replaced. On my machine ExtServers shows 1.1.1.1 and 8.8.8.8 because I set dns in daemon.json. Your list will differ.

The header also tells you something useful: once you edit the file, Docker stops updating it in that running container, even when the host’s resolver config changes. A restart or recreate regenerates it. If a long-running container’s DNS looks frozen while its neighbors picked up a change, somebody edited the file by hand.

Now the default bridge, for contrast.

docker run -d --name plain nginx:alpine
docker exec plain cat /etc/resolv.conf

# Generated by Docker Engine.
# This file can be edited; Docker Engine will not make further changes once it
# has been modified.

nameserver 1.1.1.1
nameserver 8.8.8.8
search example.lan
options timeout:2 attempts:2

# Based on host file: '/run/systemd/resolve/resolv.conf' (legacy)
# Overrides: [nameservers options]

No 127.0.0.11, no ndots:0. The container gets real upstream servers directly, and nothing on the box answers for container names. That is why name lookup fails on the default bridge. It is not a missing feature flag. The resolver is simply never wired in. The fix is a user-defined network, and Docker Network Aliases shows how to get more out of one.

Job one: service discovery

The resolver knows every container attached to the same user-defined network as the asker. Here is what resolves from inside demo-api. I query 127.0.0.11 explicitly so the host’s search domain does not muddy the output.

docker exec demo-api nslookup web 127.0.0.11
docker exec demo-api nslookup demo-api 127.0.0.11
docker exec demo-api nslookup backend 127.0.0.11

What I confirmed by running it:

• Compose service name (web): resolves, one A record per replica.
• Container name (demo-api, or the generated <project>-web-1): resolves.
• Network alias (backend): resolves to the container that declared it.
• Short container ID (the 12-character form): resolves.
• web-1 alone: does not resolve. The generated name includes the project prefix, so the lookup needs the full <project>-web-1.
• hostname: webhost in compose: resolves from other containers, to every replica. Plain docker run --hostname plainhost on a custom network registers too.

You don’t have to guess the list. Docker records every name it registered for a container in the DNSNames field of each network:

docker inspect -f '{{range $n, $c := .NetworkSettings.Networks}}{{$n}}: {{$c.DNSNames}}{{println}}{{end}}' demo-api

<project>_front: [demo-api api backend 1f0c2a9b7e3d]

Container name, Compose service name, short ID, aliases, and the hostname when you set one. If a name is not in that list, the resolver will not answer for it.

Replicas get multiple A records

Scale web to three and ask for it repeatedly.

for i in 1 2 3; do
  docker exec demo-api nslookup web 127.0.0.11 | grep Address | tail -n 3 | tr '\n' ' '
  echo
done

Address: 172.22.0.2 Address: 172.22.0.3 Address: 172.22.0.5
Address: 172.22.0.5 Address: 172.22.0.3 Address: 172.22.0.2
Address: 172.22.0.2 Address: 172.22.0.5 Address: 172.22.0.3

Same three addresses, different order each time. That is DNS round robin, and it is all you get. There is no health checking. A dead replica stays in the answer until the container is gone. Clients that take the first record spread load roughly evenly. Clients that cache one address forever do not.

One more gotcha: getent hosts web prints a single address because getent reports the first record only. Use nslookup or dig when you want to count records.

Other networks are invisible

docker exec demo-api nslookup other 127.0.0.11

** server can't find other: NXDOMAIN

demo-api sits on front. other sits on back. The resolver answers per network, so a container only sees names on networks it shares. That is a feature: it is how you keep a database invisible to the public-facing proxy. If a container needs both, attach it to both networks and the resolver merges the views.

The mechanism: how 127.0.0.11:53 reaches dockerd

127.0.0.11 is a loopback address, inside the container’s own network namespace. Nothing listens on port 53 there. Docker cheats with NAT. You can see the rules by attaching a throwaway container to the target’s network namespace.

docker run --rm --net container:demo-api --cap-add NET_ADMIN alpine \
  sh -c 'apk add -q iptables && iptables-save -t nat'

*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [13:746]
:POSTROUTING ACCEPT [33:1806]
:DOCKER_OUTPUT - [0:0]
:DOCKER_POSTROUTING - [0:0]
-A OUTPUT -d 127.0.0.11/32 -j DOCKER_OUTPUT
-A POSTROUTING -d 127.0.0.11/32 -j DOCKER_POSTROUTING
-A DOCKER_OUTPUT -d 127.0.0.11/32 -p tcp -m tcp --dport 53 -j DNAT --to-destination 127.0.0.11:42739
-A DOCKER_OUTPUT -d 127.0.0.11/32 -p udp -m udp --dport 53 -j DNAT --to-destination 127.0.0.11:56718
-A DOCKER_POSTROUTING -s 127.0.0.11/32 -p tcp -m tcp --sport 42739 -j SNAT --to-source :53
-A DOCKER_POSTROUTING -s 127.0.0.11/32 -p udp -m udp --sport 56718 -j SNAT --to-source :53
COMMIT

Read it top to bottom:

1. A packet leaves the container for 127.0.0.11:53. The OUTPUT chain jumps to DOCKER_OUTPUT.
2. DOCKER_OUTPUT rewrites the destination port from 53 to a random high port. I saw 42739 for TCP and 56718 for UDP. Those numbers change per container.
3. Something is listening on that high port, still inside the container’s namespace. That listener is a socket dockerd opened by entering the namespace, so the query lands in dockerd’s resolver code even though the process lives outside the container.
4. On the way back, DOCKER_POSTROUTING rewrites the source port to 53 so the client sees an answer from the port it asked.

On this machine the rules come from iptables-nft, so nft list ruleset shows the same chains with a warning that the table is managed by iptables-nft. The chain names are DOCKER_OUTPUT and DOCKER_POSTROUTING either way.

On the host you can reach the same namespace with sudo nsenter -t <pid> -n iptables -t nat -S, where the PID comes from docker inspect -f '{{.State.Pid}}' demo-api. I could not run that variant without root here, so the attach-a-container method above is the one I tested.

Job two: the forwarder

Names the resolver does not own go upstream. Which upstream decides everything.

docker exec demo-api nslookup example.com 127.0.0.11

The resolver forwards the query to the servers listed in ExtServers, and it dials them from inside the container’s network namespace. So the upstream has to be reachable over the container’s own routes. (There is one exception, covered in the 127.0.0.53 section below.) Where does that list come from? In order of precedence:

1. --dns on docker run (or dns: in compose) for that container.
2. dns in /etc/docker/daemon.json for every container.
3. The host’s resolver configuration.

I verified the first two. With --dns 9.9.9.9 on a custom network, the comment line read ExtServers: [9.9.9.9] and example.com resolved. The third case I took from the Docker docs, which say the embedded server forwards external lookups to the DNS servers configured on the host.

services:
  api:
    image: nginx:alpine
    dns:
      - 9.9.9.9
      - 1.1.1.1

{
  "dns": ["1.1.1.1", "8.8.8.8"]
}

The 127.0.0.53 problem

Ubuntu, Fedora and anything running systemd-resolved points the host’s /etc/resolv.conf at the stub listener 127.0.0.53. That address is useless inside a container: loopback in a container is the container’s loopback. Docker handles the two network types differently:

• Default bridge: there is no embedded resolver to help, so Docker detects the stub and copies /run/systemd/resolve/resolv.conf instead, the file that lists the real upstream servers. The default-bridge output above shows it: the Based on host file line names that systemd path and says (legacy).
• User-defined networks: Docker reads the host’s /etc/resolv.conf as-is and keeps 127.0.0.53 as an upstream. ExtServers lists it as host(127.0.0.53). For upstreams marked host(...), dockerd dials from the host’s namespace instead of the container’s, so your containers get the same answers the host gets, split DNS and VPN domains included. In the Moby source this is the HostLoopback flag in resolver.go.

For how the stub works on the host side, see systemd-resolved Deep Dive.

The practical trap: if you set dns in daemon.json, it overrides all of that. A typo there takes down external DNS for every container on the box at the next restart. Test with a throwaway container before you restart a production daemon.

Search domains bite

Remember search example.lan? The container inherits it from the host. If your host’s search domain has a wildcard record, an unqualified lookup of a missing name can get a real answer from the internet. When I ran busybox nslookup web without naming the server, it returned web.<search domain> results from the public DNS. That is a corporate-VPN classic: the name “works” and points somewhere absurd. Pin the resolver in your test commands, and override with dns_search: [.] in compose if the inherited domain causes trouble.

internal: true networks

An internal network has no route out, so external DNS fails too. Upstreams in ExtServers are dialed from the container’s namespace, which has nowhere to send the packet. Docker also refuses to use the host(...) loopback route for internal networks, so the host’s stub resolver can’t leak answers in either. The lone service shows it.

docker exec <project>-lone-1 nslookup example.com 127.0.0.11

** server can't find example.com: SERVFAIL

The container still gets nameserver 127.0.0.11 and still resolves names of its internal neighbors. Only the forwarding fails, with SERVFAIL rather than a timeout, which makes it quick to spot. If an internal service needs one external name, give that container a second, non-internal network instead of poking holes in the first.

Two quirks before you debug

The resolver answers on both UDP and TCP port 53, and the NAT table above has a rule for each. Standard DNS retries over TCP when an answer is too big for UDP, so keep TCP 53 open if you add your own firewall rules inside a container.

The second quirk is that the resolver is per container, not per host. Each container gets its own listener on its own random high port, which is why the port numbers in the NAT rules differ between containers. You cannot point a host-side tool at 127.0.0.11 and expect an answer.

Debugging checklist for your 2 AM self

Work from the inside out.

1. Read the file. docker exec <c> cat /etc/resolv.conf. Is 127.0.0.11 there? If not, the container is on the default bridge, uses --network host, or someone hand-edited the file.
2. Ask the resolver directly. docker exec <c> nslookup <name> 127.0.0.11. Add dig if the image has it. Direct queries remove search-domain confusion.
3. Check the libc path too. docker exec <c> getent hosts <name> follows nsswitch.conf and /etc/hosts like your application does. If nslookup works and getent fails, the problem is in the app’s resolver config, not Docker.
4. Check the networks. docker inspect -f '{{json .NetworkSettings.Networks}}' <c>. Two containers must share a user-defined network for names to resolve.
5. Check the NAT rules. Use the iptables-save -t nat command from earlier. Missing DOCKER_OUTPUT rules mean the resolver is not wired up for that container, usually because the container is not on a user-defined network or was started in an odd way.
6. Turn on daemon debug logs. Set "debug": true in daemon.json, restart dockerd, and read journalctl -u docker. I have not catalogued the log lines for DNS here, so grep for your query name rather than trusting a format from memory.
7. Test the upstream alone. docker run --rm --dns 9.9.9.9 alpine nslookup example.com. If that works and your normal container fails, the configured upstream is the culprit.

If you want to see how a namespace and its loopback work without Docker doing the plumbing, Network Namespaces by Hand rebuilds the same pieces.

The SumGuy Take

Treat 127.0.0.11 as a small, dumb, reliable service. It does round robin, it respects network boundaries, and it forwards what it does not know. It does not do health checks, SRV records or weighted routing, so past a few replicas you want a real load balancer in front. Keep daemon.json DNS settings boring, read ExtServers before you theorize, and clean up the rig with docker compose down. Your 2 AM self will thank you.

Common Questions

Why can’t my Docker container resolve another container by name?

Container name lookups fail when the two containers do not share a user-defined network. Containers on the default bridge get no embedded DNS at all, and containers on different networks cannot see each other’s names. Attach both containers to one custom network and the lookup works.

What is 127.0.0.11 in Docker?

The address 127.0.0.11 is Docker’s embedded DNS server, reachable only inside containers on user-defined networks. NAT rules in the container’s namespace redirect port 53 on that address to a random local port where dockerd answers. Names it does not own are forwarded upstream.

Does Docker DNS load balance between replicas?

Docker DNS returns every replica’s address in rotating order, which is simple round robin. Docker does not check health, so a failing replica stays in the answer. Clients that cache the first address bypass the rotation. Use a proxy when you need real balancing.

Why does DNS fail on an internal Docker network?

An internal network has no route to the outside, so the embedded resolver cannot reach upstream servers and returns SERVFAIL for external names. Internal container names still resolve. Attach the container to a second, non-internal network if it needs one outside name.

Related Reading

• Docker Networking Demystified
• Docker Network Aliases: The Feature Nobody Uses
• systemd-resolved: The DNS Resolver You’re Already Using Wrong
• Network Namespaces by Hand, No Docker
• DNS Troubleshooting from the Command Line