Assume Your App Gets Popped

Here’s a sentence that should make any sane sysadmin’s stomach drop: “I run a DNS resolver that anyone on the internet can point their devices at.” No auth. No allowlist. Just an open DoH/DoT endpoint sitting there, resolving queries for adblocking, for total strangers, 24/7.

I built exactly that. And as far as I can tell, nobody ever got in.

That “as far as I can tell” is doing real work in that sentence, and I mean it. I’m not going to stand here and tell you the box was unhackable, because nothing is, and anyone who tells you their box is bulletproof is either lying or hasn’t been running it long enough. What I can tell you is that I stacked enough layers between “random person on the internet” and “root shell on my box” that even if one layer cracked, the next one was there to catch it, and the one after that would’ve screamed bloody murder the second anything moved.

This is the architecture. Not a step-by-step install guide — I’m assuming you already know how to stand up a DNS resolver and how Docker works. This is the “why did I build it this way” tour. And here’s the thing: the service happened to be DNS, but almost none of this is DNS-specific. It’s the pattern for shoving any possibly-vulnerable app onto the public internet and containing the damage when — not if — something gets poked. Swap AdGuard for a game server, a webhook receiver, a self-hosted whatever, and the same three layers apply.

Full example: Clone the working files at github.com/KingPin/sumguy-examples/tree/main/security/assume-your-app-gets-popped

The threat model, briefly

A public DNS resolver is a genuinely juicy target. It’s not glamorous like a database full of credit cards, but think about what it touches: every query tells you what domains a device is about to talk to, it’s a great vantage point for DNS cache poisoning shenanigans, and if you can pop the host running it you’ve potentially got a foothold that sees traffic patterns for everyone using it. Add “exposed on purpose, to strangers, forever” and you’ve built yourself a permanent, low-effort target that scanners will find within hours of the port going live.

So the question isn’t “how do I stop attacks” — you can’t, not against internet background noise. The question is “when something does get through the front layer, how far can it actually go, and how fast will I know about it.” That’s the whole game. Defense in depth isn’t a buzzword here, it’s the only strategy that scales against an attacker you can’t identify in advance.

Layer 1: Rootless container, unprivileged user

The DNS resolver itself runs as a rootless Docker container, launched by a dedicated Linux user that owns nothing else on the box and has no sudo rights. Not “runs in a container as root inside the container” — actually rootless, where the container runtime itself has no root privileges on the host.

I’ve already written up how to get rootless Docker running end to end — see the Related Reading below — so I won’t re-derive it here. What matters for this article is why it’s non-negotiable for a public-facing service:

The math on blast radius changes completely. A traditional “run Docker as root, just don’t add --privileged” setup means a container escape — and they do happen, they’re rare but they happen — drops the attacker onto your host as root. Game over, they own the box. With rootless Docker, that same escape drops them into a normal, unprivileged Linux user’s context. No write access to /etc, no ability to install a backdoor service, no touching other users’ files, nothing they can do to persist beyond what that one low-privilege account could already do.

It’s the difference between hiring a valet who has the keys to your car versus one who has the keys to every car in the garage. Even if the valet turns out to be a crook, you’d rather it be the first kind.

Here’s the compose file, run under a dedicated dns-svc user with no shell access and no sudo:

services:
  adguard:
    image: adguard/adguardhome:latest
    container_name: public-dns
    restart: unless-stopped
    ports:
      - "443:443/tcp"   # DoH
      - "853:853/tcp"   # DoT
      - "3000:3000/tcp" # admin UI, firewalled to a management VLAN only
    volumes:
      - ./work:/opt/adguardhome/work
      - ./conf:/opt/adguardhome/conf
    security_opt:
      - no-new-privileges:true
    cap_drop:
      - ALL
    cap_add:
      - NET_BIND_SERVICE   # the ONE capability it needs: bind 443/853
    read_only: true
    tmpfs:
      - /tmp

Notice the pile of extra hardening even inside the rootless setup: cap_drop: ALL strips every Linux capability, then cap_add: NET_BIND_SERVICE hands back exactly one — the ability to bind low ports like 443 and 853, which AdGuard genuinely needs and nothing else does. (Drop it all without adding that back and the container just fails to bind its ports — a good reminder that “cap_drop: ALL” alone isn’t a magic hardening incantation.) On top of that, read_only: true means the container’s own filesystem can’t be modified at runtime (state goes to the mounted volumes only), and no-new-privileges blocks privilege escalation via setuid binaries even if one somehow ended up in the image. Rootless is the outer wall; these flags are the murder holes in it. For more of this — seccomp profiles, distroless images, the whole checklist — see the Docker Security Hardening piece linked below.

Layer 2: The tripwire that watches everything

This is the part of the setup most people haven’t touched, and it’s honestly the one I’d tell you to steal first if you only take one idea from this article.

The idea is simple and old — Tripwire and AIDE (Advanced Intrusion Detection Environment) have been doing this since the 90s — but it’s criminally underused on home lab and small-VPS setups. You take a cryptographic snapshot of every file that shouldn’t change (binaries, configs, cron jobs, systemd units, SSH host keys, the works), and you check it on a schedule. If a hash doesn’t match what you recorded, something touched a file it had no business touching, and you want to know about that in minutes, not months.

Think of it like a home alarm system’s motion sensors, except instead of “something moved in the living room” it’s “/usr/bin/curl is now 400 bytes bigger than it was yesterday and its checksum changed.” That’s the kind of anomaly that a rootkit or a webshell drop produces, and it’s exactly the kind of thing that doesn’t show up in a ps aux glance.

I used AIDE. Here’s the practical config — trimmed to the parts that matter for a box like this:

# Rule definitions: what to check per class of file
NORMAL = p+i+n+u+g+s+m+c+md5+sha256
DIR    = p+i+n+u+g

# System binaries and libraries — full checksum, alert on ANY change
/bin        NORMAL
/sbin       NORMAL
/usr/bin    NORMAL
/usr/sbin   NORMAL
/lib        NORMAL
/usr/lib    NORMAL

# Configs and units — same, these should be static between deploys
/etc        NORMAL
/etc/ssh/sshd_config NORMAL
!/etc/mtab
!/etc/adjtime

# SSH host keys — if these change without you doing it, panic
/etc/ssh/ssh_host_rsa_key.pub NORMAL
/etc/ssh/ssh_host_ed25519_key.pub NORMAL

# The container runtime's own binaries and unit files
/usr/lib/systemd/system NORMAL
/home/dns-svc/.config/systemd/user NORMAL

# Don't bother watching things that legitimately change constantly
!/var/log
!/home/dns-svc/adguard/work
!/tmp

AIDE builds a baseline database (aide --init), you move it into place, and then a cron job (or a systemd timer, which is the better option and covered in a future article) runs aide --check on a schedule and mails or pages you the diff:

# Run AIDE integrity check every 2 hours, mail any diffs
0 */2 * * * /usr/sbin/aide --check | mail -s "AIDE report: $(hostname)" [email protected]

The actual alerting matters more than the tool. A diff sitting unread in a mailbox is worthless — I piped mine through a tiny script that forwarded any non-empty AIDE report to a webhook that pings my phone. The point isn’t “have logs of the intrusion after the fact,” it’s “get woken up before the attacker finishes what they’re doing.” An inotify-based watcher (like auditd with file watch rules, or something as simple as inotifywait -m -r /etc /usr/bin piped to a script) gets you near-real-time alerts instead of a 2-hour polling window, at the cost of a bit more noise to tune out. Pick your tradeoff — for a box this exposed, I’d rather tune out some noise than miss a two-hour window.

Layer 3: No inbound SSH. None. Zero.

Here’s the part that gets people’s eyebrows up, and it’s the actual centerpiece of this whole setup.

The public-facing box has no SSH daemon listening on any interface reachable from the internet. Not “SSH on a weird port,” not “SSH behind Fail2ban,” not “SSH with key-only auth and a strict allowlist.” Those are all fine practices — I’ve written about several of them — but they all share one property: there’s a listening socket for an attacker to find and try things against. Every one of those defenses is a wall. Walls can be found, mapped, and eventually leaned on.

So I removed the wall entirely. There is no door to knock on.

Instead, the flip is this: the public box makes an outbound connection to a bastion host I control, and that outbound connection carries a reverse SSH tunnel. The bastion — which itself is locked down hard and NOT exposed the same way — ends up with a local port that, when I connect to it, actually lands me back on the public box. The public box dialed home. I don’t dial it.

This is a genuine mental model flip and it’s worth sitting with for a second. Normally you think “I need to reach my server, so I open a port on it and connect in.” Here, the server reaches out to a machine I trust, and I ride that same connection backwards. From the internet’s point of view, there’s nothing to scan on port 22 (or any port) on the public box for SSH, because it was never listening in the first place. nmap finds 443 and 853 doing their DNS job, and that’s the whole surface.

The mechanism is autossh, which is just OpenSSH with a watchdog that reconnects the tunnel if it drops — because a reverse tunnel that silently died at 3 AM is worse than useless, it’s a false sense of security. Wrapped in a systemd unit so it survives reboots and gets restarted if it ever exits:

[Unit]
Description=Reverse SSH tunnel to bastion (outbound only, no inbound SSH)
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=tunnel-svc
Environment=AUTOSSH_GATETIME=0
Environment=AUTOSSH_POLL=30
ExecStart=/usr/bin/autossh -M 0 -N \
  -o "ServerAliveInterval=15" \
  -o "ServerAliveCountMax=3" \
  -o "ExitOnForwardFailure=yes" \
  -o "StrictHostKeyChecking=yes" \
  -i /home/tunnel-svc/.ssh/id_ed25519_bastion \
  -R 127.0.0.1:22022:localhost:22 \
  [email protected] -p 443
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

A few details that matter:

-R 127.0.0.1:22022:localhost:22 is the reverse forward: it tells the bastion “bind port 22022 on your loopback only, and anything that hits it gets forwarded back down this tunnel to port 22 on me.” Binding to 127.0.0.1 on the bastion side (not 0.0.0.0) means that port isn’t reachable from the internet either — only from processes on the bastion itself, which is where I SSH in from after already authenticating to the bastion normally.
[email protected] -p 443 — yes, port 443. Outbound connections on 443 blend into normal HTTPS egress and survive restrictive outbound firewalls, which matters if this box ever lives somewhere with locked-down egress rules.
The tunneling account (tunnel-svc) on the public box has a key that can only open this one forward, enforced bastion-side in authorized_keys with restrict,permitlisten="127.0.0.1:22022". The restrict keyword turns everything off (no pty, no agent forwarding, no X11, no port forwarding), and permitlisten then re-enables exactly one thing: binding that single loopback port for the reverse tunnel. (Note it’s permitlisten, not permitopen — the latter governs outbound -L forwards; a reverse -R forward is constrained by permitlisten. Easy mistake, and getting it wrong leaves the key more capable than you think.) Even if someone fully compromised the public box and stole that key, it can’t do anything on the bastion except open this exact tunnel.
StrictHostKeyChecking=yes — pin the bastion’s host key ahead of time. Don’t let this silently accept a new key from a MITM’d connection.

To actually get in, from the bastion:

ssh -p 22022 admin@localhost

That’s it. I SSH into the bastion (which does have inbound SSH, hardened separately, on a machine that isn’t running a public-facing service), then hop through the loopback port the reverse tunnel created. If someone burns down the public box entirely, the bastion is unaffected — it never trusted the public box with anything beyond “you may open this one forward.”

What about chisel?

If you’ve spent any time in the tunneling space you’ve probably run into chisel, and it’s fair to ask why I didn’t reach for it here.

Chisel is a genuinely good tool: a single Go binary containing both client and server, it tunnels TCP and UDP over HTTP (secured with crypto/ssh under the hood), auto-reconnects with exponential backoff out of the box, supports reverse port forwarding just like what I described above, and even ships a built-in SOCKS5 proxy. Its headline use case is punching through restrictive firewalls — environments where outbound traffic is locked to 80/443 and nothing else gets out, so tunneling over plain HTTP/WebSocket is what actually survives.

Here’s my honest take: for this specific use case — hardening a box whose entire job is minimizing attack surface — a plain SSH reverse tunnel is the more defensible default, and I’d tell you to reach for chisel only if you have a specific reason SSH won’t work.

The reasoning isn’t “SSH is trendier.” It’s:

OpenSSH’s crypto and protocol implementation has had decades of hostile-eyes auditing. Chisel’s crypto/ssh usage is fine, but it’s a smaller, younger codebase carrying less scrutiny than the software literally everyone runs.
Every binary you add is attack surface and a patch obligation. The whole point of this box is “as little running on it as possible.” sshd is already there, already patched by your distro’s security team, already the thing you understand cold. Chisel is a third-party static binary you now have to track for CVEs and update out-of-band.
You gain ergonomics, not security. Chisel’s real wins — one static binary, free reconnect logic, no need for sshd on the far end, built-in SOCKS5 — are convenience features. autossh gets you the reconnect logic for free too, and I already have sshd on both ends.

Where chisel genuinely earns its spot: if your egress is locked down to 80/443 and a raw SSH connection on any port gets dropped by an upstream firewall, if you flatly can’t or won’t run sshd on the far end, or if you want multiple tunnels and a SOCKS5 proxy multiplexed over a single connection without hand-rolling it. Those are real scenarios. Mine wasn’t one of them — outbound 443 to my own bastion was never going to get blocked by anything upstream, and I wasn’t trying to avoid running sshd. Pick the tool for your actual constraint, not because one of them is newer.

Putting it together

Strip away the specifics and the pattern is: minimize what a breach can reach (rootless, capability-dropped, read-only containers), maximize how fast you notice one anyway (file-integrity monitoring with real alerting, not just logs nobody reads), and eliminate entire categories of attack by removing the thing attackers need (no listening SSH, full stop, replaced by a connection the box initiates itself).

None of these layers is exotic. Rootless Docker is documented. AIDE is decades old. Reverse SSH tunnels are a FAQ-page trick. What makes it work is running all three at once, so a failure in one doesn’t hand over the whole box — and building the habit of assuming layer one will eventually get poked, so layers two and three are there waiting when it happens.

Did it work? Nothing ever showed up in the AIDE reports. No unexplained processes, no modified binaries, no mystery cron entries. I can’t prove a negative and I’m not going to pretend I can — “nobody got in that I could tell” is the honest, hedged version of that sentence, and it’s the only version worth trusting from anyone telling you about their own security setup.