The Router Said No: Eight Hours of Fighting Consumer Firmware

Eight hours. That's how long it took me to accept that a router is not a server.

It started, as these things always do, with a feature list and a feeling of genuine excitement. It ended with me reading iptables documentation at midnight, a cold cup of coffee at my elbow, and a home network that had briefly stopped working for everyone in the house. My family was asleep. The router was fine. I was the one with the problem.

This is a story about hubris. The router did nothing wrong.

I Bought a Fancy Router. It Runs Docker. What Could Go Wrong?

The Seductive Feature List

The router in question is a high-end consumer unit. Not a hobbyist board, not a rack-mount appliance. A proper home router with a slick app, automatic firmware updates, and a marketing page that lists Docker support in the same bullet column as parental controls and guest Wi-Fi. It has a quad-core ARM processor, 512MB of RAM, and a genuine Linux kernel underneath the polished interface.

That last part is what got me.

The box doesn't lie. Docker runs. You can pull images, start containers, and watch them appear in a little container management panel inside the admin UI. It feels like having a tiny server tucked inside your networking cabinet, which is a genuinely appealing idea. I have a 2014 Intel Atom home server that runs a dozen services and draws about 10 watts. The router is always on anyway. Why not consolidate?

The Reasonable-Sounding Plan

The goal was modest. I wanted to run Caddy as a lightweight reverse proxy on the local network, handling HTTPS termination for a handful of internal services. Caddy is small, its configuration is readable, and it handles certificate management gracefully. Running it in a container on a device that's always powered on seemed elegant. It seemed, to borrow a word I now use with caution, obvious.

The belief underneath all of this was simple: the router runs Linux, Docker runs on Linux, therefore I can use the router's Docker the way I'd use Docker anywhere else. That belief is the villain of this story. Not the router. Not the firmware engineers who built a genuinely impressive piece of consumer hardware. The villain is the assumption that "runs Linux" means "is Linux, for my purposes."

It isn't. It really, really isn't.

Act One: The Port Lottery

Port 443: Already Taken (By What, Exactly?)

The first thing Caddy needs to serve HTTPS is port 443. This is not a negotiable detail. Port 443 is HTTPS. So I wrote a minimal Caddyfile, built a compose file, and started the container.

Error: binding port 443: address already in use

Fine. Something is on port 443. That something is the router's own admin web interface, which runs HTTPS because of course it does. You wouldn't want your admin panel served over plain HTTP. The router is being responsible. I moved to port 80 for a quick test.

Port 80 was also taken. The router uses it to redirect HTTP traffic to the HTTPS admin panel.

Port 8443, 4443, and the Bargaining Phase

This is where the session shifted from setup into negotiation. Port 8443 seemed like a safe bet. It's a common HTTPS alternative, widely used for exactly this kind of situation. It was occupied by the router's remote management service, which allows the manufacturer's app to reach the device from outside the network.

Port 4443 was free. I mapped it, started the container, and got a connection. For about ninety seconds I thought I was done.

Then I realized that Caddy was serving on 4443 but couldn't bind to 80 for the HTTP-to-HTTPS redirect that Let's Encrypt's ACME challenge needs. The ACME flow was broken. The certificate wouldn't issue. I was serving HTTPS on a non-standard port with a self-signed certificate, which is exactly as useful as it sounds.

The router has more opinions about port assignments than a homeowners association has about mailbox colors. Port 80, port 443, port 8080, port 8443, and a handful of high-numbered ports for various management protocols were all spoken for. The router wasn't hoarding them maliciously. Each occupied port represents a service that makes the device useful to the 99% of users who buy it to route packets and forget about it. I am not that user, and that is my problem, not the router's.

Act Two: The Network Mode Gauntlet

MACVLAN and the Broadcom Wall

If the port situation is fundamentally about address conflicts, the logical next move is to give the container its own address. That's what MACVLAN does: it creates a virtual network interface with its own MAC address, which the network treats as a distinct device. The container gets its own IP from the DHCP pool. No port conflicts. No sharing. Clean isolation.

I'd used MACVLAN before on an actual Linux server. It works beautifully there.

On this router, it failed immediately. The error pointed at a missing kernel module: macvlan wasn't compiled into the firmware's kernel. This is common on consumer routers built around Broadcom-class SoCs. The kernel is stripped to the minimum required for routing, wireless, and NAT. Modules that serve general-purpose Linux use cases are simply not there. The feature was never needed for the device's intended purpose, so it was never included.

This isn't a bug. It's a deliberate engineering decision. Every kilobyte of kernel module that isn't compiled in is a kilobyte that doesn't need to be maintained, tested, or updated. For a device that ships to millions of homes and needs to work reliably for years, that tradeoff makes complete sense.

Bridge Mode, Host Network, and Progressive Obscurity

Bridge mode was next. Containers on a bridge network can communicate with each other, and with some configuration, they can reach the broader LAN. I got the container running. I could ping it from within the router. I could not reach it from any other device on the network. The routing between the Docker bridge subnet and the LAN was not behaving the way it would on a standard Linux host, and the error messages were becoming progressively more abstract.

Host network mode was the last option. In host mode, the container shares the router's network stack entirely. No virtual interfaces, no bridge, no isolation. The container sees the router's interfaces directly.

This partially worked, which is almost worse than not working at all. Caddy started. It bound to a port. Traffic reached it from the LAN. But the container was now sharing a network namespace with the firmware's own processes, and interactions between Caddy's listeners and the firmware's management services produced behavior that was genuinely difficult to reason about. Requests would succeed intermittently. Some source addresses were reachable and others weren't. The error messages at this stage were the kind you find in one Stack Overflow answer from 2019 with three upvotes and no accepted solution.

Each of these failures is the firmware doing exactly what it was built to do: optimize for routing, not for container orchestration. The two goals are not compatible in the way I had assumed.

A Brief Interlude: Googling Errors at 11pm

At some point around hour four, the search queries got longer.

The first searches were clean and confident: "Caddy Docker router port 443." The results were helpful. By hour four, the queries looked like this: "Docker bridge network LAN routing asymmetric consumer firmware iptables FORWARD chain drop." The results were a forum post from 2019 with no replies, a GitHub issue marked "stale" and closed by a bot, and one genuinely useful thread written entirely in German.

"Solved it. Just had to add the rule before the FORWARD chain. Hope this helps someone." The post with the solution. No further context provided.

The German post had the answer, probably. Google Translate gave me the gist. The gist required three more things I hadn't configured yet.

This is the sunk cost trap, and it's a specific flavor of it that developers know well. It's not "I've spent money, I can't stop now." It's "I should be able to do this." That phrase is a cognitive distortion dressed up as engineering confidence. The belief that a problem is solvable, combined with the belief that you're capable of solving it, combines into a stubbornness that keeps you at a keyboard well past the point of diminishing returns.

Every developer has a version of this night. The specific technology changes. The 11pm forum-browsing energy is universal. You're not debugging anymore. You're just refusing to lose.

Act Three: SSH, iptables, and the Thirty-Second Victory

The Rules That Worked (Briefly)

Docker wasn't cooperating. The firmware's network management was getting in the way. The obvious next step, obvious in the way that "I'll just rewire the whole thing" is obvious at midnight, was to bypass Docker's networking entirely and write iptables rules by hand.

The router accepts SSH connections. I had the credentials. I opened a terminal, connected, and started reading the existing iptables state. It was dense. Consumer firmware iptables configurations are not written for human readers. They're written to be fast and correct, and they accomplish both goals at the expense of legibility.

I added a PREROUTING rule to redirect traffic on port 4443 to the container's address. I added a FORWARD rule to allow the traffic through. I flushed the relevant chains and re-applied. Then I opened a browser on my laptop, typed the router's LAN IP with the port, and the Caddy welcome page loaded.

Genuine triumph. Real, uncomplicated satisfaction. I had done it.

Why the Internet Broke and What That Means

Thirty seconds later, my phone showed no internet connection. My laptop's browser timed out on a different tab. I checked another device. Same result. The entire network had lost outbound connectivity.

The explanation, once I understood it, was straightforward. The firmware's NAT setup depends on a specific sequence of rules in the POSTROUTING chain. My additions didn't conflict with those rules directly, but they changed the order of evaluation in a way that broke the masquerade rules the firmware uses to send traffic from the LAN to the WAN. Everything on the local network routes through those rules. When they stopped working correctly, nothing could reach the internet.

There's a second problem that would have appeared even if I'd gotten the rules right initially. The firmware periodically rewrites its iptables state. DHCP renewals trigger it. Firmware housekeeping cycles trigger it. Any custom rules written directly into the live iptables configuration get overwritten. The thirty-second victory would have had an expiration date measured in hours regardless of the masquerade conflict.

This is not a flaw in the firmware. It's the firmware maintaining a known-good routing state, which is exactly what it's supposed to do. A consumer router that lets arbitrary SSH sessions permanently alter its routing behavior is a consumer router that breaks in ways the manufacturer can't predict or support. The design decision is correct. It just isn't compatible with what I was trying to do.

The thirty-second victory is funnier now than it was at the time. At the time, I was rebooting the router and explaining to the household why the internet had stopped working.

The Firmware Is Not a Liar. It Just Has One Job

What Consumer Firmware Actually Optimizes For

Step back from the port conflicts and the iptables chaos for a moment. Look at what the firmware is actually trying to do.

A consumer router has a clear set of priorities. Uptime is first. The device needs to run for years without intervention. NAT performance is second. Every packet from every device in the house passes through the routing engine, and it needs to do that quickly and correctly. Wireless stability is third. Dropped connections are the most visible failure mode for home users. And underneath all of it: a non-technical user should never need to touch the command line, read a log file, or understand what iptables is.

The Linux kernel inside is real. It's not a marketing fiction. But it's been stripped, patched, and constrained to serve those four goals. Kernel modules that don't contribute to routing, wireless, or NAT are absent. The iptables configuration is managed by firmware processes that treat it as internal state, not as a user-editable configuration file. The Docker runtime is a genuine feature, but it runs on top of a kernel that was optimized for a different purpose, and that optimization shows up in every edge case.

The Hidden Cost of "It Runs Linux"

"It runs Linux" is technically true the same way "a cargo ship floats" is technically true. Floating is a property of cargo ships. Surfing off the back of one is not something the design accommodates.

The engineers who built this firmware made good decisions. The port assignments make sense. The iptables management makes sense. The stripped kernel makes sense. The Docker support is a real feature that works well for its intended use case, which is running simple, self-contained applications that don't need non-standard ports, custom network modes, or direct access to the host network stack.

Caddy as a reverse proxy is not that use case. It needs port 443. It needs to manipulate traffic at the network level. It needs the kind of control over the network stack that a general-purpose Linux server provides and a consumer router, by design, does not.

The router isn't a liar. The feature list isn't misleading, exactly. It runs Docker. That's true. What the feature list can't tell you in a bullet point is the difference between "runs Docker" and "is a Docker host," because that difference requires understanding what the firmware is optimized for. That understanding is what eight hours bought me.

The 2014 Atom server is still running. I pointed Caddy at it the next morning. It took eleven minutes.

Act Four: The 2014 Atom Server Wins in Five Minutes

Moving the Service to Literally Anything Else

Somewhere around hour seven, the creative problem-solving stopped feeling creative. The router's kernel was missing the overlay filesystem module. Without it, Docker couldn't build its layered storage. Compiling a custom kernel for the router was technically possible and practically insane. So the pivot happened.

In the back of a closet sat an old low-power x86 machine: an Atom-era server from 2014, originally built for light NAS duty, decommissioned when a slightly newer box took over. It had been sitting unpowered for about eighteen months. The plan was simple. Pull it out. Plug it in. Grab the Caddy binary. Write twelve lines of config. Point DNS at the new IP.

The machine booted. Linux came up clean. The Caddy binary ran without complaint. No port conflicts, because nothing else was competing for port 443. No kernel module gaps, because a general-purpose kernel ships with everything. No iptables interference, because the machine's only job was to run Caddy. The reverse proxy served its first request in under five minutes from the moment the machine was plugged in.

What Five Minutes Felt Like After Eight Hours

There's a specific kind of silence that follows a problem finally resolving. This one felt different. Not triumphant. More like the quiet after you've been arguing with someone and then realized they were right the whole time.

The Atom server didn't care about elegant architecture. It didn't have opinions about which services should coexist on which hardware. It didn't protect anything. It just ran the binary.

That's the lesson. Not delivered as a lecture, not buried in documentation. Delivered at 2 AM by a decade-old machine that cost forty dollars used and didn't ask a single question before doing exactly what it was told.

The Most Expensive Belief in Homelab Culture

'I Should Be Able to Do This'

The router's spec sheet said Docker support. The firmware had an Entware package manager. The CPU was technically capable. Every prerequisite checked out on paper, and that's where the trouble started.

The developer brain reads capability as invitation. If the hardware supports it, the reasoning goes, then running it there is valid. More than valid: it becomes a challenge, a puzzle worth solving, a proof of skill. This belief is the most expensive one in homelab culture, and it almost never announces itself as a belief. It shows up as stubbornness, framed as competence.

There's a real cost attached. Eight hours of evening time. Network downtime for everyone in the house while iptables rules were being rewritten. A spouse or roommate asking why the internet is down again. These aren't abstract costs.

When Principles Become Constraints

Homelab culture celebrates creative reuse. That's mostly good. Turning old hardware into useful infrastructure is genuinely satisfying, and the skills built along the way are real. But the celebration of creative reuse has a shadow side: it can make the straightforward solution feel like a failure.

"I should be able to do this on the router" is a principle. Principles are useful until they become constraints that prevent you from shipping the thing you actually wanted to ship.

There's a meaningful distinction between two kinds of stubbornness. The first kind is productive: you're learning something, exploring the edges of a system, building understanding that will pay off later. The second kind is sunk-cost stubbornness: you're eight hours in, the goal hasn't changed, and you're still going because stopping feels like losing.

Hour two was productive stubbornness. Hour seven was something else entirely.

The eight hours weren't wasted in any absolute sense. The understanding of how Docker's storage drivers interact with kernel modules, how iptables chains get ordered on OpenWRT, how Entware packages the userland: all of that is real knowledge. It lives somewhere useful now. But the goal, getting Caddy running as a reverse proxy, could have been reached in five minutes if the right tool had been picked up first.

The right tool for the job is a cliché because it keeps being true. Clichés earn their status through repetition, and repetition happens because the underlying lesson keeps needing to be relearned.

What the Router Actually Taught Me

Appliances Are Not Servers

Consumer routers are appliances. That sentence sounds obvious until you've spent eight hours treating one like a server. An appliance has a specific, well-defined job. The router's job is to move packets between your LAN and your ISP reliably, apply firewall rules, hand out DHCP leases, and stay out of the way. It's very good at that job.

The Docker support in the firmware is a marketing feature. It exists because "runs Docker" is a bullet point that sells units. It's not an architectural invitation. The firmware wasn't designed around the assumption that you'd be running long-lived services on it. The entire system is built to protect network stability, and it will always prioritize that over your custom configuration. It will win. Every time.

Separating concerns makes both functions more reliable. When the router routes and the server serves, each one can be restarted, reconfigured, or replaced without taking down the other. That's not just a software design principle. It's a practical homelab truth.

Respecting the Design Boundary

A cheap general-purpose machine is almost always a better host for self-hosted services than a network appliance. Not because the appliance is bad hardware. The Atom server from 2014 isn't faster or more capable than the router in any meaningful way. It wins because it has no opinions. It doesn't protect anything. It runs what you tell it to run.

The router isn't the villain of this story. It's still sitting on the shelf doing exactly what it was designed to do, and it's doing that job perfectly. It routes. It firewalls. It hands out leases. It hasn't dropped a packet in months. The problem was never the router's performance. The problem was asking an appliance to step outside its design envelope and then being surprised when it pushed back.

Use tools within the boundaries they were designed for. The router is excellent at being a router. Let it be excellent at that.

Before You Try This Yourself: A Practical Checklist

Questions to Ask Before Hosting on a Network Appliance

This checklist was written in retrospect, at 2 AM, with the clarity that only comes from having already made every mistake on it. Consider it a gift.

None of these questions are meant to discourage the exploration. The exploration is worthwhile. But knowing which question you're answering before you start saves the kind of evening that ends at 2 AM with a dusty Atom server and a complicated relationship with your own principles.

The Router Is Fine. It Was Always Fine.

The router did its job the entire time. Every packet it was supposed to route got routed. Every DHCP lease got handed out. Every firewall rule it was designed to enforce got enforced. It never failed at anything it was built to do. The only thing it failed at was being a general-purpose server, which was never its job in the first place.

The Atom server is still running Caddy. It has been running Caddy without interruption since the night it was plugged back in. The router is still routing. The network is stable. Everyone in the house has working internet. The situation is, by every practical measure, fine.

The best homelab decisions are boring ones. Right tool, right job, ship the thing, move on to the next problem. There's no glory in it. There's also no 2 AM debugging session, no network downtime, and no eight-hour detour through kernel module documentation.

The router is fine. It was always fine. Go check your closet for old x86 hardware before you spend an evening fighting firmware that was never on your side.