Self-Hosted DNS Outages Show Tested Redundancy Is Critical

A self-hosted DNS outage at home is a reminder that resilience has to be designed and tested, not just bought with hardware. George Michaelson, writing on the APNIC blog, the website of the Asia Pacific Network Information Centre, describes his recent effort to build improved home DNS services with a goal of blocking spam and adverts. He used Pi-hole and AdGuard to run his own in-house DNS, and redirected some known domain names to unrooted IP addresses. The DNS, or Domain Name System, is the service that turns human-friendly website names into the numeric IP addresses computers use to reach each other. His setup made things easier for himself and his home users, and so far it seems to be working well. But the journey to get there was not entirely smooth.

At home, things work until they do not. Michaelson notes that running such a system means he is effectively standing in for a competent, paid systems administrator. The people in his house and the devices on his home network all rely on him not to randomly turn things off or break them. He lists the Digital Video Recorder, the sound system, the home file server that backs up the PCs every night, and even a tiny GPS sensor he runs for a European project monitoring GPS and GLONASS satellite visibility. All of these depend on him as the systems administrator. Unfortunately, DNS systems sometimes need to be rebooted, and when that happens, everything else on the home network stops. Changing the router configuration can bring down the entire wireless network while it reloads. Changing the router's DNS configuration affects DHCP, the network service that automatically assigns IP addresses to devices, which in turn causes a freeze-and-thaw cycle across the network. The real problem, he writes, is that critical services become invisible once they are working. People stop seeing them as infrastructure and start treating them as appliances. That is usually when they discover they never built any resilience into them at all.

The answer, he suggests, is more resilience through more redundancy. In the real world, at work, on a phone's mobile network, and even at conference venues, these services are not run as single instances when they do not have to be. There should be at least two, configured independently, and you should avoid rebooting both at the same time whenever possible. Michaelson admits he had a backup plan, but only in theory. He had not actually implemented it. While moving his DNS to a new platform, he forgot to confirm there was an alternative in place before rebooting the primary. As things around the home network started to break, he discovered it was not a primary at all. It was the only instance. This is the exact moment a home network quietly becomes a production environment, and the consequences are the same as an office outage, even when nobody else is watching.

Now he is trying to reacquire the habits that apply in the workplace. Before any change to his home network, he asks three questions. First, do I understand all the device dependencies related to the thing I am working on? Second, is there an alternative service that my home network will adopt if I change this one? Third, if it turns out I have made a mistake, can I put things back the way they were before I decided to change them? He notes that none of these questions is specific to the DNS. They are the same questions he would ask before making a change to any production system. The only difference is that he forgot his home network had quietly become one. He does not think every home network needs enterprise-grade redundancy. Most of us can tolerate the occasional outage. What the experience reminded him of is that resilience is not something you buy when you purchase hardware. It is something you design into a system and then repeatedly test. As homes accumulate more devices, services and dependencies, the distinction between a home network and a small production environment keeps shrinking. Unfortunately, the systems administrator is still him.

For readers who run their own DNS at home or in a small business, the lesson is direct. Before making any change, test the backup, document the dependencies, and be ready to roll back. If maintaining two independent DNS servers feels like too much work, a managed encrypted DNS service can reduce the burden. AEU DNS offers a private, secure alternative to running and maintaining your own resolver, removing the single-administrator dependency that caused the outage described in the source. It is not a fix for every self-hosted service, but it is a reasonable way to remove the single point of failure in DNS while offering a professionally run alternative.