Simple power outage recovery with cronjobs

First of all let me quickly explain (non-exhaustively) what my setup is like.

Being pretty cognisant of the dramatic cost of computing hardware production on our environement, I try to minimize that cost. One way to do so is to only use second-hand hardware and that is exactly what I do. Of course, this is a "hack" indirectly benefitting from our current mode of production, but it's still a stance that showcase that I am unhappy with the current state of affair!

Now for the hardware itself, I am also cognisant of the power consumption of servers and data centers, I used to play with industry-grade servers at home as a young undergrad and the utility bill bizarely quadrupled during that period… Suffice to say that I unhooked everything and disposed of the servers (which I also saved from the thrash initially) right after receiving that bill!

More recently I realized that those SFF desktop computer from Dell and HP are quite a good bang for your buck when it comes to homelabs. They are not that much more expensive than a raspberry pi (given you buy them second-hand and old enough), they are more powerful, you can upgrade the RAM and storage, and they don't even consume that much power (usually less than 30-40W, around 10-15W idle). To top it all off, since they usually have intel proco, you get some nice perf when virtualizing. In short, ideal for relatively low-tech homeservers.

So I have this SFF HP ProDesk that is sitting comfortably behind an OpenWrt router.

Since my servers are hosted at home and we live in a rural area, the power sometimes goes off. From a few seconds, to a few hours. And this happens quite a few times every year.

I am not always home, so ideally I wouldn't want to have to ask the person who is home to reboot my server everytime the power goes off and comes back on. And even when I am home sometimes there are micro-outages at night, so I wouldn't realize until the next time I try to ssh into my server, or just realize that the light is not blinking anymore…

Thus, when I installed OpenBSD on my server, I made sure to configure the BIOS so that the machine would always reboot when power comes back. The only hiccup is that I never managed to make that feature work properly even tho it was enabled in the BIOS.

The solution that I used for years was that I had a Raspberry Pi that would always reboot when the power came back. I would then ssh in the Raspberry Pi and send a WOL packet to my main server which would wake it up. The trouble started once my Raspberry Pi install got corrupted and would not reboot without hooking the Raspberry Pi to a screeen and manually running fschk to recover the filesystem. That setup was very unstable and required a second machine to run constantly, which isn't ideal.

I finally decided to remedy this unpleasant status quo very recently after talking to an acquaintance who was also self-hosting his website. I reached out to him to let him know that his website was down and he told me that he faced intermittent power outages, which reminded of my own situation!

I then proceeded to advise him to setup a cronjob that would regularily check if his server is up and if not, send a WOL packet. I suggested to do that from a second machine that is always up, and that's when things clicked for me. I thought that a laptop repurposed as a server would be great because of the battery acting as a poor man's UPS. Which sounds great in theory, but the practice is that depending on the health of the battery and the length of the outage, the laptop may turn off as well…

And that's when it all clicked in my mind. There was a machine on my network that would always be back up when the power returns. Namely, my OpenWrt router! Since it's based on Linux (hence UNIX-like), it does have cronjobs as well. A quick lookup shows that OpenWrt has a package suited for WOL: Etherwake. Once installed, then it's a matter of adding the cronjob, either with crontab -e if you are SSH'd in the machine, or System > Scheduled Task in the UI. The cronjob looks like this:

*  *  *  *  *  /bin/ping -c 1 <IP ADDRESS> || /usr/bin/etherwake -i <IFACE> <MAC ADDRESS>

Where IP ADDRESS and MAC ADDRESS are the respective IP address and MAC address of the server you want to reboot after a power outage. And IFACE is the local network interface on the router, for me it's br-lan for instance.

Lastly, if you added the cronjob by editing the crontab file directly with crontab -e, you should restart the cron service with the service cron restart command.

And with that simple setup, my server just became resistant to power outages!