My Server Keeps Turning Off!

I have a number of personal computers, servers, laptops, microcomputers, PCs and other devices that I work with. Most of the services are published to the Internet via server that hosts this blog. Quite some time ago, I dug out some parts, put them together and had a build server running on a tabletop. Recently I moved it into a random full tower case that I found on top of a wardrobe. To do it, I had to turn it off. And then it hit me.

photo of a piece of junk

Why is this piece of junk running 24 hours a day and 7 days a week?

I'm no longer running builds with it. I plan to in near future but not yet. The only service that I run on it is a git mirror. Once a day, it pulls all of my repositories from all of the hosting services that I use. Thanks to its distributed nature git takes care of "backups" even with a decent redundancy. Still, there are cases in which you may lose it. For example when it isn't used that often by anyone in your near circle. I don't have friends so, here I am.

The server runs 24/7 and the significant service is active for how much? Let's see...

Jan 23 00:00:15 mayohiga systemd[1]: Starting Git Repositories Mirror...
Jan 23 00:01:39 mayohiga systemd[1]: Finished Git Repositories Mirror.
Jan 24 23:10:33 mayohiga systemd[1]: Starting Git Repositories Mirror...
Jan 24 23:12:05 mayohiga systemd[1]: Finished Git Repositories Mirror.
Jan 25 01:18:09 mayohiga systemd[1]: Starting Git Repositories Mirror...
Jan 25 01:19:34 mayohiga systemd[1]: Finished Git Repositories Mirror.

A minute and a half per day.

Well, that's a waste. To be honest it could be even shorter but I didn't bother introducing any parallelism and so it pulls about 70 repositories sequentially.

Right away I configured the server to react to Wake on LAN magic packets. Each odd or so day I manually ran a script that was waking up the server, running the mirror, and then shutting everything down.

Luckily no failures happened during this time.

None at all.

The internet-facing server never broke when running a regular apt upgrade at 4:00 AM on the 31st of December. It didn't get stuck and crash on man-db update somehow after removing old initramfs image and before running fresh mkinitcpio. And even if this actually happened, OVH's KVM console running emergency boot would obviously load right away so that I wouldn't fall asleep while it was still loading at 5:00 AM.

That didn't happen.

Anyway. From time to time I was running a script that triggered the update. At some point I configured better timer unit that started mirrors soon after boot, but I still had to wait for it to turn off the server manually. Of course, in the long run using my usual workstation for running scheduled (or manual) wake ups wouldn't make sense because it uses more power than the target server, it's on only when I'm doing something, and I'm not cron or a timer.

It had to be something smaller. My first idea was to use a Raspberry Pi that would run all the time maybe providing some additional services but in the end I chose something even smaller. An ESP8266 I had laying around.

This thing at 5 V pulls at minimum around 9 mA in deep sleep (in which it spends most of the time) and about 126 mA when using WiFi in a full extent (e.g., for scanning). In comparison, based on purely theoretical estimation, the server was slightly above 2 kWh per day. Of course, ESP turns on the server, so it also consumes power. Still, it stays up for a very limited period. Even assuming an hour of uptime, it's at least about 24 times lower.

coal power plant

In a grand scheme of carbon dioxide emissions it doesn't really change much. The cut to payments isn't really that significant either - maybe up to $250 USD in local currency per year. I wanted this for different reasons. As always, I simply decided to act on a hopefully decent idea, for fun. Second reason is to learn and look at everything else I do with the newly earned perspective (even if it's not yet matured). This especially affects how I will look at queueing builds for this box.

ESP runs a simple set of Lua scripts that periodically connects to network, syncs clock with SNTP, check if it's time for waking the server up based on a simple configuration, and then either goes back to deep sleep for at most 50 minutes or waits at most 5 minutes and broadcasts WoL packets. The sad part is that server's motherboard does not have USB ports with constant voltage, so I connected the little guy somewhere else. For a moment I was considering buying pure ESP8266 not as NodeMCU development board but it was time to wrap up the project.

After ESP was ready, I realized I can and should shutdown the server once all scheduled jobs are done with a systemd unit. This realization came to me only after all work was done. Better late than never. I wrote a target for shutdowns and meanwhile I fixed a race for DNS readiness. I connected all of these units with a couple of simple drop-in configs, for example /etc/systemd/system/ready-for-shutdown.target.d/10-mirror.conf:

[Unit]
After=mirror.service

[Install]
WantedBy=mirror.service

Now, instead of worrying about not running the mirror manually, I patiently wait each evening to watch a spectacle: