This year's cold start exercise was a disaster

2026-09-21 tinkering homelab infrastructure

About once a year, I do an uncontrolled shutdown of my apartment by turning off the power at the breaker panel to see how well it recovers when the power comes back on. The place is full of networked electronics depending on each other, and new services are typically provisioned into an environment that is fine and operational. So, plenty of stuff could go wrong when things don’t come back up in the required order, or not at all.

After restoring power, I check all services reported unhealthy in Grafana and try to restore function remotely from outside the home network to simulate the worst possible workable recovery conditions, which also exercises auto-recovery of the main VPN, and the backup connectivity solutions.

This is useful to reduce stress when such an outage happens while traveling, it highlights opportunities for improvement, and sometimes uncovers something that was on the verge of breaking anyway.

Lots of things went wrong this year.

VictoriaMetrics data corruption🔗

I’ve had data loss with VictoriaMetrics before during an accidental power cut. With bad timing, metadata.json and parts.json can list empty/corrupted partitions, which blocks startup. The root cause is partition files and metadata not being fully flushed to disk (or in my case, network storage) before the power goes out.

Solution🔗

The correct solution would be a uninterrupted power supply (UPS), but that’s bulky and expensive. I’d rather go solar with batteries, but that’s difficult in a rental apartment, so I might have to bite that bullet.

On a past occasion, I restored from the last backup and sucked up the data loss not covered by the backup, hoping that it wouldn’t happen every time. It doesn’t happen every time, but it happened again this time, so I commissioned an agent to generate a data recovery script that iteratively removes the youngest partition (assuming it’s most likely to be corrupted) from the partition list, retries startup, and keeps doing that until startup succeeds.

Crude and lazy, but I ended up losing only minutes of data, which is an upgrade. I’m tempted to make the hack permanent by running the script on boot.

Keycloak/OAuth2 proxy race🔗

Keycloak can take a while to start, because it’s a beast with endless features. The OAuth2 proxy server that controls access to Frigate with Keycloak-provided SSO and role management starts up much faster, and seems unable to process logins if the OIDC server backing it wasn’t up first. At least that’s what I’m observing.

The quick fix was to simply restart the OAuth2 proxy.

Solution🔗

Keycloak and the OAuth2 proxy run on different hosts, so I can’t make one container depend on the other.

Next best thing is probably adjusting the OAuth2 proxy health check to cover functional Keycloak connectivity, paired with a generous restart policy.

PoE failure🔗

This one is hard to protect against, and gives me the most anxiety during prolonged absences due to the potential impact: One of the Raspberry Pi boards powered via Power-over-Ethernet (PoE) simply didn’t come back up. It ran Home Assistant, which made it extra inconvenient, because that’s what I use to toggle PoE power when something goes wrong.

Solution🔗

I looked up the MAC address to powercycle the PoE port using the web interface of the managed switch that powers all the PoE devices, but that didn’t change the situation.

With the device not responding to pings, and repeated powercycling being ineffective, I was out of remote options. If I had been traveling, I’d have to kindly ask my less technically inclined friend who stops by to feed the cats to be my remote eyes and hands. It was just a test, so I could walk over to the rack myself to pull out the dead board for inspection. All LEDs were out, so I tried a fresh Ethernet cable. That too didn’t work, so I tried different ports with the original cable.

In the end, two rows of ports on top of each other, 4 ports each, were not supplying any power. Switching over to the next port group worked again.

Permanent solution?🔗

This gives me anxiety, because most of my services run on PoE-powered Raspberry Pi boards, the only exception being data storage and compute-heavy stuff. This includes all my vectors for external network access.

If PoE goes out on the whole switch, I’m effectively locked out of my home network until I can get physical access and attach dedicated power supplies to the Raspberry Pi boards. Keeping an extra power supply attached permanently would defeat the purpose of PoE a little, because it would no longer be a convenient way to reset a device that stopped responding - PoE would be ignored in favor of the “normal” power supply. Adding a small management/KVM device just for resetting seems a bit excessive and costly.

A normal-size UPS won’t help either, because it’s only intended to bridge a limited time period that allows for graceful shutdown. A pretty beefy UPS could bridge longer outages.

To work around this amateurish hardware setup that doesn’t support redundant PoE, a two-pronged approach could get us much further:

  1. For emergency network connectivity, add a completely separate machine with a traditional power supply and direct router connection.
    • As an added benefit, this also work around internal networking issues from the main switch failing.
  2. Instead of traditional USB-C/microUSB power supplies required by Raspberry Pi boards, use a PoE-to-USB adapter whose Ethernet side is hooked up to a secondary PoE switch that I don’t own yet.
    • For extra network redundancy, I could also add an extra network interface via USB.

Redundant switching, at least for essential services, would definitely be most robust solution.

Encrypted network shares🔗

The one that always gets away. I take data protection seriously, so sensitive data like copies of important documents is stored in encrypted network shares. That makes it a bit harder for burglars to steal my identity on top of my stuff.

In the past, I’ve had my share of boot failures due to unavailable NFS volumes, so I already subscribed to the cult of the nofail mount option. Hosts involved in remote access are clear of remote dependencies other than the network to minimize risk of remote access loss.

So far, I’ve been entering encryption keys manually whenever the NAS finished booting. Theoretically, there are solutions for pulling encryption keys from attached devices like a YubiKey, but that assumes that the key owning device sits in a secure environment, like a data center that offers strong, enforceable contractual and legal guarantees.

There is no security team permanently on site at my apartment (easily scared cats don’t count), so I’d rather keep the keys in my head.

Challenge/response solution?🔗

Here is a half-baked idea I had while writing down my experience:

  1. The NAS most likely has Internet connectivity after booting up, so it could use a trusted asymmetric key pair to request decryption keys via push notification from a “key management” app on my phone.
  2. If the request is signed by the key I trust, the app can verify the request and send back the key encrypted using the NAS’ public key.
    • Add in a human confirmation step so the key is only transmitted when it seems plausible.
  3. Vanilla mount options are probably not complex enough to combine nofail with generous retries and timeouts, so SystemD auto mounting might be the only option to mount encrypted volumes with enough grace period so I have time to respond to the challenge.
    • As much as I have qualms about SystemD becoming a hardly negotiable dependency of Linux, I would lie to myself if I didn’t appreciate its features for more involved admin tasks.
  4. If already using SystemD auto mounts, I can turn services requiring the NFS volume into dependents of the auto mount to make sure they only start once the data is accessible.

There is still a hole in the chain of trust: physical access. The NAS would need the ability to detect tampering, triggering automatic deletion of the private key, so bad actors with physical access can’t just ask me for an encryption key.

That’s a lot of infrastructure, considering I have nothing specific to fear, but this is a hobby, so I might do it anyway for the fun of it. At least after checking who did a better version of this before.

What went well?🔗

All that sounds like I’m running a ship with gaping holes that only stays afloat because the water pump’s throughput exceeds the holes’, and I help out with a bucket whenever there is a bottleneck. It’s not all bad, though - the stuff that matters most just works.

  • My home-grown rack cooling solution recovered perfectly.
  • Outside connectivity via VPN as primary and persistent SSH tunnel as secondary method recovered perfectly.
    • I still added a second backup SSH tunnel on a different host, because the aforementioned methods were running on the same one.
  • DNS, networking gear (save for the PoE issue), and the sensors that pepper the apartment all came back fine.
  • Unencrypted NAS volumes were available.
  • Frigate (video surveillance) was back up and detecting/recording well before web access was restored manually.
  • Grafana worked fine after fixing VictoriaMetrics.
  • HomeAssistant also worked fine after solving the PoE issue.
  • The gaggle of small services that aren’t worth a dedicated mention came back without issues.
  • The battery-equipped cat fountain continued supplying my feline family members with potable water.
    • My qualms about putting down cash for a UPS don’t apply when it comes to living beings.

So, unless the main switch suffers more problems the next time I’m traveling, I should be able to recover remotely from a power outage just fine. Plus, I’m just a few measures away from solving the most grievous plausible issues:

  • UPS for graceful VictoriaMetrics shutdown.
  • Secondary PoE switch with PoE-to-USB power supplies for redundancy.
  • Better network share dependency management with SystemD managing mounts and dependent services.