Writing tagged "homelab"
This year's cold start exercise was a disaster
About once a year, I do an uncontrolled shutdown of my apartment by turning off the power at the breaker panel to see how well it recovers when the power comes back on. The place is full of networked electronics depending on each other, and new services are typically provisioned into an environment that is fine and operational. So, plenty of stuff could go wrong when things don’t come back up in the required order, or not at all.
After restoring power, I check all services reported unhealthy in Grafana and try to restore function remotely from outside the home network to simulate the worst possible workable recovery conditions, which also exercises auto-recovery of the main VPN, and the backup connectivity solutions.
This is useful to reduce stress when such an outage happens while traveling, it highlights opportunities for improvement, and sometimes uncovers something that was on the verge of breaking anyway.
Lots of things went wrong this year.