This should be a fun post. I’m pretty proud of my server room and the infrastructure I’ve built. Having worked in the I.T. industry specializing in enterprise compute, storage, virtualization etc, I felt my own setup is pretty resilient. But then disaster struck. On Friday last week, we had a power outage. No problem, everything is on UPS backup batteries and good for a half hour. I got notified in the middle of the night that I needed to shutdown some key systems to protect data. Well, that’s where the problem came in. I’m not gonna lie, I was out getting piss drunk and didn’t know my ass from my elbow. The power outage was an extended one and a number of systems went down hard after their UPS ran out of juice.
Typically this isn’t an issue however when the systems came back up, my vSAN was corrupt and I lost a significant number of virtual machines. Luckily, I always keep backups so I had to spend the entire weekend and the better part of this week to recover everything. Happily I haven’t lost any data because of good backups. It just takes a lot of time to recover (considering I house over 100 TB). Anyway, I’m almost done rebuilding and should get back to the 2.0 plugin release soon.