Nobody touch the RAID (part 1)
On Tuesday afternoon, October 13, one of the hypervisors at work lost its storage. It was a RAID 5 of eight SSDs. One drive dropped, the hot spare started rebuilding, and about three hours later a second drive dropped partway through the rebuild. The controller marked the array failed. The hypervisor's own OS lived on that same array, so the host went down with it, and so did every VM on it.
Honestly, I could have just wiped it and been fine. It wasn't a production box, and everything that mattered was backed up. Some stuff had been parked on it temporarily, so wiping it would have meant a few days of rebuilding things by hand, but nothing would have been lost that I couldn't get back.
But you don't get many chances to practice disaster recovery on real failed hardware where the stakes are low. So I treated it like the real thing. Get every byte back, never write to the original drives, prove it all with checksums. Three days later all 30 disk volumes were recovered and verified, and I learned a ton. I did it with Claude doing a lot of the analysis and scripting, and a few of the best saves came from it checking something before trusting it.
#What was on it
A Dell R740xd running XCP-ng, eight consumer 2 TB QLC SSDs in one RAID 5 on a PERC controller. On that one array: the hypervisor itself, the ERP dev server the developers use, a second Dev ERP web server, a new 1.75 TB SQL Server that wasn't in use yet, a Windows template, some test and staging machines, an IT server, and a secondary domain controller. The DC and the IT server were in the nightly backup. The rest was dev and test, rebuildable, just annoying to rebuild.
Nothing users depend on went down. Production ERP runs on other hardware, and the main domain controller lives on a different host and holds all the FSMO roles, so Active Directory never blinked.
#About the server
This was a used R740xd I picked up off eBay a few years back. When I bought it, and still today, it was meant to be a play box. Something I could run tests on, experiment with, and nuke and rebuild if needed. That's why it got stacked with consumer SSDs that were $150-$200 at the time instead of server SSDs at $1k each. The thing is, other than the SSDs the server is pretty solid. Just to give you an idea, it has 2 × Intel Xeon Gold 5222 @ 3.80 GHz and 256 GB of RAM with a PERC H730P RAID controller, and with the SSDs I had 13 TB of fast storage. That's not bad for a box meant for experimenting and setting up dev servers.
#Why it happened
None of this is exotic, it just stacked up.
Consumer SSDs don't have bounded error recovery. When an enterprise drive struggles with a read it gives up fast and lets the controller rebuild from parity. A consumer drive keeps retrying, sometimes for a long time, and to a RAID controller a drive that goes quiet looks dead. A rebuild is the worst load you can put on them, every surviving drive read end to end while the array has no redundancy left.
The twist is the drive that actually killed it. It passed SMART with zero errors. When I imaged it the next morning it read at 2 to 7 MB/s for about an hour, then jumped to 300 MB/s and stayed there with zero read errors for the rest of the disk. My best guess (not proven) is firmware cleanup on a DRAM-less drive after losing power. If that's right it never really failed. It just went quiet longer than the controller would wait, at the one moment the array couldn't afford it.
#The buttons I didn't press
The controller screens offer clear foreign config, discard preserved cache, initialize, rebuild. Any of those at the wrong moment turns a failed array into a wiped one. Everything I read that afternoon said the same thing: image before you write. So that was the rule for the whole week. The original drives are evidence. Copy them, then only work on the copies, and the dead drives never go back in the server.
I did try one controlled thing Tuesday, importing the array from the drive that had just dropped. The import didn't stick, that drive reset again 38 seconds after going in, and after a couple of power cycles the RAID controller stopped showing up at all. I called it for the day with the server off.
Wednesday morning I pulled both power cords for eight minutes, a full drain, and the controller came back exactly how I'd left it. Array failed, preserved cache still there, nothing cleared. That was the last time I asked it to do anything. Everything after that happened on a different machine, working from copies.
Next post: getting 30 volumes off a dead RAID 5 without guessing a single setting.