Let the array describe itself (part 2)
In part 1 a RAID 5 on a non-production hypervisor lost two drives and I decided to treat it as a real disaster recovery drill instead of wiping it. This is the part where the data comes back.
#The recovery box
I set up a Linux box as a recovery station. Before any drive from the array got plugged in, it got a udev rule that sets those drives read-only in the kernel the second they attach, RAID and LVM auto-assembly turned off, and the disk monitoring daemon off, so nothing but the imager touched them.
The slow drive went first, since its data existed nowhere else. ddrescue read the whole 2 TB in 2 hours 55 minutes with zero errors. I almost planned the whole recovery around a "six day" estimate measured on the first half a percent of the disk. Re-measure before you re-plan.
To read a RAID 5 in software you need the exact geometry, and if any of it is wrong you get garbage that looks fine. I didn't have to guess. The controller writes its config in a standard format at the end of every drive, and mdadm decoded it: drive count, slot order, chunk size. The partition table cut the parity layout down to two options, and LVM picked the winner, because LVM checksums its own metadata and it only came out valid one way. That also proved the rebuilt missing drive was coming out right.
The half-rebuilt hot spare turned out to be useful too. Over the part it had finished I had all eight slots, so I could check parity directly instead of trusting it. 4,704 out of 4,704 sampled chunks matched.
For the map of which disk belonged to which VM, I rebuilt the hypervisor's own root partition off the dead array and pulled its database out of it. Last written four minutes before the array failed. Every VM, every disk, every setting.
A second script walked each volume through the reader and wrote out a normal VHD file. Every one got its VHD checksums checked, was made immutable on disk, and got a SHA-256 fingerprint next to it.
The first one out was the second Dev ERP web server, 80 GB, in about ten minutes. Mounted read-only, the ERP install was there and the newest event log write was nine minutes before the array died. I imported a copy onto a scratch VM on a different host with no network card, and Wednesday at 1:39 PM it booted to the Windows login screen. A disk rebuilt in software from a dead hardware RAID, booting on another server. By 3:37 AM Thursday all 30 volumes were out and verified.
#Two things I'd rather not admit
The read-only rule didn't fire. One drive sat writable for three minutes before I caught it, because the rule ran before the system rule that sets the serial number it was matching on. I fixed the order, tested it with udevadm, and proved nothing had been written by hashing the first and last 64 MB of every drive against copies from that morning. Test the safeguard, don't just install it.
And installing the SMART tools started smartd, which poked the slow drive mid-image and caused a USB reset. On a recovery box nothing but the imager should touch the source.
#The shelf copy
Thursday morning everything started copying to a brand new 14 TB drive: all 30 volumes, the slow drive's raw image, the tools, and a README with an inventory and step by step restore notes. When it finishes, every file gets re-hashed on the new drive and compared to its fingerprint, then it gets unplugged and put on a shelf. The original drives stay untouched until that check passes.
Next post: the domain controller, and the safeguard that wouldn't have fired.