Back online without taking down the office (part 4)
By the end of part 3 every volume was off the dead array, checksummed, and the domain controller was back. That left the boring part, which turned out to have its own puzzles: getting the rest of the VMs running again on other hosts, while the office kept working on the same network.
#Don't take down the office to save a server
The recovered VMs went to two lab hypervisors. The problem is that a few terabytes moving across the network during the day will happily eat the links everyone else is using, and the nightly backups ride the same path. So I made a rule for myself up front: every transfer gets throttled. 30 MB/s during business hours, 60 MB/s after. Never full link speed. Nothing on this box was worth a slow afternoon for everybody else.
Every transfer landed as a temporary file, got its SHA-256 checked on both ends, got a VHD integrity check, and only then got renamed into place. Same rule as the rest of the week. Don't trust it until it proves it.
The Dev ERP web server went first. Hashes matched on both ends, and Thursday at 3:24 PM it booted on the new host in about two and a half minutes with every app pool running.
The IT server I restored from its nightly backup instead of the crash-time copy. The backup was 17.5 hours older, but it was simple and proven, and the crash-time copy is still on the shelf if I ever need something newer. Not every recovery has to be the clever one.
#Two jobs fighting over one disk
The ERP dev server is the one the developers actually use, so it was next. Its stream started at about 7 MB/s. The cause turned out to be me. The shelf copy from part 2 was running at the same time, pulling around 130 MB/s off the same storage on the recovery box, and the disk scheduler wasn't sharing nicely. I paused the shelf copy about halfway through (4,735 of 9,574 GB) and set it to resume on its own the moment the dev server transfer finished. The dev server went straight up to its throttle limit.
#A switch from the parts bin
The two lab hosts sit in different racks and only meet over a single 1 Gb uplink, and that uplink also carries users and a couple of wireless access points. Throttling kept things polite, but it also meant a terabyte took most of a day.
So Friday morning I grabbed a dumb gigabit switch and cabled it only to spare ports on the two lab hosts and the recovery box. Its own little subnet, no gateway, no DNS, never plugged into another switch, and picked so it couldn't leak out over the VPN routes. Before setting anything up I checked that zero packets were showing up on those ports, so I knew it really was isolated. After that, host to host ran at about 116 MB/s, and none of it touched the network anyone else was on.
#The cipher was the bottleneck
The first big transfer over the new link only hit about 71 MB/s. Not the switch, not the disks. The CPU was pegged doing SSH encryption. Switching to AES-GCM, which the CPU can do in hardware, and a faster checksum got the 1.1 TB dev server across at about 102 MB/s. Checksums matched on both ends.
It booted Friday afternoon. Windows ran a quick disk check after the unclean shutdown, SQL Server recovered its database cleanly, and once IIS got restarted after SQL finished starting, the ERP came up. The developers had their server back.
The new 1.75 TB SQL Server was the last big one. It came across the same storage link with matching checksums on both ends and booted on its new host. It wasn't in use yet, so nobody was waiting on it, but it had days of setup in it that I didn't have to redo.
#A dashboard so nobody has to ask
With this many multi-hour transfers going, "how's it going?" becomes its own interruption. So I put up a small read-only web page that showed percent done, speed, ETA, disk load on the hosts being written to, checksum results, and the shelf copy's progress. Anyone waiting could just look.
#The shelf copy
Once the dev server finished, the shelf copy picked back up on its own and ran to the end. Every file got re-hashed on the 14 TB drive and matched its fingerprint. Then it got unplugged and put on a shelf, and only after that did the original drives stop being evidence.
#What's changing
This was a play box and it's going to stay one, but it's getting rebuilt so the next failure is boring:
- Enterprise SSDs with power-loss protection instead of consumer QLC drives, in RAID 6 with a hot spare. One bad drive is routine, two isn't fatal.
- A separate mirrored boot device for the hypervisor, so a failed data array never takes the host down with it.
- Every VM gets backed up, not just the ones that seemed important. The hand-built dev servers turned out to be the hardest things to rebuild and the ones without a backup.
#What I'd keep from the whole week
I could have wiped this box on Tuesday and been fine. Instead I got four days of real disaster recovery practice on real failed hardware with nothing on the line. Image before you write. Let the system describe itself instead of guessing. Check the safeguard before you trust it. And don't wreck everyone else's day to save a box that wasn't production.