The safeguard that wouldn't have fired (part 3)
One of the VMs on the dead array was a secondary domain controller. In part 1 I said Active Directory never blinked, and that's true. The main DC lives on another host and holds all the FSMO roles, so nobody could tell anything had happened.
The easy fix was to build a fresh VM, promote it, and let it replicate. That's a clean, boring afternoon. But the DC was in the nightly backup from the night before, and restoring a domain controller from a VM backup is one of those things everyone has an opinion on and not many people have actually done. So same deal as the rest of the week. Treat it like it matters and do it the careful way.
#Why restoring a DC is scary
Every DC numbers its changes and keeps track of how far it has seen its partners' changes. Bring back a DC from an old image and it starts reusing numbers its partners already saw, so they skip those changes. No error, no alert. You just end up with DCs that slowly disagree, a password that works on one and not the other, deleted accounts coming back. That's USN rollback, and it's why the old advice was never snapshot a DC.
Windows Server 2012 fixed that with VM-Generation ID. The hypervisor hands the VM an ID, and the DC remembers the last one it saw. If it boots and the ID changed, it knows it's a restored copy and protects itself: new replication identity, throws away its pool of IDs for new objects, and re-syncs AD and SYSVOL from its partners instead of pushing its old data out. XCP-ng supports it. On paper it should have been fine.
#The research said "should"
Before touching anything I went through the source code. The hypervisor gives a new generation ID to a VM that gets copied, cloned, snapshotted or imported, and keeps it on a migration since that's the same machine. The backup restore rebuilds the VM from the backup snapshot, which already had a new ID. So the restored DC should see a new ID and the safeguard should fire.
There was also one forum thread from 2019 where someone restored DCs from the same backup tool and got USN rollback anyway. Never resolved.
"Should" plus one person it didn't work for isn't good enough for the thing everyone logs into. So the rule was the restored DC gets no network until its own event log proves the safeguard ran. That's event 2170.
#The check before the check
I restored the DC and left it powered off. Before booting it I wanted to know one more thing: what generation ID did it actually get?
It was the same as the original. Not the snapshot's ID, the original VM's. This version of the backup tool carried the old ID over on restore. If I'd booted it, Windows would have seen no change, the safeguard would never have fired, and I'd have had a DC with day-old data and a replication identity its partner already trusted. That 2019 forum thread made a lot more sense after that.
The fix was one step. A copied VM gets a new generation ID, so I copied the restored VM, checked the copy's ID was different, and set the original aside, renamed so nobody would boot it.
#The gate
I took the network card off the copy and booted it by itself. Signed in and checked the Directory Service log. Event 2170, a generation ID change was detected, and the old value it had cached was exactly the ID the plain restore would have handed it. Right behind it, event 1109, new replication identity. That's the whole story in two log entries. Then I plugged it in.
Within a few minutes it said it was caught up with the main DC, SYSVOL re-synced from the healthy partner, and replication came up clean. One direction showed old failures for a bit until the main DC's retry timer came back around. Don't try to fix that, it clears on its own.
#The surprise after that
It was answering on its old address, and on a second one. Windows was showing two network adapters with the same MAC, the Xen network card with the static IP and an emulated Intel card that's normally hidden once the Xen drivers load. The Intel one had grabbed a DHCP address and the DC had registered itself in DNS under it, so clients could have been sent to the wrong address. Disabled the Intel adapter, Windows cleaned up the extra DNS records by itself, done.
It also landed on a different physical host from the main DC this time, which is the point. One host dying should never take out more than one DC.
By that evening the steps were a written runbook, so next time it's a checklist instead of a research project.
#What I'd tell anyone doing this
- Check the safeguard's input, don't assume it. The docs, the source and a reasonable reading all said the ID would change. One look at the VM said otherwise.
- With Xen Orchestra: restore, copy, and only ever boot the copy.
- Make event 2170 a hard gate. No 2170, no network.
- Watch for the emulated NIC and turn it off before it gets into DNS.
Next post: getting everything back online without taking down the office.