Make It Simple Systemadmin: USN Rollback

USN Rollback

Summary:

This article describes condition that occurs when a domain controller that is running Active Directory database that has been incorrectly restored or copied into place and how to detect and recover from USN Rollback in windows server 2003, windows server 2008 and windows server 2008 R2.

When a USN rollback occurs, modification to objects and attributes that occur on one domain controller do not replicate to other domain controllers in the forest. Because Replication partners believe that they have an up-to-date copy of the Active Directory database monitoring and troubleshooting tools such as repadmin.exe do not report any replication errors.

After hotfix 875495 or windows server 2003 SP1 is installed, a Microsoft windows server 2003 domain controller logs Directory Service Event 2095 when it encounters a USN rollback. The hotfix included in the windows server 2003 SP1 as well as in windows server 2008 and 2008 R2.

Introduction:

The article that we are going to discuss.

1) Supported methods to backup Active Directory.

2) Typical behaviour that occurs when you restore an Active Directory Aware System State Backup.

3) Active Directory Database without restoring the System State can lead to a USN rollback.

4) How Replication is affected when Microsoft 2003 based domain controller experiences a USN Rollback.

5) Recover Domain Controller after it experience a USN Rollback.

6) Enhancements in Hotfix 875495, to detect USN Rollback, and to quarantine affected Domain Controllers.

Supported methods to use Rollback the contents of Active Directory:

1) Use Active Directory Aware Backup and Restoration utility that uses Microsoft provided and Microsoft tested API. These API is Non-Authoritative - Authoritative Restore a System State Backup. The backup should originate from the same Operating System.

2) Use an Active Directory Aware Backup and restoration utility that uses Microsoft volume Shadow Copy Service APIs.

3) Evaluate whether valid system state backups exist for this Domain Controller.

Typical Behavior that occurs when you restore an Active Directory aware system state backup:

When Active Directory is restored on domain controller by using Microsoft designed & Tested Active Directory Aware Backup the Invocation ID is correctly RESET on the restored domain controller. Domain controller in the forest receive notification of the Invocation reset.

Source Domain Controllers use USNs to determine what changes have already been received by the destination domain controller that is requesting changes.

Destination Domain Controllers use USN to determine what changes should be requested from source domain controllers.

The Invocation IDs identifies the version of the Active Directory database that is running on a given domain controller.

Software and methodologies that cause USN Rollbacks:

1) Active Directory Database restoration by using Norton ghost

2) Previously saved virtual Hardisk image of domain controller.

The effects of USN Rollback:

When USN Rollback occur, modifications to object and attributes are not inbound replicated by destination domain controllers that have previously seen the USN.

The destination domain controller believe they are Up-to-date, no replication errors are reported in directory service event logs or by monitoring and diagnostic tools, USN rollback may affect the replication of any object or attribute in any partition.

Detecting a USN Rollback on Domain Controller that are running windows server: -

Detecting USN Rollback is quiet difficult why because there is no log flagged into Event and Replication.

One way to check a USN rollback is to use the windows server version of Repadmin to run

Repadmin /showutdvec command

This version of Repadmin displays the up-to-date vector of USN for all Domain Controllers that replicate common naming context.

Note:

A correctly restored domain controller reset its local Invocation ID attribute when it restarts into AD after its system state is restored. By using a supported backup and restore method.

The following example shows the output of the Repadmin /showutdvec on DC1 & DC2 in Test.Local domain

Repadmin /showutdvec dc1 dc=test,dc=local

Repadmin /showutdvec dc2 dc=test,dc=local

The output from DC1 shows a local USN of 12426, DC2 has inbound replicated USN 82148 and will ignore the Active directory updates that correspond to the next 69722 USN numbers from the originating DC1.

Detecting a USN rollback on windows server domain controller that has the 875495 hotfix installed.

Windows server 2003 domain controller that has the 875495 hotfix functionality installed it will log event 2095 when source domain controller sends previously acknowledge USN number to a destination domain controller without a corresponding change in the Invocation ID.

To prevent unique originated updates to Active Directory from being created on the incorrectly restored domain controller the Net Logon Service is paused.

When the Net Logon service is paused, user and computer accounts cannot change the password on a domain controller that will not outbound replicate such changes.

On a domain controller that has the 875495 hotfix functionality installed,

Event messages that resemble the following are recorded if the following condition are true.

· A Source domain controller sends a previously acknowledged USN number to a destination domain controller.

· There is no corresponding change in the invocation id.

Message 1:

Event Type: Error

Event Source: NTDS Replication

Event Category: Replication

Event ID: 2095

Recovering from a USN rollback:

1) Remove the domain controller from the domain.

2) Restore the System State of a good backup.