Understanding Directory Replication

At its foundation, the replication process is simply an effort to keep the copy of the AD DS database identical on all domain controllers for a particular domain. For example, if an administrator removes a user from a group, the change is made on the domain controller that the administrator is currently logged into. For those few seconds after the change, that domain controller alone has the most current copy of the database. Eventually, though, after replication takes place, all domain controllers will have exact replicas of the database, including the change in group membership.

AD DS replicates information between domain controllers using different methods, depending on the topology of your network—in particular, how many sites you have configured within AD DS. In a single-site situation, all domain controllers in a domain will discover one another through published records in both AD DS and the DNS system for the domain. But to cut down on network traffic, not every domain controller needs to actually replicate with every other domain controller. AD DS uses a "loop" method. Take, for instance, four domain controllers—A, B, C, and D, as shown in Figure 5-34.

In this example, AD DS will replicate using two loops. Let's assume that a change was made on domain controller A. A will tell B and C that it has new information, and eventually B and C will ask A for that information. Once the information is received, both B and C will attempt to tell D about their new information. D will ask for the new information from the first domain controller that reaches it—there isn't a good way to determine whether that would be server B or C in our case—but when the second "message" telling D that it has new information arrives, server D will simply respond, acknowledging that it already has that information, and that will be the end of the transmissions because all domain controllers now have the most up-to-date information. In contrast, consider using only one loop and not two. In that case, A would tell B, B would tell C, C would tell D, and D would then tell A again. That doesn't happen. In the actual case, news is spread more quickly and network traffic is reduced, making the entire process more efficient. In fact, this entire process triggers every five minutes, and if there's new information, the process will engage. If there is no new information, the domain controllers won't transmit anything; however, if 60 minutes pass without any new information, each domain controller will send a message to its partners, making sure there's no new information.

You might wonder how this loop is designed. The Knowledge Consistency Checker, or KCC, wakes up approximately every 15 minutes and tries to detect changes in its idea of how many domain controllers there are and where they're located. The KCC will look at any changes that have occurred—you might have taken a domain controller offline for maintenance, for example, or even added a new domain controller for load control purposes. Then it adjusts the loop for best performance.

In larger sites, the KCC might find it necessary to add more than two replication partners for each domain controller, or it might do so for traffic control purposes. In still larger sites, even those with only two replication partners per domain controller, it can take more than three hops to transmit replication information completely. The KCC looks for this situation and, if it detects it, simply adds more links between domain controllers, changing the simple "loop" structure into more of a "mesh" structure.

For replication to function properly, it is crucial for all domain controllers in a domain and/or forest to be in sync in terms of the current time. The reason points to Kerberos, the underlying authentication scheme for the entire AD DS: if any domain controller is more than five minutes out of synchronization, authentication will fail.

The Windows Time Service is the tool Microsoft provides to consistently keep your entire domain or forest at the same moment in time. Windows Time Service offers a hierarchy for members of AD DS domains and forests, with the machine holding the PDC emulator operations master role as the "big kahuna" of sorts, holding the trusted time. The trusted time at the very top level does not need to be synchronized from anywhere—synchronization matters only within the domain, as all members must think it is the same time, regardless of whether that time is the actual time. In other words, everyone has to be the same, but it doesn't matter if everyone is wrong.

From the bottom up, the rest of the hierarchy looks something like this:

You can synchronize the domain controller at the PDC emulator operations master role in a few ways, through the command line. First, though, you must choose a time source. Microsoft provides the host time.windows.com, synchronized to the U.S. Army's Atomic Clock, which is as good a choice as any. Once you have selected a time source, run the following from the command line of the PDC emulator domain controller:

net time /setsntp:<TIMESOURCE>

Replace <TIMESOURCE> with the full DNS name of the time source you have selected. For example, if I were using time.windows.com as my time source, I'd run:

net time /setsntp:time.windows.com

Once you have set the time source for the PDC emulator domain controller, it will attempt to synchronize its time with the time source. It will try once every 45 minutes until it has successfully synchronized three times in a row. Once it has done so, it pings the time server only once every eight hours. If you want to trigger time synchronization manually, run:

w32tm /resync

Tip

The Windows Time Service requires outbound UDP port 123 to be open on your firewall for time synchronizations to occur.

Time zones also play a role. Windows operates internally at Greenwich Mean Time, and although each server can be set to a different time zone depending upon either its physical location or the location of the administrator who manages the box, within Windows itself the current time is translated to GMT. Be wary of this, and ensure that time zones are set correctly on all your servers. The object is to get Windows' internal clocks to synchronize—even though the time might seem right to the naked eye, if the time zone is set incorrectly, Windows is unforgiving when it comes to AD DS operations in that state.

Loops and meshes are just two examples of what Microsoft terms replication topologies—essentially, maps of the ways domain controllers replicate to one another. And to confuse things, there is almost always more than one replication topology existing simultaneously within any particular forest. Let's take a closer look at that.

Four types of data need to be replicated among domain controllers:

With many domain controllers in a forest, you can see where one replication topology might either not suffice or not be the most efficient way to transmit information between these selected subgroups of domain controllers. Figure 5-35 shows this scenario graphically.

The Active Directory Sites and Services console again comes to your aid if you want to try to piece together all these replication topologies for your environment. Open the console, expand the site in question in the left pane, and expand each domain controller's node. Click NTDS Settings in the left pane, and in the right pane double-click the "<automatically generated>" objects.

If you see <Enterprise Configuration> in one of the fields at the bottom of the screen indicating replicated naming contexts, it shows that that particular link replicates the schema and configuration naming contexts. In the Partially Replicated Naming Context field, if you see a server name, this indicates that your server is a GC server and is receiving updates from the GC server listed in the field. It is perfectly acceptable for this field to be empty on servers not acting as GCs.

Replication is great in and of itself, but there it has one major, inherent problem—each domain controller uses its own copy of the database and, no matter how often each copy is updated, for a few moments in time each copy is unaware of actions taken on other copies of the database around the network. How might this design situation manifest itself as a problem?

Consider a large site, with branch offices in Sydney, Boston, and Los Angeles. An employee, Robert Smith, is being transferred to the Sydney office from L.A. because of personnel reorganization. The company uses groups within AD DS, SYDUSERS and LAUSERS, for distribution list purposes and other security boundary assignments. On Robert's last day in the L.A. office, his manager changes Robert's group membership, moving him from LAUSERS to SYDUSERS in anticipation of his transfer. The Los Angeles domain controller notes this change and creates a record looking roughly like this:

Object: LAUSERS
Change: Remove RSMITH
Version: 1
Timestamp: 30 June 2004 5:30:01 PM GMT
Object: SYDUSERS
Change: Add RSMITH
Version: 1
Timestamp: 30 June 2004 5:30:02 PM GMT

Look closely at these records. They denote changes to attributes of objects—in this case, the member list is an attribute of a particular group object—not changes to the entire object. This is important for network traffic reduction reasons; if the LAUSERS group comprised 2,000 members, it's good to transmit only the removal of RSMITH and not the entire membership list. Also, note the version numbers: the field is very simple and is designed to be used whenever domain controllers update an attribute for a particular object. Each time a change is made to a particular object attribute, the numeral in the version number field is incremented by 1. So, one object can have many version numbers, each representing the attributes of that object.

With that background out of the way, let's return to our fictional situation. Perhaps there was a miscommunication between Robert's old manager in Los Angeles and his new manager in Sydney, and each incorrectly thought she was supposed to make the change in group membership with AD DS. So, at almost exactly the same time (we'll ignore time zone differences for the purposes of this demonstration), Robert's new manager makes the previously described change, which is recorded on the Sydney domain controller as follows:

Object: LAUSERS
Change: Remove RSMITH
Version: 1
Timestamp: 30 June 2004 5:32:08 PM GMT
Object: SYDUSERS
Change: Add RSMITH
Version: 1
Timestamp: 30 June 2004 5:32:10 PM PT

There are two things to note about this record: one is the closeness of the timestamps. They seem to indicate that the L.A. and Sydney domain controllers haven't replicated yet. The second item of interest is the version number field in each record, which does not appear to have been incremented. The reason for this is simple: version numbers are incremented on the local domain controller. If a domain controller doesn't know about any changes to an attribute, there is no need to further increment the version number on that record. Because L.A. and Sydney haven't passed changes between each other yet, the Sydney domain controller doesn't know that a similar change has processed on the L.A. domain controller and therefore doesn't know to increment the version number field from 1 to 2.

This situation might seem harmless because although the changes are different only in time, the net effect is the same; on both domain controllers, RSMITH is a member of the correct group and not a member of the former group. But in reality there are two changes. Which one is accepted by AD DS? And to ask a more specific question, when both the L.A. and Sydney domain controllers replicate to their partner, the Boston domain controller, which change will Boston accept?

When changes to the same object compete, the tie is broken two ways:

In our case, the change made on the Sydney domain controller would be the one formally accepted in AD DS, and the L.A. manager's modification, although its intent was the same, would be rejected because it was made at 5:30 p.m. and not at 5:32 p.m.

Version numbers have siblings, called update sequence numbers (USNs), which measure the increments of every change to every attribute of every object within AD DS. That is to say, whenever any change is made within AD DS, a domain controller increments its current USN by 1. For example, if you have a pristine domain controller and you add a user with an initial password (update 1), change his password (update 2), add another user (update 3), create a new group (update 4), and put the first user in that group (update 5), the USN for that domain controller would be 5. Keep in mind that version numbers coexist with USNs; let's look at the information in Table 5-2 to see how USNs and version numbers interact in the preceding example. (Note that the table assumes we're talking about a domain with only one domain controller; it gets a bit more complicated when you add more domain controllers, and I'll discuss that later in this section.)

From this table, you can glean that version numbers are incremented only when attributes change. We changed the password for our first user, so the version number for that attribute became 2, and we added that user to our newly created group, so the version number for the attribute containing the member list for that group increased to 2. But all the while, the USNs were incrementing because USNs measure every individual change. Our USNs went from 0 to 4 because (a) USNs start at 0, not 1, and (b) 5 individual changes were made.

Note that earlier I said this scenario revolved around a single domain controller. Let's change that now: if we were to add a domain controller to this example domain, the domain controllers would attempt to replicate with each other. Let's assume that once a domain controller is added, our two accounts and one group are transmitted immediately. The version numbers and USNs, as seen on the new domain controller, would shape up as shown in Table 5-3.

We learn two things from this table:

USNs really act as "signatures" showing the timeliness of information in each domain controller's copy of the directory, and essentially guide a domain controller's replication partners as to exactly how much updating is needed to fully replicate. Each domain controller tells its replication partners the USNs it has assigned to each piece of data, and the partner domain controllers keep track of that information. The partners then know that the last piece of data they received—for example, from domain controller X—had a USN of 6093, and they can then tell domain controller X upon the next replication to start transmitting data with a USN of 6094 (one number higher than the last USN) or more. There is no need to send USNs 1-6093 again, as they already possess that data. If the USNs haven't changed on domain controllers during the regular five-minute breaks from replication, domain controllers assume that no new information is available, and they go back to "sleep" for another five minutes. On the other hand, if domain controller X's replication partners contact domain controller X and ask for its highest USN, and it replies with 7000, the partners know they need the last six pieces of information, and those will then be replicated. Then the partners would make a note that domain controller X's highest USN is now 7000, and everyone is secure in the knowledge that they have the most current directory possible for at least the next five minutes.

So, let's return to the example and see where we are with version numbers and USNs. Table 5-4 sums that up.

Now consider this scenario: an administrator changes the password for User 2, and that administrator is currently using domain controller 1. That change would be assigned USN 5 because it's the sixth change that domain controller has seen. Five minutes later, replication is initiated, and domain controller 2 queries domain controller 1 for its highest USN, which it tells domain controller 2 is 5. Because domain controller 2 thinks the highest USN for domain controller 1 is 4, it knows it has missed a change, so it asks for the new information. The new password is pushed along, and the change on domain controller 2 is assigned a USN of 3 (it is a unique change, after all). Then domain controller 2 makes a note of domain controller 1's new highest USN, domain controller 2 is up-to-date, and everyone is happy.

They're happy, that is, until a few minutes later, when domain controller 1 asks domain controller 2 what its highest USN is. Domain controller 2 will faithfully reply that it is 3, and domain controller 1 will know that figure is higher than its recorded high USN for domain controller 2 (which is 2, as shown in Table 5-4). However, that change is the original change pushed through from domain controller 1 to domain controller 2. The domain controllers don't know that, however, just from looking at USNs, so they push through the replication, and domain controller 1's highest USN now becomes 6 because of this "change." Five minutes later, the entire process starts again, with this one change being propagated over and over and over again, in an infinite loop.

Microsoft identified this problem and introduced two other values to the mix, called originating USNs and up-to-date vectors, specifically to prevent this situation from occurring. Originating USNs simply keep track of the domain controller from which a change was initially transmitted, and the USN on that domain controller. So, when we first introduced the brand-new domain controller into our example domain and a copy of the directory was first replicated, more information was transmitted than I discussed earlier. Table 5-5 contains a more detailed representation of the results of that replication than Table 5-4 does because it includes originating USNs.

In essence, originating USNs tell all domain controllers where information first came from and what USN that first domain controller assigned to each piece of data. But just as domain controllers keep track of the highest USNs for their replication partners, they also keep track of the highest originating USN they have ever come across from any and all domain controllers. This table of maximum originating USNs is known as the up-to-date vectors. Let's look at this more closely.

Our situation now is shown in Table 5-6.

I just formulated the up-to-date vectors shown in Table 5-6; all they represent is the latest originating USN that each domain controller knows from the other domain controller. Now, flip back a couple of pages and refresh yourself on the scenario that previously would have created an infinite loop: a change of User 2's password, made by an administrator attached to domain controller 1. Domain controller 1 gives this change a USN of 5, and consequently domain controller 1 updates its table of up-to-date vectors with the highest originating USN that it knows from itself—so, it changes from 1 (our arbitrary first number) to 5.

Replication is initiated once again, and domain controller 2 asks domain controller 1 if it has any new information higher than USN 4, which it knows is its partner's highest USN, and whether the originating USNs are higher than 1 for domain controller 1, and 1 for domain controller 2. Domain controller 1 checks its copy of the directory and finds the password change, and then sees that it originated the change itself, with an originating USN of 5. However, domain controller 2 asked for any information from domain controller 1 with an originating USN higher than 1, so now domain controller 1 knows that domain controller 2 has no idea of this new information and it passes it along. Domain controller 2 records the change and assigns a USN of 3, and then makes a note that its partner's highest USN is 5 and the highest originating USN it has seen from domain controller 1 is 5. Our values, after that process, are shown in Table 5-7.

Let's go a bit further. Replication kicks off again, and this time domain controller 1 contacts domain controller 2 and asks whether it has any new information higher than USN 2, which it knows is its partner's highest USN, and whether the originating USNs are higher than 5 for domain controller 1, and 2 for domain controller 2. Domain controller 2 checks its copy of the directory and sees that it has a change to which it assigned a USN of 3, but it also checks to see where that change came from—and it sees that the change came from domain controller 1 and that domain controller 1 assigned a USN of 5 to it. Domain controller 2 decides that even though the change was new to it, domain controller 1 clearly already knows about it and therefore doesn't need that change replicated. Domain controller 2 tells domain controller 1 about its currently highest USN (3), and domain controller 1 makes a note of that. What does this entire process accomplish? It ensures that a change is replicated only between partners, because each partner can figure out who knows about what changes and when they were made by looking at USNs and up-to-date vectors. So, now everyone is happy—really, this time—as shown in Table 5-8.

In summary, domain controllers use this up-to-date vector table and the originating USN data fundamentally to construct a more specific replication request. So, instead of simply asking a replication partner for any data higher than the highest USN a requestor knows, it asks for any data higher than the highest USN it knows is also higher than the ones for each domain controller in its up-to-date vector table.

Replication Administrator, or REPADMIN, is a command-line utility that can control a lot of aspects and behaviors of AD DS replication. In case you're wondering why you've never seen the utility, REPADMIN is part of the Windows Server 2008 Resource Kit Tools—not the standard kit—and you can find it on the distribution CD within the SUPPTOOLS.MSI installer file in the \SUPPORT\TOOLS folder.

Although AD DS uses loops and meshes to create and manage replication topologies within a particular site, using that many links across an expensive WAN connection can cost you dearly as well as take a lot of time. For that reason, when AD DS replicates between sites, it uses a minimal spanning tree—in other words, a tree with as few branches as possible to span the link between multiple sites.

Let's use an example environment, with two servers in a site called MAIN (representing the headquarters in Charlotte) and a single domain controller in another site, called WEST (located in San Francisco). Recall that the KCC facility creates replication topologies within sites automatically—you, the administrator, do not have to intervene. Replication between sites isn't as simple; AD DS needs to know several things about your individual sites before it can figure out how to replicate traffic among them.

By creating site links, you give AD DS three key pieces of information it must have before it can determine the most efficient way to force replication traffic across your sites:

Let's discuss the third bit of information first: AD DS will allow you to create links based over IP (using RPC calls) or via SMTP for less reliable or less secure connections. Unfortunately, SMTP-based site links are extremely limited in functionality. For one, SMTP links will only transfer updates to the forest schema naming context and configuration naming context; it will not perform cross-site domain controller information updates. Also, you need a secure mail server, hardened against outside interception using encryption and certificates, to transfer even that bit of information. For these reasons, the vast majority of site links you create will be IP-based links.

Returning to our example, let's create a site link between MAIN and WEST. To do so, follow these steps:

To further configure the site link, right-click the new link in the IP folder of the left pane of Active Directory Sites and Services. Choose Properties, and the screen in Figure 5-37 will appear.

This screen contains two critical items. First, the Cost field allows you to determine a cost quotient—in essence, an index of the expense of using a connection—for each site link you create. If you have more than one site link, AD DS will choose the lowest-cost link to perform the replication. Unfortunately, Microsoft doesn't give you much guidance on how to arrive at your cost quotient figure; I recommend taking into account the base link cost, rates for prime and overnight periods, traffic limits, and link availability. Second, the "Replicate every" box allows you to specify how often AD DS will attempt to initiate replication over this specific site link. You can set the shortest replication interval to 15 minutes, and there is no functional maximum value (although all AD DS sites must replicate at least once every 60 days). Click the Change Schedule button to see the screen depicted in Figure 5-38.

Use the mouse to select the hours during which the link will not be available; AD DS will use this information and not even attempt replication during that time period. Click OK to exit that dialog, and then click OK to finish configuring the link.

Once you have specified the information for the site, the Sites and Services equivalent of the KCC, called the Inter-Site Topology Generator (ISTG), will begin developing the minimal spanning tree needed to pass replication traffic among sites.

And that's a basic but thorough treatment of AD DS replication.

Think back to the days of Windows NT 4.0, when there was one king of the hill, the primary domain controller (PDC), and then any number of subservient princes below that king on the same hill—the backup domain controllers, or BDCs. It was easy to see the flow of information—changes were made to the master copy of the domain security information on the PDC, and from there it flowed outward, unidirectionally, to the BDCs. When AD DS came around, however, this distinction was eliminated, and in practice a domain controller became equal to any other domain controller, without any designation of primary, backup, or so on. (Well, in actuality, some DCs are a little more equal than others when you factor operations master roles into the equation, but that's not relevant to this discussion.) While this new design increased the fault tolerance and distributed deployment capabilities of the operating system, it's somewhat of an issue if a domain controller anywhere on the network pushes corrupt or otherwise incorrect data to other DCs; how would you prevent that? In branch office scenarios, this issue is particularly serious, since the designated administrator in a branch office needs Domain Admin credentials to administer the DC in her office; this actually gives her the right to administer any DC, not just the one she's responsible for looking after. It's not the best security situation.

While this equality of domain controllers is still the case in Windows Server 2008's AD DS implementation, there is now the concept of a read-only domain controller. A read-only domain controller (RODC) is just that—it receives information replicated to it from full domain controllers, but it doesn't permit any changes to be made to its own copy of the directory database, and thus no information can be replicated back to the full DCs in the domain of which it's a member. This is a great win for branch offices whose companies are large enough to have a comprehensive AD DS structure. Now, you don't have to deploy a full-blown domain controller to your remote locations—you can simply place a RODC there. The benefits are significant and include the following:

During the Active Directory Domain Services Installation Wizard, when you're first deploying a RODC, DCPROMO recommends that you install a DNS server locally on the RODC. That server is fed zones from the AD DS infrastructure, and the process will add the RODC's local IP address to the RODC's local DNS client properties, so queries will be directed automatically to the AD-integrated zone on the machine itself.

However, for maximum fault tolerance, if there is only one DNS server and RODC (for instance, if the two services are running on the same machine) at a branch office, Microsoft recommends using the options in your DHCP scope—assuming you are using dynamic IP addresses at your branch office—to include not only the local RODC-based DNS server, but also a DNS server at the main office, where your other AD servers are centrally located. Make sure the local RODC-based DNS server is first on the list, so that only queries that fail to get a response from your local server are directed over the wire to your hub site.

If you have a larger branch office contingent, and you have multiple RODCs deployed at one branch office, you can install a DNS server instance on each RODC. Remember that since RODCs are indeed read-only, any dynamic update requests from your clients have to go directly to a writeable domain controller at your central office. The DNS server instances on RODCs will not attempt to replicate with one another; rather, they will receive all updates directly from the central AD DS infrastructure, so there may indeed be certain times wherein one local RODC has received an updated DNS entry for a client, but another has not. This issue resolves itself (no pun intended) within a replication cycle, but to avoid it, make sure the clients themselves have identical DNS server lists—use DHCP options to assist with this—and only install one DNS server instance per site.