One of those lesser known Terminal Server (Citrix) concepts which is often overlooked.
What is it and what does it do?! I learned about the shadow key in more detail about 6 years ago and since then I noticed that not all, or very few to be honest, system admins know about its existence, or that they do know, but don’t know how to treat it… does that make any sense? First I’ll briefly try and explain the use, advantages and disadvantages. Also, there have been made a few small changes to the overall architecture with the introduction of 64 bit systems and Server 2008R2 which I’ll address later on
We probably all know, or should know, about the ‘change user \install’ command or the ‘add\remove programs’ feature used on Terminal Server systems to install applications for multiple user access. There are basically two modes a Terminal Server can be in: Install mode, as explained above, or execute mode in which users connect to the Terminal Server to launch applications, desktops and so on… This is the mode the Terminal Server is in 99% of the time. Both these modes have a few underlying processes which not everyone is aware of.
When an application gets installed, in most cases it needs to create and run certain registry keys during the installation process. These ‘keys’ hold initial application configuration values and are written to the HKCU\Software hive of the user currently logged on installing the application, which will be the Administrator configuring the Terminal Server for production use. At a later stage this could create a problem, as the users who connect to the Terminal Server to start these applications will also need this initial configuration information, and if that info is only written to the HKCU\Software hive of the Administrator who installed the application, how does this info get copied to the HKCU\Software hive of the users logging on? This is where the shadow key comes in. let’s start with a short general overview first.
As mentioned earlier, when an application gets installed on a Terminal server, one way or the other, install mode kicks in, this is what happens:
During the installation process, when an application writes registry information to the HKCU\Software hive, these writes also get copied to the HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\CurrentVersion\Terminal Server\Install\Software key, the so called shadow key! It does this separately for every application that gets installed and every key that gets modified. This is normal behavior in 32 bit environments like we were used to a few years ago. I’m using this as an example just to make clear how the process works, more on 64 bit later on (some small changes).
Each time a write to the shadow key takes place, as mentioned above, the system updates a timestamp on the server called ‘LatestRegistryKey’ which can be found under: HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Terminal Server\Install\IniFile Times. The value this record holds are the seconds since January 1st 1970.
In fact, every key has its own time stamp that shows the last time it was updated and or modified. So for every write on every key to the shadow key location a separate time stamp on each individual key gets updated.
When the installation is done, and assuming all the other configuration stuff is finished as well, the server is put back into execute mode, manually or automatically, depends on the installation method used. Next, users can start connection to the Terminal Server and launch the application. There now needs to be some kind of way to get the shadow key information into the users HKCU\Software registry key, here goes…
User logon process:
What happens during logon: A process called userinit.exe runs, in Terminal Server environments this executable compares the hidden timestamp of the servers ‘IniFile Times’ key to a timestamp in the users HKCU\Software hive that holds the last synchronization time: HKCU\Software\Microsoft\Windows NT\CurrentVersion\Terminal Server\LastUserIniSyncTime.
This way the system can figure out which one is newer, the settings in the users HKCU\Software hive or the settings on in servers HKLM IniFile Times file. If the timestamp in IniFile Times key is newer the systems knows that new software must be installed or that an update has taken place. If the users HKCU\Software hive information is newer then nothing happens.
If the system detects that the IniFile Times key is newer then userinit.exe will enumerate all the keys from the shadow key area and compare each time stamp to the corresponding key in the users HKCU\Software hive. Remember that every registry key has its own time stamp. Every time the system finds that a shadow key is newer it will delete the corresponding users HKCU\Software key and replace it with the most recent one. If it can’t find an corresponding key in the users HKCU\Software hive, for example the first time you log on and your current profile is still fresh, or an new application is installed, it will copy over all the key’s it can’t find and will compare them the next time you log on, the way it all starts to begin with.
This is how all the registry information that gets written into the Administrator HKCU\Software hive during installation of one or more applications (in this example) when setting up the Terminal Server, will eventually end up in de user profile logging on to the system. I won’t go into any further detail regarding user profiles, the different types, the way they are loaded etc… I assume we all know the basics ;-)
But what if…
So, no worries right? Hold on… What happens if you add a few new servers to your farm for extra capacity with the same applications installed and published? Let’s say the original farm was set up and configured a year ago. These extra servers are all ‘new’ so the applications installed will also be ‘new’ and will generate ‘new’ timestamps. Now imagine the same process as explained above repeating itself with users logging on to those ‘new’ servers accessing the same applications as installed on the ‘old’ servers. Timestamps will again be compared and since these servers, and thus the applications installed on them, are new(er) so will be the timestamps. This will result in deletion of the users application specific configuration key’s which were obtained a year ago when the original farm was set up.
Something needs to be done. This isn’t a process that can be disabled in any way. When adding new servers, you need to make sure that the shadow key timestamps on these servers are older than the timestamps in the users HKCU\Software hive, which they got from the ‘old(er)’ farm servers a year ago (in this example). There are three ways this can done:
1. Use Sysprep and “image” new servers. Inherit the registry timestamps from original build.
2. Write to HKCU\Software in Install mode with the system clock set in the past.
3. Remove / delete all shadow keys that could potentially overwrite user preferences
Each one has its own drawbacks. I won’t go into any details regarding the first two, let’s just say that number three is the preferred one to work with. This does mean that you’ll have to find an alternative way to get the application specific configuration information into the users HKCU\Software hive when they log on. So monitor what happens to the HKCU\Software hive when installing the application and use some sort of login script, user preferences, app sense or whatever, to load the configuration information into the users HKCU\Software hive during log on. Ok, that the general idea on how the shadow key works.
As mentioned earlier there are a few exceptions regarding 64 bit systems and windows Server 2008R2. The idea and infrastructure behind the shadow key is almost identical, the process itself is just slightly off. Let’s first start with 64 bit systems.
It is possible to install and run 32 bit applications on 64 bit systems, as we all know. This is made possible by the WoW64, or, Windows-on-Windows 64 layer. When a 32 bit application gets installed on a 64 bit system the WoW64 layer redirects the shadow key location from: HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Terminal Server\Install to: HKLM\SOFTWARE\Wow6432Node\Microsoft\Windows NT\CurrentVersion\Terminal Server\Install. This means that when the Administrator installs an 32 bit application on an 64 bit system the writes to the Administrators HKCU\Software hive are echoed \ copied to the HKLM\SOFTWARE\Wow6432Node\Microsoft\Windows NT\CurrentVersion\Terminal Server\Install shadow key location. Same process as before, different (shadow key) location.
When the Administrator installs a 64 bit application on the same system the HKCU\Software writes are echoed to the 64 bit shadow key location, which is: HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Terminal Server\Install. In fact the whole shadow key principle is exactly the same like I explained on a 32 bit environment, only now there are two different shadow key locations, provided by the WoW64 layer.
When an application gets installed all the modifications taking place in the HKCU\software hive (of the logged on user performing the installation) are copied to the 32 or 64 bit shadow key location within the registry. When users log on to the system (the normal execute mode) the content of both keys gets copied to the users HKCU\Software hive. When the system encounters a conflict, because both the 32 and 64 bit shadow keys are copied and could contain the same settings, the 32 bit shadow key overrides any conflicting 64 bit shadow key settings. Same here, the overall idea is still the same the process is just a bit different.
Since the introduction of Server 2008R2 applications have become TSAWARE. As the name implies, the application is ‘aware’ that it is being installed on a Terminal Server enabled system. This doesn’t mean that all applications installed on an R2 system have this feature, they just started developing applications differently around the release of 2008R2. So, when installing a TSAWARE app nothing gets written to the shadow key right? Right! Newer applications have a special bit enabled (flag) in the header of the executable which tells the system that the application is terminal server aware and doesn’t need to copy anything to the shadow key location, this goes for 32 and 64 bit applications as well as 32 bit applications on a 64 bit system. Shadow key echoing is only enabled if the executable being installed is NOT marked as TSAWARE. Same rules apply for Windows 8 and 2012.
With all the application and operating system virtualization going on these days the whole shadow key concept isn’t always applicable or might function slightly different… it all depends. Fact is, these are the basics you need to know!
Well… That’s about it, my first Blog done!
Please let me know what you think, all comments are welcome!
Bas van Kaam ©
Reference materials used: Citrix.com, Brian Madden.com and Google.com