How Group Policy Impacts Logon Performance #1: CSEs
This article is based on my Citrix Synergy 2015 session and is the first in a mini-series on Group Policy performance. All measurements by uberAgent on Windows Server 2012 R2 with Citrix XenApp 7.6 in a steady state.
Gpupdate /force is for wimps!
Say you have changed a Group Policy setting in the domain and want to test its effects on a member computer. You open a command prompt and type:
gpupdate /force
Please pause and think this over before hitting enter. Why the /force switch? To show that stupid machine who is its master? Are you one of those people that click Apply before they click OK? Do you wear both belt and suspenders? Of course you do not! So let us take a look at the help text for the /force parameter:
Reapplies all policy settings. By default, only policy settings that have changed are applied.
That is quite telling. Group Policy keeps track of what has been applied and does not reapply settings that are already present. Nice! So why would we override this optimization? We would not. Using /force typically is only required when your Group Policy infrastructure (i.e., AD and/or DNS) are broken. Go fix it instead of telling poor old Group Policy to forego optimizations!
Modules
Group Policy may seem like a monolithic beast to you, but in fact, it is not. It is more like a framework that can be extended quite easily. Microsoft provides the engine and a significant set of modules. Vendors like Citrix add their own.
To make things sound more elaborate and bestow yet another acronym on us modules are not called modules, they are called client-side extensions (short: CSEs). All of these nodes in Group Policy Editor are individual CSEs:
CSE Count
Microsoft keeps adding new capabilities to Group Policy. The number of CSEs that ship with the operating system changed quite significantly over time:
- Windows 2000: 9
- Windows XP: 13
- Windows 7: 42
- Windows 8.1: 47
CSE Registration
CSEs live in DLLs that are registered in the registry key HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon\GPExtensions:
CSEs are identified by GUID. Important values per CSE (GUID) registry key:
- DllName
- EnableAsynchronousProcessing
- Determines whether or not group policy waits for the CSE to complete.
- Default: 0 (= synchronous)
- ProcessGroupPolicy/ProcessGroupPolicyEx
- Function in the DLL to be called when Group Policy processing is passed to the extension
- NoMachinePolicy
- Determines whether or not the client extension will process a group policy when a machine policy is being applied.
- Default: 0 (= process)
- NoUserPolicy
- Determines whether or not the client extension will process a group policy when a user policy is being applied.
- Default: 0 (= process)
- NoSlowLink
- Determines whether or not the client extension will process a group policy when a slow link is detected.
- Default: 0 (= process)
- NoBackgroundPolicy
- Determines whether or not the client extension will process a group policy when a background refresh of the group policy occurs.
- Default: 0 (= process)
- NoGPOListChanges
- Determines whether or not the client extension will process a group policy when there are no changes between the cached list of GPOs previously processed and the current list.
- Default: 0 (= process)
- PerUserLocalSettings
- If enabled, user policies are cached on a per-user and per-machine basis.
- Default: 0 (= disabled)
- RequiresSuccessfulRegistry
- If enabled, requires the successful registration of client-side extension components to occur before it processes a group policy.
- Default: 0 (= disabled)
CSE Processing Order
Client-side extensions are processed in the following order: administrative templates first (i.e., policy settings that are simply written to the registry). Everything else afterwards, in numerical order of the CSEs’ GUIDs.
Microsoft Documentation
As the documentation states: The Group Policy Registry Extension MUST always execute first.
And: The Extension List is in ascending order by CSE GUID (source).
List of CSEs
On Windows 8.1 the order is as follows:
- Administrative templates
- GUID: 35378EAC-683F-11D2-A89A-00C04FBBCFA2
- Wireless Group Policy
- GUID: 0ACDD40C-75AC-47ab-BAA0-BF6DE7E7FE63
- Citrix Group Policy
- GUID: 0D0C7034-2EBD-4A87-A9B9-9015E3F2E6E0
- Group Policy Environment
- GUID: 0E28E245-9368-4853-AD84-6DA3BA35BB75
- Central Access Policy Configuration
- GUID: 16BE69FA-4209-4250-88CB-716CF41954E0
- Group Policy Local Users and Groups
- GUID: 17D89FEC-5C44-4972-B12D-241CAEF74509
- Group Policy Device Settings
- GUID: 1A6364EB-776B-4120-ADE1-B63A406A76B5
- Folder Redirection
- GUID: 25537BA6-77A8-11D2-9B6C-0000F8080861
- Citrix Profile Management
- GUID: 26F29E43-DA55-459d-A045-5FEB25F8AB15
- Microsoft Disk Quota
- GUID: 3610EDA5-77EF-11D2-8DC5-00C04FA31A66
- Group Policy Network Options
- GUID: 3A0DBA37-F8B2-4356-83DE-3E90BD5C261F
- QoS Packet Scheduler
- GUID: 426031c0-0b47-4852-b0ca-ac3d37bfcb39
- Scripts
- GUID: 42B5FAAE-6536-11d2-AE5A-0000F87571E3
- Remote Desktop USB Redirection
- GUID: 4BCD6CDE-777B-48B6-9804-43568E23545D
- Internet Explorer Zonemapping
- GUID: 4CFB60C1-FAA6-47f1-89AA-0B18730C9FD3
- RemoteApp and Desktop Connections
- GUID: 4D2F9B6F-1E52-4711-A382-6A8B1A003DE6
- Work Folders
- GUID: 4D968B55-CAC2-4FF5-983F-0A54603781A3
- Group Policy Drive Maps
- GUID: 5794DAFD-BE60-433f-88A2-1A31939AC01F
- Group Policy Folders
- GUID: 6232C319-91AC-4931-9385-E70C2B099F0E
- Group Policy Network Shares
- GUID: 6A4C88C6-C502-4f74-8F60-2CB23EDC24E2
- Group Policy Files
- GUID: 7150F9BF-48AD-4da4-A49C-29EF4A8369BA
- Group Policy Data Sources
- GUID: 728EE579-943C-4519-9EF7-AB56765798ED
- Group Policy Ini Files
- GUID: 74EE6C03-5363-4554-B161-627540339CAB
- Windows Search Group Policy Extension
- GUID: 7933F41E-56F8-41d6-A31C-4148A711EE93
- Internet Explorer User Accelerators
- GUID: 7B849a69-220F-451E-B3FE-2CB811AF94AE
- Security
- GUID: 827D319E-6EAC-11D2-A4EA-00C04F79F83A
- Deployed Printer Connections
- GUID: 8A28E2C5-8D06-49A4-A08C-632DAA493E17
- Group Policy Services
- GUID: 91FBB303-0CD5-4055-BF42-E512A681B325
- Group Policy Folder Options
- GUID: A3F3E39B-5D83-4940-B954-28315B82F0A8
- Group Policy Scheduled Tasks
- GUID: AADCED64-746C-4633-A97C-D61349046527
- Group Policy Registry
- GUID: B087BE9D-ED37-454f-AF9C-04291E351182
- 802.3 Group Policy
- GUID: B587E2B1-4D59-4e7e-AED9-22B9DF11D053
- Windows To Go Startup Options
- GUID: BA649533-0AAC-4E04-B9BC-4DBAE0325B12
- Group Policy Printers
- GUID: BC75B1ED-5833-4858-9BB8-CBF0B166DF9D
- Windows To Go Hibernate Options
- GUID: C34B2751-1CF4-44F5-9262-C3FC39666591
- Group Policy Shortcuts
- GUID: C418DD9D-0D14-4efb-8FBF-CFE535C8FAC7
- Microsoft Offline Files
- GUID: C631DF4C-088F-4156-B058-4375F0853CD8
- Software Installation
- GUID: C6DC5466-785A-11D2-84D0-00C04FB169F7
- TCPIP
- GUID: CDEAFC3D-948D-49DD-AB12-E578BA4AF7AA
- Internet Explorer Machine Accelerators
- GUID: CF7639F3-ABA2-41DB-97F2-81E2C5DBFC5D
- IP Security
- GUID: E437BC1C-AA7D-11D2-A382-00C04F991E27
- Group Policy Internet Settings
- GUID: E47248BA-94CC-49c4-BBB5-9EB7F05183D0
- Group Policy Start Menu Settings
- GUID: E4F48E54-F38D-4884-BFB9-D4D2E5729C18
- Group Policy Regional Options
- GUID: E5094040-C46C-4115-B030-04FB2E545B00
- Group Policy Power Options
- GUID: E62688F0-25FD-4c90-BFF5-F508B9D2E31F
- Audit Policy Configuration
- GUID: F3CCC681-B74C-4060-9F26-CD84525DCA2A
- Group Policy Applications
- GUID: F9C77450-3A41-477E-9310-9ACD617BD9E3
- Enterprise QoS
- GUID: FB2CA36D-0B40-4307-821B-A13B252DE56C
- ProcessConnectivityPlatform
- GUID: FBF687E6-F063-4D9F-9F4F-FD9A26ACDD5F
Changing CSE Processing Order
While the order in which CSEs are processed is deterministic, there is no way to change it. This means that if you want to set a registry value (CSE name: Group Policy Registry) before you run a scheduled task (CSE name: Group Policy Scheduled Tasks) you are out of luck.
CSE Overhead
Each client-side extension that is called during Group Policy processing adds a certain overhead. After all, the underlying DLL has be to loaded, initialized, and so on. These are the minimum processing times of some popular CSEs as measured in my lab:
CSE Settings Storage
How client-side extension settings are stored in the domain is really up to the developer of the CSE. Microsoft primarily uses these two formats:
- Registry.pol
- Where used: administrative templates
- File location:
\\<domain>\SysVol\<domain>\Policies\<GPO-GUID>\[User|Machine]\registry.pol - Binary format (MSDN documentation)
- SDM Software have a free viewer utility
- Here is a VBScript to read and modify registry.pol files
- XML
- Where used: Group Policy Preferences
- File location:
\\<domain>\SysVol\<domain>\Policies\<GPO-GUID>\[User|Machine]\Preferences\<CSE name>\<CSE name>.xml
Performance Impact of CSEs Spread Across GPOs
Group Policy settings are stored on the domain controllers in one file per CSE and GPO. If a CSE’s settings are spread across GPOs, the number of files that have to be fetched by the client during Group Policy processing increases. This may adversely impact Group Policy processing, especially if a client is talking to a domain controller over a slow WAN link.
To test the effects of CSE settings spread across GPOs I compared the processing time for two scenarios:
- 20 GPP environment settings in one GPO
- 20 GPP environment settings, each in its own GPO
As you can see, the overhead per GPO is about 3.5 ms – in a LAN environment. Things may be very different in a WAN or if the domain controller hosting the settings files responds slowly due to high load.
Like this article? Go on reading part 2!
13 Comments
great article, one query
you say that the impact per GPO is ~3.5ms, however the graphic shows that the impact of 19 GPOs is ~3.5ms (20 GPOs Vs 1 GPO).
Am I reading that wrong?
The numbers in the graphic are per setting, as mentioned in the graphic’s title.
Helge thank you for the great write up. Looking forward to #2.
I’ve found XML processing to be a factor as well for really large GPP items (in the sense of ~100ms per CSE) but the largest by far is enabling RSOP. WMI evaluates each CSE/GPP item level targeting and *kill* logon performance. Each RSOP evaluation can be ~100-200ms and make a ~5s evaluation of a complex registry GPP to 10-20 seconds for that single CSE.
Oddly, IIRC, turning on WMI debug tracing in event viewer will show each evaluation and call. I’ll have to write a blog post about it too one day.
Excelent post!!
This is such a great detailed look into Group Policies, how did you measure the speeds of Group Policy been applied.
Hi Helge,
congrats for the post, a great summary of what needs to be known, as Robert was asking, could you please share which tool did you use to group the duration based on CSE and show it graphically?
I used uberAgent for the CSE measurements.
Helge, thanks for this post. My company uses a CSE (I believe it’s for Credential Guard) that does not show a friendly name when it’s being applied at boot, it actually just shows the GUID. I’ve dug into this in the registry and found that the friendly name for each CSE is set by the (Default) value for each GUID key, and for this CSE in particularly, that value is not set. Have you ever seen anything like this? Any idea how it could be caused?
HI Helge,
even if the article is some years old, I have a question, with which you maybe could help us with (would be awesome).
We switched our domain controllers from 2012 R2 to 2016 (no change in domain level). In our XenApp 6.5 farm the logon time increased drastically since then. Today we did some troubleshooting and found out, that each GPP setting (Registry, Shortcuts, Files, whatever) needs approx 2 seconds longer than before. For example registry hadling was in the range somewhere between 150 and 400ms and now raised up to approx. 2500ms. If you sum that up, the logon times increased by 10-15 seconds. Do you have any idea what this could be? We created both logs and traces on the citrix servers but no matter how high the loglevel is, we just see a gap in the logs for nearly 2 seconds… Also a network trace only shows successful communication… DNS and everything else is clean.
I’ve also captured logon times with a clean 2016 XenDesktop 7 install, and there the logon duration seem to be a lot faster, but I can not say for sure, because we do not have any data for comparison.
Any help /idea /direction /whatever is highly appreciated. Maybe you stumbled over somthing similar in the past.
Thank you very much
Jörn
NIce article, thanks mate, helped a lot.
Boris
Hi Helge,
Great article! May i know how do you validate the GPO logon performance?
All performance measurements were made with uberAgent (https://uberagent.com/)