Citrix XenApp 7.6 Logon Slow – Long Black Screen Phase

Update 2015-04-28: Citrix provides the limited release hotfix ICATS760WX64009 that fixes this issue. More information below.

During the research for my session about the XenApp 7.6 logon process, to be presented at Citrix Synergy and BriForum London, I noticed that the logon to my XenApp 7.6 lab server was taking a bit long. Longer, in fact, than the combined durations of the main logon phases user profile loading, group policy processing, logon script execution and shell startup. Much longer. Also much longer than on an otherwise similar XenApp 6.5 machine.

Examining uberAgent’s brand-new logon process performance dashboard I found a 10 second gap:

uberAgent- XenApp 7.6. slow logon gap

After the user profile has been loaded and group policy has been processed, which are both finished after three seconds, nothing happens for about ten seconds. Finally, at 13 seconds into the login is the Userinit process started which ultimately loads the shell, Explorer.exe. During most of that time the end user is presented with a black screen, getting no feedback as to what might or might not be happening as you can see in this video:

I experiemented a lot trying to find the root cause of the prolonged black screen phase, disabling all kinds of redirection, printer mapping, even IPv6. None of that fundamentally changed the duration of the black screen phase. Then I had the idea to log in via RDP (for which you have to make the user a member of the group Direct Access Users on the XenApp machine):

Now, that is a lot faster! RDP logons are finished in less than 4 seconds compared to around 14 seconds for ICA logons.

Digging around some more I stumbled upon CTX135782 which suggests setting the following registry value:

Key: HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Citrix\Logon
Value: DisableStatus
Type: REG_DWORD
Data: 1

The effect was immediate, bringing the ICA logon from 14 seconds down to around 5.

Happy End?

Not quite. With DisableStatus set to 1, every time I logged on the LogonUI.exe process crashed due to a fault in the module CtxWinlogonProv.dll. Events like the following were logged:

Faulting application name: LogonUI.exe, version: 6.3.9600.16384, time stamp: 0x5215f6c5
Faulting module name: CtxWinlogonProv.dll_unloaded, version: 7.6.0.5018, time stamp: 0x541cfd05
Exception code: 0xc0000005
Fault offset: 0x00000000000019e1
Faulting process id: 0x1e2c
Faulting application start time: 0x01d080da8b0d6ae2
Faulting application path: C:\Windows\system32\LogonUI.exe
Faulting module path: CtxWinlogonProv.dll
Report Id: cb285960-eccd-11e4-80c0-00155d001b10
Faulting package full name: 
Faulting package-relative application ID:

CtxWinlogonProv.dll, also called CitrixRemoteLogonFilter, is registered as a credential provider filter via the registry key HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Authentication\Credential Provider Filters\{3571A91D-713C-427c-AA0C-BBF4F618A819}. When that key is renamed or deleted LogonUI.exe does not try to load CtxWinlogonProv.dll and consequently does not crash any more while logons are still fast.

However, since I do not know what the consequences of disabling the registration of CtxWinlogonProv.dll are I cannot really recommend that approach. Ultimately Citrix are the only ones who can truly fix this.

Update 2015-04-28: a Hotfix

Shortly after the initial publication of this article Shane Kleinert pointed me to limited release hotfix ICATS760WX64009 (CTX142036) which seems to be available for server OS (i.e. XenApp) only. As he explains in the comments below Shane had noticed similar issues at customer sites and found CTX142036 in spite of the more than vague problem description.

I am happy to report that CTX142036 fixed this long black screen phase issue on my machines, too. So this story does have a happy end, after all!

, ,

41 Responses to Citrix XenApp 7.6 Logon Slow – Long Black Screen Phase

  1. Andrew April 27, 2015 at 21:15 #

    Great post! I was actually looking into something very similar to this over the weekend as well. I’ll give your suggestion a try to see what it does. fingers crossed.

  2. Shane Kleinert April 28, 2015 at 01:09 #

    Hi Helge!

    Great post! I ran into the same issue at a few Customer sites, once it was determined that this was on the Citrix side, I opened a case and pushed it through to escalation. While I was waiting, I came across http://support.citrix.com/article/CTX142036 Which has the worst possible description. It was like “Published applications may run slow” I decided to give it a go in the test environment. Immediately it dropped 11 seconds off the login time. I pinged my case owner, and asked for a reasoning on WHY this fixed the issue. after some research he let me know there was a section in the code which needed to be tweaked due to an infinite loop, this was reduced to resolve the issue. He put an a request to change the description of the article which no includes “slow user logins” which is much better. Would have been nice to include the root cause, but happy it works!

    I tried this at two locations, and it worked fine. I Tweeted it out few weeks back and a few people wrote back saying it resolved the black screen for them as well.

    Love the breakdown with UberAgent – It just keeps getter better and better! Keep up the great work!

    • Shane Kleinert April 28, 2015 at 01:10 #

      — For those wondering the hotfix is “ICATS760WX64009” This is specifically applied to the Server VDA, NOT the workstation VDA.

    • James Perconte June 1, 2015 at 14:42 #

      Hey Shane,

      Have you ever seen the slow logon with Xenap 6.5, with receiver 3.4. I have been trying to troubleshoot this issue for a couple of weeks now with no success. After a visit to Synergy in May 2015, I learned about some really cool tools from Helge Klein, and the Uber agent. Still trying to figure out how to view the Logon Times Screens Helge presented at Synergy.

      We have seen logon times to up 3 minutes to enumerate the icons to the desktop with the PNagent.

      Shoot me an email if you are interested in some consulting work up in Stuart.

      James.

  3. Trentent April 28, 2015 at 07:05 #

    Do you use Mandatory profiles?

    • Helge Klein April 28, 2015 at 10:33 #

      No, just a regular Windows roaming profile.

  4. David April 28, 2015 at 15:16 #

    Hi Helge,

    I’m having this issue on XenApp 7.6 environment. None of the fixes I found worked. I’m trying to investigate with UberAgent, how can I find the “Logon Process Performance” dashboard you speak of ?

    Thanks, and congratulation for this great tool, very good work.

    Regards

    • Helge Klein April 28, 2015 at 15:26 #

      The logon process performance information will be part of the next release of uberAgent after the current version 3.0.

      • Roman August 14, 2015 at 10:13 #

        Hi Helge
        When will the “logon process performance” be avaible? Do you have a Date?

        Regards

  5. Raj April 30, 2015 at 22:10 #

    Is it safe to assume that only XD7.6 has this issue and not 7.5? If not, is there be a similar HF for XD 7.1/5?

    • Helge Klein April 30, 2015 at 22:15 #

      While this specific issue seems to affect XenApp 7.6 only there have been similar problems in earlier versions. See CTX135782 which applies to XenApp 6.0, 6.5, XenDesktop 7.0 and 7.1. You should examine your logons for longer “black screen” phases and contact Citrix support if you are affected by similar issues.

      • Raj May 1, 2015 at 14:24 #

        Thanks Helge. I’ve applied that registry on Windows 2008 R2 and it helped speed up the logon. I also have it setup in Windows 2012 R2 without any issues. I’ll test the hotfix and see if it further helps the logon.

  6. Jörg von der Ohe May 4, 2015 at 14:21 #

    Hi there,
    the main problem on this hotfix is, that you have to open a support ticket to get it.

    • Helge Klein May 4, 2015 at 14:21 #

      I was able to download it after logging into My Citrix.

  7. Jörg von der Ohe May 4, 2015 at 14:35 #

    I get:

    If the Download link is not available on this page and you wish to obtain this limited distribution release, visit our support site at http://www.citrix.com/support and open a support case using your Citrix account credentials, or contact your reseller at http://www.citrix.com/partners/locator.

    • Helge Klein May 4, 2015 at 14:40 #

      I just tried and cannot open CTX142036 any more, either. Hopefully Citrix is transitioning this into a regular public hotfix.

  8. mathias wuersten June 26, 2015 at 06:24 #

    we have the same issue with XenApp 7.5 – after upgrade to XenApp 7.6 still the same – after installing CTX142036 (only for XenApp7.6 possible) still the same :-(
    we will open a citrix case this days

  9. julien June 26, 2015 at 09:38 #

    I know it’s not supposed to be related, but does anyone try this with receiver 4 ?

    http://support.citrix.com/article/CTX138197

    have a look in the Receiver 4.1 release note http://support.citrix.com/article/CTX138197

    When launching a published application in seamless mode, the progress bar window remains in the background.
    To enable the fix, set the following registry keys on the client-side:
    For Windows 32-bit systems:
    HKEY_LOCAL_MACHINE\SOFTWARE\Citrix\ICA Client
    Name: ForegroundProgressBar
    Type: DWORD
    Data: 00000001
    For Windows 64-bit systems:
    HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Citrix\ICA Client
    Name: ForegroundProgressBar
    Type: DWORD
    Data: 00000001
    [#LA3491]

  10. Paul August 20, 2015 at 15:17 #

    We applied the fix just now and time from clicking a published application to getting the app logon screen dropped from 27/28 to 21/22 seconds. But only on subsequent tries.
    What i’m saying is that first time selecting this application makes no change in getting to your app logon screen, but the next attempt will be shortened by 5 or 6 seconds.
    This sounds like session lingering, but we do not use this and if I check the RDS server, the user definitely logs off when he/she closes the application.
    Do you have this effect as well?

  11. Morten September 23, 2015 at 09:54 #

    Anyone fixed this permanently? The fix helped for some time, hours, then the problem is back. Tried the fix one more time, no help. Also tried the reg settings mentioned, but no better. Checked with ControlUp that confirmes that it is the Shell loading part that is causing the problem. Profiles, GPO etc are all finishing fast.

  12. Peter October 8, 2015 at 12:24 #

    Seeing this on XD7.6. Our black screen lasts for a minute. I thought it might be negotiation of graphics mode.

  13. Peter Singer November 10, 2015 at 23:09 #

    Hi,

    For a time the following hot fix seems to have worked and solved the problem,hotfix ICATS760WX64009 (CTX142036). But now it is happening again. Has anyone had this issue return and is there any further information available?

    Peter

    • Matt Sliva December 3, 2015 at 23:34 #

      Peter,

      I’ve seen the issue return since upgrading our XA servers to XA 7.6 FP3.

      Matt

      • Alex December 17, 2015 at 00:48 #

        Matt, I’ve the same issue with XA 7.6 FP3

        • Jörg von der Ohe December 22, 2015 at 09:27 #

          Same here, after upgrading to XA 7.6 FP3 the issue returns

  14. Steve Walker January 4, 2016 at 11:16 #

    Does anyone have any update to this issue? I’m seeing exactly the same 10 second delay during launches of published apps when using XA 7.6 FP3. Just for info I was not able to install the previously mentioned VDA update ICATS760WX64009 because FP3 includes a newer version which invalidates the upgrade path. Bit disappointing really..

  15. Steve Walker January 4, 2016 at 13:05 #

    Unfortunately this latest update did not appear to solve my logon delay problem. I had a strange experience during the install however which I will outline in case it helps anyone else here. Basically I was not able to install ICATS760WX64048 (dated 29th December 2015) on top of my FP3 system because it appears that this is either included within FP3 or is invalidated by the presence of it. The exact message displayed is ‘The upgrade patch cannot be installed by the Windows Installer service because the program to be upgraded may be missing..’). On a test server I uninstalled the VDA entirely, installed the 7.6 base release VDA and then applied successive patches (ICATS760WX64032, 34, 40 and then 48) with a reboot in between each. At this point I tested logon performance with each individual patch and it did not immediately improve/degrade further. I then applied FP3 VDA (7.6.300) and attempted to re-apply hotfix 048, at which point I was told by the installer that it was not compatible with the system. So I can only guess that FP3 is interfering with a later hotfix installation or it does indeed contain hotfix 48 already!

    • Markus Zehnle January 7, 2016 at 11:55 #

      Have you tried removing the full path from the AppInit_DLLs key like stated here?
      http://support.citrix.com/article/CTX205179

      Thx to Rene Bigler (@dready73) who pointed me to that article…
      https://twitter.com/dready73/status/685048592273420288

      • Steve Walker January 20, 2016 at 21:38 #

        No – thank you Markus, I will check it out because it’s been driving us mad. I just tried using the new VDA in XA7.7 and we still had the same issue, although it appears variable in the amount of time the black screen is displayed.

        • Frank February 1, 2016 at 07:47 #

          Hey Steve,

          I also have the issue on a new installed XA 7.7

          Has anybody a solution ?

  16. Daniel February 3, 2016 at 11:29 #

    Same probleme here, after upgrading to XA 7.6 FP3 the issue returns

  17. Jerome March 2, 2016 at 11:16 #

    Same issue here with XA 7.6 FP3 -but only when reconnecting to a disconnected session – logon of new Session is quite fast….

  18. Paul March 18, 2016 at 03:47 #

    I have an intermittent black screen issue with XD 7.7 W2K12R2 as well. I have a call logged with Citrix, but we can’t reproduce the issue when we need to capture any traces. This is a nightmare! We have no issue reconnecting to disconnected sessions, we only see the issue for new user logins

  19. TheBogtrotter March 22, 2016 at 23:17 #

    We have found one thing slowing down our logons on XA7.6 and Win2012R2 – both with and without FR3.

    Using Process Monitor we could see Windows Explorer was trying to delete registry entries in HKCU\Software\Microsoft\Windows\CurrentVersion\UFH\SHC

    These keys are for the icons on the Start Menu, and for us get recreated on each login (not sure why, but it may be because we point to a redirected Start Menu). The ID of the keys is incremental and for some users were starting from 60,000. For some reason Windows was trying to delete all entries from 1 to 60,000 – and all the end user saw was a black screen until it completed.

    We deleted all entries in this registry Key for a user, then when the user next logged in the logon was around 30 seconds faster!

    • Edwin April 11, 2016 at 12:47 #

      Using Process Monitor I also discovered a lot of activity trying to delete entries in HKCU\Software\Microsoft\Windows\CurrentVersion\UFH\SHC. I previously used GPO do generate Start Menu shortcuts. I now use a new share with “Access Based Enumeration” and placed all shortcuts there. I redirect my Start Menu with GPO to my new share. Also made a script to clean up Registry:
      REG DELETE HKCU\Software\Microsoft\Windows\CurrentVersion\UFH\SHC /va /f
      Redused logon time from 55 sek to 16-17 sek.

      • Dennis October 11, 2016 at 08:20 #

        Hi Edwin,

        Do you have an example of the script. And how do you run it for all the users?

        Kind regards

  20. dubsdj May 31, 2016 at 18:51 #

    I tried all of these nothing helped. After exhausting myself on google search I decided to sit down one evening and follow this:

    https://support.citrix.com/servlet/KbServlet/download/36991-102-715256/Windows_8.x_Optimization_Guide_Final.pdf

    sorted the long delay black screen problem.

    Warning don’t disable device setup manager service if you want to script adding printers etc.

  21. CRIOSS October 12, 2016 at 18:17 #

    Has someone fixed this issue?? it’s become a headache!

  22. Markus Zehnle August 16, 2017 at 10:42 #

    “Finally” Citrix and Microsoft may have fixed the issue:
    https://support.citrix.com/article/CTX225819

    At the moment I have no access to any affected environment, maybe someone else can try!?

Leave a Reply