Taming Black Holes: Parallel Session Creation

Have you ever tried to log on to a terminal server and, after entering your credentials, been forced to stare at a grey screen for a lengthy period of time wondering what the machine might actually be doing? Of course you have, along with a few million other terminal server users. Being a technical guy (you would not be reading this otherwise) you have checked CPU / memory / hard disk utilization and the current session count when users complain that logons are slow. You will probably have noticed that all relevant metrics are in the green and logons are the slower, the more users try to log on to a server concurrently. It turns out that parallel logons are the root cause of the problem. Why?

The Old Way

Prior to Windows Server 2008 (and Vista) sessions were created serially. New logons were queued and users would have to wait until the previous session initialization had completed. That essentially means that if you are the last of five users trying to log on to a terminal server in a short time span you will be staring at that grey screen until first user 1 is logged on, then user 2, then user 3… You get the picture.

The New Way

Beginning with Windows Vista, at least 4 sessions will be initialized concurrently. If you have more than 4 processors the number of parallel initializations is lifted to the number of processors installed. What I have not found out yet is whether Microsoft refers to the number of processor cores or actual CPUs.

Background

Instances of smss.exe (Session Manager SubSystem) are now started per session. It is as simple as that.

Smss.exe is the parent process of any other process that runs in a Windows session. It spawns off child processes that prepare a user’s environment, load the profile and finally start the shell. On top of all this sits one initial instance of smss.exe that creates up to n copies of itself which each initialize their own session (n ranging from 4 to the number of processors installed).

The first session-specific smss.exe performs the task of initializing session 0, the services session. It does this by starting first csrss.exe and then wininit.exe which in turn starts services.exe, lsass.exe and lsm.exe.

Note: Winlogon.exe is not needed here since session 0 is non-interactive. Instead, wininit.exe starts the service control manager (services.exe), the Local Security Authority SubSystem (lsass.exe) and the new Local Session Manager (lsm.exe).

When an instance of smss.exe has completed its task it ends, making way for other instances that initialize additional sessions. Thus, session 0 smss.exe makes sure that machine startup is well under way and then quits while its child processes in session 0 prepare the system by initializing subsystems and starting services.

Terminal Services

One of those services implements, of course, our well-known terminal server functionality. By the time the “terminal services” service has started, session 0 smss.exe presumably has finished its work so that at least four new instances of smss.exe can be created in parallel.

When the first user logs on a new instance of smss.exe is created that prepares the user’s session by launching the Client/Server Runtime SubSystem (csrss.exe – the user mode portion of the Win32 subsystem) and winlogon.exe. Winlogon in turn starts LogonUI.exe and userinit.exe. All this finally leads to the loading of the shell as defined in the registry value HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\CurrentVersion\Winlogon\Shell. As you have experienced it can take some time before explorer.exe starts decorating the screen with your favorite icons.

When a second user logs on while a previous session initialization is still under way Windows Server 2008 does not wait like its predecessors. Instead, it does what any OS on today’s multi-core multi-Gig machines should do: run a second session initialization routine concurrently. And a third. And a fourth. And a fifth, if it deems your hardware powerful enough. And so on.

Taming black holes

This sounds promising. And it might be the long-sought answer to a problem Citrix has been battling with for quite some time: the black hole effect. Imagine a large terminal server farm with many logons per minute which are, under normal circumstances, distributed evenly across the servers. What, do you think, would happen, if a new server was booted up? The load balancing routines would direct every user trying to connect to the newly started server which would effectively collapse. Bringing in a second server would not help – the process would only start all over again.

While I would not dare say that Parallel Session Creation in Windows Server 2008 will eliminate the black hole effect I do hope that it provides a welcome relief on the way to ever higher capacity servers that will have to process a steadily increasing number of logons per minute.

Additional Information

Mark Russinovich on Windows Server 2008 Kernel Changes
Jason Conger’s Blog
Dan Shappir’s Blog