User Profile and Home Directory Storage: Distributing the Load Across Multiple File Servers

This article is part of Helge’s Profile Toolkit, a set of posts explaining the knowledge and tools required to tame Windows user profiles.

The easiest way to assign user profile and home directories is via group policy. But that can only be done per computer. There is no (simple) way to point different users’ directories to different file servers. So what? No problem at all, until the number of users is too large for a single file server to handle. This article discusses what can be done to spread the user load.

The Situation

File servers are quite capable these days. But even with the advent of really large 64-bit Windows systems, there still remain bottlenecks. User profiles are an interesting example. Profiles put no load at all on file servers – 90% of the day. But in the morning hours, when all employees log on practically at once, and in the evening hours, when everybody logs off, profiles put the file server under heavy stress. Consider a not too large corporation with 10,000 employees who all log on between 8 and 9 am. The file server needs to serve out 10,000 profiles in a period of 60 minutes. Given an average profile size of 10 MB, that amounts to roughly 28 MB/s. Taken into account that profiles contain myriads of small files, 28 MB per second is a lot. A single file server will probably not be able to handle the load.

Now let us consider home directories. Here, the load pattern is entirely different. Home directories are not accessed during logons or logoffs, but during sessions. In many companies, the home directory is the primary storage location for individual users. People store documents they are working with on their home directory and access them frequently while at work. But that amounts only to a small fraction of the load caused by home directories. In many environments, redirected folders are responsible for a much larger portion of the load. The cause? As always, badly written applications…

Take %AppData% as an example. Redirecting it to the home directory is generally a good idea. Unless you have applications that read and/or write from/to %AppData% many times per second! Consider an application that accesses %AppData% twice per second and multiply that by 10,000 users – the file server gets hit by 20,000 IO requests per second. That is a tough number to handle.

Spreading the Load

Depending on the number of your users, their logon patterns and the applications on your network, you may employ the following tactics, in ascending order according to complexity but also load spreading capability.

1. All homes and profiles on one file server: The load is distributed only by the different access patterns to home and profile directories. Paths are best assigned via group policy. Well-suited for small to medium-sized environments.
2. All profiles on one file server, all homes on another one: Not too much of an improvement over the previous variant unless users log on frequently during the day. Paths are best assigned via group policy. Most-suited for medium-sized environments.
3. Distribution of homes and profiles across file servers by Active Directory user account attributes: Each AD account directly stores the paths to each user’s home and profile. This creates great flexibility that comes at the price of not being able to use group policy for the assignment. Instead, scripts must be used for user account creation and maintenance. Most-suited for large environments.

   Note: In order to evenly distribute users across file servers a rule must be established as to the placement of each individual user. One option is to use the personnel number (often stored in AD) modulo the number of file servers. In other words: divide the personnel number by the number of file servers. The remainder of the division specifies which server stores a user’s directories.

4. Distribution of homes and profiles across file servers by DFS namespace links: By using DFS namespaces to create a “flat” folder structure spanning all file servers group policies can again be used to assign the home and profile paths. This moves the need for scripting away from the AD accounts, but creates the requirement to create two DFS links per user, which need again be scripted. Most-suited for large environments.

   Note: Up to and including Windows Server 2003, domain-based DFS namespaces could only manage 5,000 links. Not enough for larger corporations. Beginning with Server 2008, domain-based namespaces have no fixed limit.