WFE Application Pool Limitations in SharePoint 2010

Note: an updated version of this post with revised recommendations is available here.

As the agenda looked so interesting I decided to drive to Ullesthorpe (home of Combined Knowledge in the midlands) on July 22 2010 to the SharePoint User Group UK (#SUGUK). Considering I live in West Berkshire (and drive a 1993 Ford Fiesta, which I am rather fond of) this turned out to be a 5 hour round trip. Was it worth it? Hell yes! In addition to two very different - but fantastic - presentations from @mattgroves (big on social and lead me to believe that SharePoint is in fact an elephant) and @SteveSmithCK (capacity planning extraordinaire), I managed to win myself a 12-month MSDN subscription! Given that the question was basically "who has the rights to deploy a sandboxed solution in SharePoint 2010?", I consider myself very lucky to have won given the number of other SharePoint addicts sat in the room with me. I think the fact that most of the other attendees appeared to already have one (I heard someone behind me talking about how she planned to give her "spares" away) helped considerably.

However, whilst I enjoyed both presentations thoroughly, one question from the audience troubled me somewhat: "is there a limit to the number of application pools you can host on each WFE server in SharePoint 2010"? While Steve's answer was "around 100 before you start to run into IIS issues", I couldn't help but shake a nagging feeling that I had read the TechNet recommended maximum was 10. In fact, I was so damn sure of that TechNet number (10 app pools / WFE) that I would have bet my brand new MSDN subscription on it.

Now I was faced with a dilemma here: did I take Steve Smith's guidance as gospel, or go with Microsoft's recommendation? I did what I figured every sensible SharePoint administrator would do in my situation: I sat down, got back in my box and reminded myself that on the odd occasion TechNet tells more fibs than my parents used to on Christmas eve.

I did however make a point of looking up the TechNet article in question this evening. It does indeed list 10 application pools per Web server as a recommended guideline - in this case the "Limit type" column indicates the figure is a supportability number. Now, I know MS supportability numbers and they are not typically less than you would expect - and I would certainly not expect it to be ten times less than the guidance provided by a SharePoint legend such as Steve Smith. As a recent example, @harbars kindly informed me that having over 50 Web applications in one MOSS farm was, in his words (tweet) "madness" - Microsoft recommend no more than 99 Web applications. I think you know which guidance I paid attention to and he doesn't live in Redmond.

To be fair to Microsoft, the article above does also state that the maximum number is largely determined by hardware capabilities and the workload that the farm is serving. Steve also confirmed this at the user group by telling me that the app pool ceiling has been lifted dramatically due to the move to a 64-bit-only architecture for SharePoint 2010. Now I know that 64-bit brings about numerous performance and scalability benefits (not least of which is a practically unlimited, continuous address space for user mode processes), but I am still keen to find out why there is such a huge discrepancy between Microsoft's stated supportability figure and the guidance of industry experts who have used the product for the best part of 2 years.

So, rather than pester Steve again, I did what I often do when I have a nagging capacity-related query: I sent the query to a few friendly folk in the SharePoint community who (despite the fact that they are quite clearly very, very busy people) normally find the time to respond to my numerous concerns. Did anyone reply this time? You bet.

@ToddKlindt gave me a pretty clear response by saying that "anything Steve says, I believe". Sounds like good advice right? He also told me that the documented MS advice was actually pretty good considering that they suggest the limitation is largely hardware dependant. For those of you that don't know Todd, he is a well respected SharePoint guru and a co-author for Professional SharePoint 2010 Administration. If you don't have it already and you are a SharePoint 2010 administrator I can whole-heartedly recommend that you add this to your one to your collection of geeky paperbacks.

@joeloleson's article entitled "What’s New in SharePoint 2010 Capacity Planning" also mirrors the advice kindly provided by Steve and Todd. Joel points out that Microsoft have adjusted some of their capacity planning guidance based simply on the fact that IIS and SQL scale better with 64-bit, and states that the number of application pools " totally depends on the RAM on the box".

So how many Web applications can you host on a SharePoint 2010 WFE server? It looks like the usual SharePoint consultancy response: "it depends on your hardware's capability and server farm load", and there is no magic number other than the guidance provided by Steve that you may run into IIS issues as you approach 100 pools on one server. Microsoft's supportability number of 10 is there as a rough guide (probably for those running a minimal WFE hardware configuration - think 8GB of RAM), but is almost completely dependent on your available hardware. The new SharePoint 2010 Administration Toolkit includes a load testing kit that should help administrators validate their hardware requirements as part of a capacity planning exercise.

I hope that clears things up a little bit for anyone else concerned that they have more than 10 application pools on their SharePoint 2010 WFE servers. I'll see you at the next #SUGUK!

Benjamin Athawes

Subscribe to the RSS feed

Follow me on Twitter

Follow my Networked Blog on Facebook

Security: Preparing your SharePoint farm for a corporate security review (part 2 - Auditing)

The words "security review" in the context of IT can mean different things to different organisations. For large organisations seeking to outsource an IT function to a third party hosting provider it could be simply a necessary formality; a series of hoops that must be jumped through in order to reassure users that the third party solution is secure. Secure in this context typically means that the risk of the service availability, integrity and confidentiality being impacted is no more than is usual for solutions of a similar nature.

However, for smaller organisations with fewer resources, it can sometimes result in a mixture of fear and anxiety due to the consequences of failing said review. After all, who would trust their sensitive data to a hosting provider that has not been found worthy of a security stamp of approval?

The concerns raised over a security review can be minimised by ensuring your organisation has prepared itself effectively. In preparing ourselves, we typically start by reviewing a list of "frequently asked queries" that most organisations send us as a part of their security review. This weeks category is Auditing (part one on Access Controls is available here):

Please note that the views expressed herein are my personal views and are not intended to reflect the views of my employer at the time of writing.

2. Auditing

• Q. Will common actions (e.g. accessing and editing documents) be audited?
• A. Clients like the ability to audit access to their information - ensure there is minimal impact to performance and enable it (IIS, SharePoint, Web Proxy logs).

This sounds like an obvious one but in my experience it is often missed when deploying SharePoint sites. MOSS itself has a number of different auditing options, and although to my knowledge there are no built in retention settings they do provide a fairly detailed means of auditing actions within SharePoint. A range of different events can be audited from basic actions such as opening documents to slightly more obscure events such as searching site content. I wont go into any detail here about how to go about configuring these settings in MOSS as Bill English has already provided a nice post on his blog.

I have heard rumours that SharePoint 2010 will include retention settings as well as the current settings provided by MOSS - please let me know if you know somewhere that confirms this.

Aside from SharePoint, I would also recommend you enable IIS logging (log to somewhere other than the OS drive to prevent unnecessary disk usage) and Web proxy logging (e.g. ISA Server) where applicable. In our case, we set ISA server to log to file rather than a SQL DB for consistency with the IIS logs.

• Q. How, and how often are the log files checked?
• A. Weekly is typically sufficient but this may need to be more frequent depending on the nature of your service. In particular, check SharePoint access and AD audit logs.

Manually checking logs for security breaches is a necessary (albeit dull) part of maintaining a secure hosted service. A diligent server administrator should implement a range of processes to minimise the chances of a security breach going unnoticed depending on the nature of the service offering. In the context of a SharePoint site, I would recommend that access logs (that is, SharePoint audit logs and IIS logs) are compared to your ACLs. For IIS, I would recommend you use a tool such as Log Parser; for SharePoint I would start with the built in site collection usage reports and analyse the more detailed audit reports on a less frequent (e.g. fortnightly) basis.

You should also consider checking your AD logs (Windows "Security" logs on your domain controllers) on a regular basis as a part of your audit process.

• Q. Are server log files read only?
• A. Set all log files to read only to improve log credibility. Consider moving old logs to another server to prevent tampering.

Setting log files to be read only is an essential step toward maintaining credible log files.

Obviously someone in the organisation must retain write access to the logs but this risk can be mitigated by implementing SoD and preventing those individuals from having write access to the relevant sites (as opposed to site logs) in question. In smaller organisations, it may be easier to simply grant write access only to a very small number of individuals - perhaps those in the upper echelons of management (I'll let you decide if that's a good idea).

Mark Burnett wrote a good post on this back in 2002 (here) that I recommend you refer to for a more detailed explanation.

• Q. Are failure and success logon attempts logged?
• A. Consider enabling failure and success auditing based on your disk space and log retention requirements.

This seems to be another common query. In preparing for a corporate security review I would recommend you review your current log space requirements and (if you haven't already) consider enabling both failure and success auditing. I mention disk space because enabling failure and success auditing can rapidly fill a log for a large site and may not be feasible where storage is limited. This risk is further increased by the chances of a DoS attack which would result in a large number of failure audits in quick succession (see this MS article for more information).

A workaround for this problem is to enable log overwrites and limit the size of the event log based on current size and log retention requirements. For financial systems (e.g. banks) this option is typically not viable given that there are typically legal requirements to retain logs for a fixed duration of time. However, for Web applications that are of a less sensitive nature (that are not subject to the same legal restrictions) you may want to consider this as an option.

• Q. Are logs backed up?
• A. Ensure all logs are backed up and secured in a separate location to your Web server(s).

Again, this is something I typically get asked by financial organisations due to their understandably stringent security requirements. Whether your clients have asked for this or not, however, I would recommend you implement a log backup process that meets your retention requirements. If logging to a file (e.g. IIS), this can easily be achieved by adding the log directory to your regular NTBackup routine.

When backing up logs, ensure that the logs are shipped to a location that is physically separate from your Web servers. This is useful from both a log integrity and DR perspective - if a security breach occurs on your Web server and there is a risk that the logs have somehow been tampered with (have you set them to be read only?), you have a second copy for verification. If your Web servers suffer a hardware failure, you still have log backups to meet any legal requirements that may be imposed on your service.

Subscribe to the RSS feed

Follow me on Twitter

Follow my Networked Blog on Facebook

Add this blog to your Technorati Favourites