SharePoint disaster recovery on a budget

A post over on the MSDN forums got me thinking about disaster recovery in SharePoint. I thought I would share my thoughts with you given that availability and disaster recovery are important concepts to anyone hosting (or using) SharePoint. Throughout the article I use information contained in the 2010 and 2007 technet articles entitled "Plan for availability".

In the forum post, the OP asks for feedback on his SharePoint 2007 DR strategy:

• The farm consists of two physical DB servers and two physical Web servers. One physical Web server is virtualised meaning there are 3 Web servers in total. Note that it is assumed that these servers also host the query and index role.
• There is a very limited budget (as is often the case) in that only one server has been approved for DR purposes.
• No availability (e.g. uptime %) requirements are specified.
• No capacity (e.g. number of users, RPS) requirements are specified.
• The OP plans to implement a "stretched farm" over a WAN link. We can assume here that they are not closely linked as it is stated that that backup server is "in another part of the world". The DR server would be added to the farm as a Web server, and other roles would be added if required.

Frankly, being given an entire server in a separate data centre for DR purposes is more than a lot of SharePoint administrators could hope for. Although in an ideal world we would all have a completely redundant DR farm in a nearby data centre on hot standby, the OPs question is a lot closer to reality for most organisations.

However, I did think of a few potential issues with the OPs suggestion:

• Stretched farms are only a realistic scenario where they are in close proximity with high speed links. That means less then 1ms latency and 1GB/s bandwidth.
• The SharePoint configuration and central administration content databases contain computer-specific information meaning that the restoration environment must contain the same topology and server roles - which would not be possible on a single server.
• There is no mention of the other SharePoint infrastructure requirements, including DNS and a user directory, although we will assume for this scenario that the OP has included those services as part of their DR strategy anyway.
• Aggregate capacity requirements need to be met on the single DR server for the duration of the servers use. This means whatever resources were used on the "live" farm servers need to be available on the standby. Amongst others this includes CPU cycles, RAM, disk capacity, disk IO requirements, and network bandwidth.

Sample aggregate resource requirements

Note that the aggregate totals displayed here are simply for the minimum system requirements and do not include network and disk capacity. Realistically, each server may well have a lot more hardware resources. For example, assuming each server has 8GB of RAM, 100GB of disk capacity and a 3Ghz dual core CPU (not uncommon in modern MOSS 64-bit environments) , the requirements are suddenly a lot tougher to meet on a budget: the OP would need a server with 40GB of RAM, 500GB disk space and a 10 core CPU!

From these observations I concluded that:

• A separate standby farm is required - as we only have one server available this will be a standalone server with all roles on one box.
• The standby server needs to meet the aggregate capacity requirements provided by the existing servers for the duration of the outage.

The standby type (cold, warm, hot) depends on the OPs availability requirements. These are detailed in Plan for disaster recovery (SharePoint Server 2010). Given that there is clearly a limited budget, one can assume that maintenance and configuration costs need to be kept to a minimum, in which case a hybrid approach seems preferable. Here is one possible approach based on Microsoft's guidance in provisioning a hot standby data centre:

• Create a separate DR farm and apply all Windows, SQL, WSS and MOSS updates to match that of the live environment as closely as possible.
• Deploy all SharePoint customisations to the DR farm
• Create and configure all Web applications to the DR farm, restoring all content databases from the live farm.
• Test it!

Going forward, the maintenance required really depends on required availability. If the business can't cope without SharePoint for more than a few hours than the OP needs to consider configuring SQL log shipping to ensure his content databases are synchronised, and will need to ensure all live updates are also deployed to the DR server at all times. If, on the other hand a day's worth of downtime is acceptable the OP may decide to simply document any live changes and deploy them to the DR server on an "as-need" basis.

Of course, this is just one possible approach. If you have any suggestions or improvements I'd like to hear them!

Benjamin Athawes

Subscribe to the RSS feed

Follow me on Twitter

Follow my Networked Blog on Facebook

Security: Preparing your MOSS farm for a corporate security review (part 1 of 3)

I would like to begin this post by thanking those that have recently subscribed to this blog via RSS. I always have a hard time deciding whether a blog should consume valuable space within my Google RSS reader so greatly appreciate the thought.

This post is primarily intended for those hosting or planning to host secure MOSS Web applications, although most of these points could also be applied to most Web applications that host sensitive content over the security minefield that is the Internet.

Having recently been through numerous corporate security reviews with potential clients, I thought it would be interesting to share some of the common queries that I get asked, along with an explanation and indication as to what constitutes an "acceptable" answer in terms of security. I imagine a lot of MOSS system administrators that host secure, Internet facing Web applications have been through the numerous security related hoops described below (including, in most cases a "PenTest" - I will detail preparation for that in another post). I thought it would be useful to document some of the most common queries.

Please note that the views expressed herein are my personal views and are not intended to reflect the views of my employer at the time of writing.

Without further ado:

1. Access Controls

• Q: How many different access levels are there for your application?
• A: 2 levels of access control is normally sufficient.

This one depends on the nature of your application. In most cases, an externally accessible MOSS Web application will contain multiple access levels. Similarly, in cases where said MOSS Web application is third party hosted, clients may not have access to carry out potentially dangerous functions that only an "administrator" should be able to perform. Typically, most users will have either "read" or "contribute" access, whereas power users might have "design", or even "full control" for some sites.

Security conscious organisations look for multiple access levels to allow for SoD. In my limited experience, the larger the organisation, the tighter the desired level of access control and, in MOSS terms the more users with "read" access. Having two clearly defined access levels is normally sufficient to satisfy most clients requirements but expect to implement more if your client(s) wish to implement access controls similar to their own organisational structure.

• Q: Are users configured with individual usernames and passwords?
• A: A must have for secure sites with auditing.

Pretty simple - in a secure environment where auditing is required, individual logons are a necessity rather than a luxury and your clients should demand it.

• Q: Is a password policy enforced, including forced password changes?
• A: Enforce a strong password policy; forced password changes are recommended.

We are forever told that choosing a secure password is a must when using sensitive applications - especially those hosted over the Internet. However, we as users can't be relied upon to resist entering a password any more secure than "chocolate123" or "admin1982". This seems to be a pet hate of large corporations, and rightly so. Enforce a secure password policy - at a minimum add some page level validation; preferably force the complexity requirement within your directory structure (e.g. Active Directory).

Security conscious organisations often look for forced password changes. However, it is not always required and this requirement does depend on the sensitivity of the application along any mitigating factors (e.g. a strong password policy and account lockouts, see below). In my experience, it is not essential.

• Q: Does the application support account lockouts?
• A: Enforce account lockouts with a realistic threshold

Automated account lockouts represent a basic, but effective measure against brute force attacks and a mitigating factor where password changes are not forced. I would go as far to say that it is an essential feature for applications that contain sensitive data - and so will your clients.

With regard to the threshold, many organisations will try to stick with their own internal account lockout policy. Commonly requested thresholds based on my experience are between 3 and 5, so I would recommend aiming for something around that area. For applications that do not require such a strict threshold, you may consider increasing this to a number between 6 and 10. This still provides some protection against brute force attacks whilst removing any frustration users might have over account lockouts.

• Q: Is there a documented process for revocation of a users access?
• A: Provide a clear, documented process that minimises chance of human error

In a lot of cases, I find that procedures for providing access are clearly defined and documented. Staff know exactly what is required in order to grant a new user access to a secure site. However, this is not always the case when it comes to revoking a users access, and is often left unchecked. If a user can't log in, your support staff will know about it soon enough. If they can log in after their account has "expired", you may find that you aren't ever notified there is an issue.

Reassure clients that you have a clear process and implement it. If you receive notification that a users access should be revoked by a set date, enforce account expiration in your directory structure and add a suitable description. In MOSS, you could go one step further and create a list of accounts marked for revocation.

Thats all for today. Join me soon for part 2. Still to come: Auditing, Change Management, Data Protection, Disaster Recovery, Infrastructure and Physical Security.

Subscribe to the RSS feed

Follow me on Twitter

Follow my Networked Blog on Facebook

Add this blog to your Technorati Favourites

Administration: Migrating to a new SQL Server in MOSS (SQL 2005 to 2008)

1. Overview

In this post I will describe the steps I took to successfully migrate from a SQL 2005 server to a brand new SQL 2008 server within a medium size production MOSS farm serving around 2000 users. The process makes use of the Microsoft supported method of migrating to a new DB server: SQL connection aliases.

As always, I strongly recommend you follow the steps below in a test environment prior to carrying them out in a live farm.

2. Scenario

Prior to the migration, we had a medium sized MOSS farm consisting of the following physical servers:

• 1 x86 SQL 2005 server running Windows Server 2003 SP2


• 1 MOSS application server hosting the MOSS Central Administration Web site and the query / index roles.


• 1 MOSS WFE server also hosting a MOSS Central Administration Web site for redundancy.
  

We host around 50 Web applications, each with their own content database.

The SQL 2005 server was very short on disk space (less than 20%) and, having considered a number of options including upgrading the disk capacity we decided that the simplest and safest (read "least risky") approach would be to purchase a new high capacity server and repurpose the existing SQL server (e.g. use it as an additional WFE server for redundnancy and performance).

We also decided that rather than install x86 Server 2003 and SQL Server 2005 on the new server, we would "future proof" the SQL back end by upgrading to x64 Server 2008 and SQL Server 2008.

As we would be repurposing the existing SQL server, we decided to go with the "Move all databases" approach documtned by Microsoft in this article: http://technet.microsoft.com/en-us/library/cc512725.aspx#Section1.

3. Gotchas

There are a few "gotchas" that I recommend you consider when planning to move to a new SQL server:

• There is no supported method of changing the name of the server that the MOSS Admin Content DB resides on. Instead, the documented MS resolution is to use SQL aliases to point MOSS WFE servers to the new SQL server (which in my experience seems to be very effective, but feels like a bit of a hack).
• The process for backing up and restoring MOSS SSPs is different to that used to move content databases.
• The entire farm must be stopped prior to moving any databases to prevent any synchronsiation issues. In my case downtime was around 3 hours.
• The SSP backup may fail if you have renamed any of your Web applications (see below).

With regard to the last issue, the error message I received when attempting to backup the SSP as a part of the migration was as follows: "Object Shared Search Index failed in event OnPrepareBackup." This is documented here: http://social.technet.microsoft.com/Forums/en-US/sharepointadmin/thread/9087ae96-c730-4b9f-85a4-67b4a4d67e65

The steps I took to resolve the SSP backup issue are contained in the appendix of this post, and are only relevenat if the SSP backup fails for you with the same error message.

4.1 High level steps

1. Backup all databases (including MOSS content and configuration databases)
2. Backup IIS on all WFE servers
3. Ensure integrity & dates of NTBackup file system backups on MOSS WFE servers (e.g. 12 hive, IIS directories)
4. Follow DB migration process outlined below
5. Stop SQL services on original DB server to ensure migration success
6. Test all portals
7. Add Backup & maintenance routines to SQL server
8. Modify any network or off-site backup routines your organisation may have to include the new server.

4.2 Detailed DB migration steps

Note that the steps below are basically a concise version of those contained at
http://technet.microsoft.com/en-us/library/cc512725.aspx with my own notes added in blue that I took whilst following them:

1. Follow http://technet.microsoft.com/en-us/library/cc263187.aspx to prepare the new DB server.
2. Record which Web applications are associated with the SSP
3. Back up an SSP by performing the following steps:
   
   On the drive on which SharePoint Products and Technologies is installed, change to the following directory:
   
   %COMMONPROGRAMFILES%\Microsoft shared\Web server extensions\12\Bin.
   
   If you do not already know which node you want to back up, type the following command:
   
   stsadm -o backup -showtree
   
   In our case the SSP name was "SharedServices". Note that the UNC directory must be a shared drive accessible to both the WFE and DB servers as opposed to a local path on the WFE server.
   
4. Back up each SSP by typing the following command:
   
   stsadm -o backup -directory -backupmethod full -item
   
   Ensure the shared drive ("UNC Path") is available to the SQL server - i.e. in the format (\\server name\folder name) as opposed to a directory local to the server. This should complete with 0 warnings and 0 errors. The account used to perform this action must have access to backup databases on the database server (e.g. domain Administrator account).
   
5. After backing up, remove each SSP by following these steps:
   
   On the disk on which Microsoft SharePoint Products and Technologies is installed, change to the following directory:
   
   %COMMONPROGRAMFILES%\Microsoft shared\Web server extensions\12\Bin.
   
   To remove an SSP, use the following command:
   
   stsadm -o deletessp -title -deletedatabases -force
   
   I believe that removing the content DB in central administration has the same effect.
   If using STSADM, check the content database name for the SSP Web application in central administration. In our case the SSP contentdb name was "WSS_Content_".
   
6. Delete each SSP content databases in SQL Management Studio.
7. Stop the farm by stopping all SharePoint services on all WFE servers (documented in MS article referenced above). Then, on each server that is running the Central Administration Web site, at the command prompt, type iisreset /stop.
8. Back up all databases on the existing (old) database server. I performed a full backup of all "User" databases, then copied them to a separate server within the farm for safety.
9. In Windows Explorer, locate the database backup (.bak) files that you want to move, and then copy or move them to the destination server.
   Important: Move only the backup files to the destination database server. Do not move any databases or other files at this time.
10. Restore all databases on the destination database server.
11. use http://support.microsoft.com/kb/918992/ to copy all SQL Server logins using a couple of stored procedures.
12. Refer the farm to the new database server by creating a SQL Server connection alias.
    Add aliases for both the FQDN and hostname.e.g. add hostname and hostname.domain.com. Perform this step on all MOSS servers in the farm.
13. Start the farm by starting all SharePoint services on all WFE servers (documented in MS article referenced above). Then, on each server that is running the Central Administration Web site, at the command prompt, type iisreset /start.
14. Obtain the GUID of each successful SSP backup:
    
    stsadm -o backuphistory -directory
    
15. Restore each SSP to the new location:
    
    stsadm -o restore -directory -restoremethod new -backupid -newdatabaseserver
    
    Again, Ensure the shared drive ("UNC Path") is available to the new SQL server - i.e. in the format (\\server name\folder name) as opposed to a directory local to the server. This process took around 10 minutes to complete. This should complete with 0 warnings and 0 errors. Note that this process can be rolled back (by deleting the restored SSP) using:
    
    stsadm -o deletessp -title -deletedatabases -force
    
16. Remove backup SSP directories.

5. Summary

In this post, we took a look at the process for migrating all databases from a SQL 2005 x86 server to a SQL 2008 x64 server. In future posts, I may go into more detail around the specification of the new server including decisions about the hardware choices and RAID configuration. Please post a comment if this post helped you!

6. Appendix: How to resolve SSP backup error "Object Shared Search Index failed in event OnPrepareBackup."

This is only appicable if you receive the error above when trying to backup a SSP. Note that this takes a LONG time if the SSP backup fails for multiple Web applications - you may decide to simply recreate the SSP rather than carry out this process but I chose this option as it seems to me like the least risky choice. In my case, this process took 10 hours for 23 Web applications but this time will vary depending on your experience with MOSS and number of Web parts you have.

1. Document the following MOSS Central Administration settings for each Web application that needs to be recreated to allow an SSP backup to succeed: "Policy for web application", "authentication providers", "Web application general settings" (timezone) settings, which Web parts that are deployed to which Web application, any people picker settings that need to be restored using STSADM.
2. Take the farm offline - in my case I simply created a rule to block all connections using ISA server 2006.
3. Backup IIS for all WFE servers.
4. Backup all MOSS DBs including content and configuration databases (you never know...).
5. Delete the Web application in SharePoint that failed to backup - do NOT delete the IIS Web site or content DB.
6. Ensure you have a full backup of the IIS Web site content - I used NTBackup. In particular, ensure you have a backup of the Web.config file as SharePoint overwrites this when you recreate the Web app.
7. Delete IIS Web site
8. Create new Web application with new IIS site & note down default (temporary ) content DB name (to be deleted)
9. IISreset /noforce
10. Remove temporary content db in MOSS Central Administration
11. Delete temporary content DB in SQL Server Management Studio.
12. Reattach correct content DB and confirm no. sites > 0- restore web.config from NTBackup-
13. Repeat the above for all sites that are reported when attempting to backup the SSP.
14. Backup IIS on all WFE servers again in case a manual virtual directory restore is required
15. Restore IIS backup on all WFE servers
16. Reconfigure the "Policy for web application", "authentication providers" and "Web application general settings" (timezone) settings for each restored Web application in MOSS central administration.
17. Redeploy web parts to all restored web applications
18. People picker settings for all restored web applications using STSADM.

Subscribe to the RSS feed

Follow me on Twitter

Follow my Networked Blog on Facebook

Add this blog to your Technorati Favourites