Administration: Find the needle in your (MOSS) event log haystack

Those that follow my blog may be expecting this post to be part 2 of my corporate security review series (part one is here). However, I figured I would save that post for next week and instead share something that I use on a regular basis whilst looking after our MOSS farm. I hope it's of use to you.

As a MOSS Server farm administrator, we get used to seeing a fair few common event IDs in Windows server event logs. Most of them are simply "informational" messages, but it's also common to see numerous warnings and errors. The challenge is knowing which events are a threat to your SharePoint installation and which ones are safe to ignore. As diligent server administrators we strive to remove each and every warning from the event log but in reality this can be a very difficult task in all but the smallest of farms.

To record common events, I keep a log handy (no prizes for guessing that it takes the form of an Excel spreadsheet stored within a SharePoint document library). I have included the most common issues contained in this list below. Note that although I have compiled this list based on my own experiences and research (read "googling"), I can't guarantee that the resolution will work for you as every SharePoint installation is different.

Please take a look and let me know if any of these look familiar:

Common MOSS related Windows Event IDs:

• 6398, 7076 and 6482 (Errors): "Unhandled Exception: System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt."
• Hay: No direct issues to my knowledge but 3 errors at once is an eyesore in your Windows event log. Apply the hotfix.

These three errors tends to show up together every hour or so and can certainly look rather worrying in the Windows event log. In my experience I haven't noticed a problem that has occurred directly as a result of this issue (supposedly a compatibility issue between a .NET 2.0 remoting application and Windows Server 2003 SP2), but I would certainly recommend downloading and installing the MS hotfix contained in the link below.

Permanent resolution: http://support.microsoft.com/kb/923028

Temporary resolution: restart restart the "Windows SharePoint Services Administration" & "Windows SharePoint Services Timer" services.

• 6875 (Error): "Error loading and running event receiver
  Microsoft.SharePoint.Publishing.Internal.ScheduledItemEventReceiver in
  Microsoft.SharePoint.Publishing"
• Hay: An informational error message (thanks Microsoft) - safe to ignore until you deploy MOSS SP2 which includes the hotfix.

This used to appear for us whenever we published a page in SharePoint when major versioning was turned on (which I believe is turned on by default within a publishing portal). As you might expect, the error cropped up a lot. The good news is that it is safe to ignore as the message is, according to MS an "informational" error. The best news? The hotfix is contained with MOSS SP2 so the error should disappear once you get around to deploying the service pack.

Permanent resolution: http://support.microsoft.com/kb/948323

• 7078 (Warning): "Job 'User Profile Incremental Import Job' failed. It will be re-tried again in 240 second(s). "
• Hay: Due to SharePoint starting multiple jobs at once - check your timer job definitions in Central Administration for conflicts but otherwise safe to ignore.

This one occurs for us approximately once per week at varying times and in my opinion is safe to ignore (after all, the warning does state that the job will rerun). There doesn't appear to be a documented resolution from Microsoft thus far.

Explanation: http://social.technet.microsoft.com/Forums/en-US/sharepointadmin/thread/5353c169-c713-40d0-972c-fd24a1d64184/

• 2436 (Warning): "The start address cannot be crawled "
• Needle:Typically due to incorrectly defined start addresses - check Central Administration.

For us, this was simply a case of human error and was very straightforward to correct within Central Administration > > Content Sources > Edit Content Source. In our case, we ensured that our start addresses matched the default URL's defined within "Alternate Access Mappings".

• 1113 (Error): "One of the IP/Port combinations for site 'ID' has already be configured to be used by another site. The other site's SSL configuration will be used. "
• Hay, so long as your IIS sites are up and running: A bug introduced by a new feature in Windows Server 2003 - safe to ignore.

If you host multiple Web sites on the same IP address and use host headers to differentiate your sites in IIS 6.0 (common for SharePoint installations), you may find (if using Windows Server 2003) that this error appears in your log whenever you perform an IIS reset. From what I gather, this is because prior to SP1, server 2003 could not handle host headers over SSL as host headers are encrypted and IIS had a hard job knowing which site to redirect the encrypted request to (a good example of the "chicken and egg" problem). Thankfully, the smart people over at Microsoft provided support for "wildcard" SSL certificates in SP1 but forgot to remove the error message above. I do not know of a way of removing the error but I am confident that this should not be a problem for you so long as your IIS is configured correctly.

Note that in cases where your IIS 6.0 installation is not configured correctly, you will certainly know about it if the problem is related to SSL bindings as your Web sites will effectively become unavailable, especially if publishing SharePoint via a secure proxy such as ISA Server 2006.

Explanation: http://paldev.net/blogs/hafanah/archive/2009/08/17/hosting-multiple-websites-applications-on-iis-part-1.aspx

• 1310 (Warning): "A configuration error has occured: duplicate "System.Web.extension" tags"
• Needle: Remove the duplicate web.config tags.

This used to occur for us a regular basis whilst deploying SharePoint Web applications. I never quite put my finger one what caused this problem but the fix was as simple as removing the offending duplicate tags from the applications web.config file. In the case above, SharePoint duplicated the "System.Web.Extensions" tags rendering the site inaccessible.

That concludes today's post. Let me know if you find any "needles" in your MOSS event log haystack that I haven't covered.

Subscribe to the RSS feed

Follow me on Twitter

Follow my Networked Blog on Facebook

Add this blog to your Technorati Favourites

Security: Preparing your MOSS farm for a corporate security review (part 1 of 3)

I would like to begin this post by thanking those that have recently subscribed to this blog via RSS. I always have a hard time deciding whether a blog should consume valuable space within my Google RSS reader so greatly appreciate the thought.

This post is primarily intended for those hosting or planning to host secure MOSS Web applications, although most of these points could also be applied to most Web applications that host sensitive content over the security minefield that is the Internet.

Having recently been through numerous corporate security reviews with potential clients, I thought it would be interesting to share some of the common queries that I get asked, along with an explanation and indication as to what constitutes an "acceptable" answer in terms of security. I imagine a lot of MOSS system administrators that host secure, Internet facing Web applications have been through the numerous security related hoops described below (including, in most cases a "PenTest" - I will detail preparation for that in another post). I thought it would be useful to document some of the most common queries.

Please note that the views expressed herein are my personal views and are not intended to reflect the views of my employer at the time of writing.

Without further ado:

1. Access Controls

• Q: How many different access levels are there for your application?
• A: 2 levels of access control is normally sufficient.

This one depends on the nature of your application. In most cases, an externally accessible MOSS Web application will contain multiple access levels. Similarly, in cases where said MOSS Web application is third party hosted, clients may not have access to carry out potentially dangerous functions that only an "administrator" should be able to perform. Typically, most users will have either "read" or "contribute" access, whereas power users might have "design", or even "full control" for some sites.

Security conscious organisations look for multiple access levels to allow for SoD. In my limited experience, the larger the organisation, the tighter the desired level of access control and, in MOSS terms the more users with "read" access. Having two clearly defined access levels is normally sufficient to satisfy most clients requirements but expect to implement more if your client(s) wish to implement access controls similar to their own organisational structure.

• Q: Are users configured with individual usernames and passwords?
• A: A must have for secure sites with auditing.

Pretty simple - in a secure environment where auditing is required, individual logons are a necessity rather than a luxury and your clients should demand it.

• Q: Is a password policy enforced, including forced password changes?
• A: Enforce a strong password policy; forced password changes are recommended.

We are forever told that choosing a secure password is a must when using sensitive applications - especially those hosted over the Internet. However, we as users can't be relied upon to resist entering a password any more secure than "chocolate123" or "admin1982". This seems to be a pet hate of large corporations, and rightly so. Enforce a secure password policy - at a minimum add some page level validation; preferably force the complexity requirement within your directory structure (e.g. Active Directory).

Security conscious organisations often look for forced password changes. However, it is not always required and this requirement does depend on the sensitivity of the application along any mitigating factors (e.g. a strong password policy and account lockouts, see below). In my experience, it is not essential.

• Q: Does the application support account lockouts?
• A: Enforce account lockouts with a realistic threshold

Automated account lockouts represent a basic, but effective measure against brute force attacks and a mitigating factor where password changes are not forced. I would go as far to say that it is an essential feature for applications that contain sensitive data - and so will your clients.

With regard to the threshold, many organisations will try to stick with their own internal account lockout policy. Commonly requested thresholds based on my experience are between 3 and 5, so I would recommend aiming for something around that area. For applications that do not require such a strict threshold, you may consider increasing this to a number between 6 and 10. This still provides some protection against brute force attacks whilst removing any frustration users might have over account lockouts.

• Q: Is there a documented process for revocation of a users access?
• A: Provide a clear, documented process that minimises chance of human error

In a lot of cases, I find that procedures for providing access are clearly defined and documented. Staff know exactly what is required in order to grant a new user access to a secure site. However, this is not always the case when it comes to revoking a users access, and is often left unchecked. If a user can't log in, your support staff will know about it soon enough. If they can log in after their account has "expired", you may find that you aren't ever notified there is an issue.

Reassure clients that you have a clear process and implement it. If you receive notification that a users access should be revoked by a set date, enforce account expiration in your directory structure and add a suitable description. In MOSS, you could go one step further and create a list of accounts marked for revocation.

Thats all for today. Join me soon for part 2. Still to come: Auditing, Change Management, Data Protection, Disaster Recovery, Infrastructure and Physical Security.

Subscribe to the RSS feed

Follow me on Twitter

Follow my Networked Blog on Facebook

Add this blog to your Technorati Favourites

Administration: Migrating to a new SQL Server in MOSS (SQL 2005 to 2008)

1. Overview

In this post I will describe the steps I took to successfully migrate from a SQL 2005 server to a brand new SQL 2008 server within a medium size production MOSS farm serving around 2000 users. The process makes use of the Microsoft supported method of migrating to a new DB server: SQL connection aliases.

As always, I strongly recommend you follow the steps below in a test environment prior to carrying them out in a live farm.

2. Scenario

Prior to the migration, we had a medium sized MOSS farm consisting of the following physical servers:

• 1 x86 SQL 2005 server running Windows Server 2003 SP2


• 1 MOSS application server hosting the MOSS Central Administration Web site and the query / index roles.


• 1 MOSS WFE server also hosting a MOSS Central Administration Web site for redundancy.
  

We host around 50 Web applications, each with their own content database.

The SQL 2005 server was very short on disk space (less than 20%) and, having considered a number of options including upgrading the disk capacity we decided that the simplest and safest (read "least risky") approach would be to purchase a new high capacity server and repurpose the existing SQL server (e.g. use it as an additional WFE server for redundnancy and performance).

We also decided that rather than install x86 Server 2003 and SQL Server 2005 on the new server, we would "future proof" the SQL back end by upgrading to x64 Server 2008 and SQL Server 2008.

As we would be repurposing the existing SQL server, we decided to go with the "Move all databases" approach documtned by Microsoft in this article: http://technet.microsoft.com/en-us/library/cc512725.aspx#Section1.

3. Gotchas

There are a few "gotchas" that I recommend you consider when planning to move to a new SQL server:

• There is no supported method of changing the name of the server that the MOSS Admin Content DB resides on. Instead, the documented MS resolution is to use SQL aliases to point MOSS WFE servers to the new SQL server (which in my experience seems to be very effective, but feels like a bit of a hack).
• The process for backing up and restoring MOSS SSPs is different to that used to move content databases.
• The entire farm must be stopped prior to moving any databases to prevent any synchronsiation issues. In my case downtime was around 3 hours.
• The SSP backup may fail if you have renamed any of your Web applications (see below).

With regard to the last issue, the error message I received when attempting to backup the SSP as a part of the migration was as follows: "Object Shared Search Index failed in event OnPrepareBackup." This is documented here: http://social.technet.microsoft.com/Forums/en-US/sharepointadmin/thread/9087ae96-c730-4b9f-85a4-67b4a4d67e65

The steps I took to resolve the SSP backup issue are contained in the appendix of this post, and are only relevenat if the SSP backup fails for you with the same error message.

4.1 High level steps

1. Backup all databases (including MOSS content and configuration databases)
2. Backup IIS on all WFE servers
3. Ensure integrity & dates of NTBackup file system backups on MOSS WFE servers (e.g. 12 hive, IIS directories)
4. Follow DB migration process outlined below
5. Stop SQL services on original DB server to ensure migration success
6. Test all portals
7. Add Backup & maintenance routines to SQL server
8. Modify any network or off-site backup routines your organisation may have to include the new server.

4.2 Detailed DB migration steps

Note that the steps below are basically a concise version of those contained at
http://technet.microsoft.com/en-us/library/cc512725.aspx with my own notes added in blue that I took whilst following them:

1. Follow http://technet.microsoft.com/en-us/library/cc263187.aspx to prepare the new DB server.
2. Record which Web applications are associated with the SSP
3. Back up an SSP by performing the following steps:
   
   On the drive on which SharePoint Products and Technologies is installed, change to the following directory:
   
   %COMMONPROGRAMFILES%\Microsoft shared\Web server extensions\12\Bin.
   
   If you do not already know which node you want to back up, type the following command:
   
   stsadm -o backup -showtree
   
   In our case the SSP name was "SharedServices". Note that the UNC directory must be a shared drive accessible to both the WFE and DB servers as opposed to a local path on the WFE server.
   
4. Back up each SSP by typing the following command:
   
   stsadm -o backup -directory -backupmethod full -item
   
   Ensure the shared drive ("UNC Path") is available to the SQL server - i.e. in the format (\\server name\folder name) as opposed to a directory local to the server. This should complete with 0 warnings and 0 errors. The account used to perform this action must have access to backup databases on the database server (e.g. domain Administrator account).
   
5. After backing up, remove each SSP by following these steps:
   
   On the disk on which Microsoft SharePoint Products and Technologies is installed, change to the following directory:
   
   %COMMONPROGRAMFILES%\Microsoft shared\Web server extensions\12\Bin.
   
   To remove an SSP, use the following command:
   
   stsadm -o deletessp -title -deletedatabases -force
   
   I believe that removing the content DB in central administration has the same effect.
   If using STSADM, check the content database name for the SSP Web application in central administration. In our case the SSP contentdb name was "WSS_Content_".
   
6. Delete each SSP content databases in SQL Management Studio.
7. Stop the farm by stopping all SharePoint services on all WFE servers (documented in MS article referenced above). Then, on each server that is running the Central Administration Web site, at the command prompt, type iisreset /stop.
8. Back up all databases on the existing (old) database server. I performed a full backup of all "User" databases, then copied them to a separate server within the farm for safety.
9. In Windows Explorer, locate the database backup (.bak) files that you want to move, and then copy or move them to the destination server.
   Important: Move only the backup files to the destination database server. Do not move any databases or other files at this time.
10. Restore all databases on the destination database server.
11. use http://support.microsoft.com/kb/918992/ to copy all SQL Server logins using a couple of stored procedures.
12. Refer the farm to the new database server by creating a SQL Server connection alias.
    Add aliases for both the FQDN and hostname.e.g. add hostname and hostname.domain.com. Perform this step on all MOSS servers in the farm.
13. Start the farm by starting all SharePoint services on all WFE servers (documented in MS article referenced above). Then, on each server that is running the Central Administration Web site, at the command prompt, type iisreset /start.
14. Obtain the GUID of each successful SSP backup:
    
    stsadm -o backuphistory -directory
    
15. Restore each SSP to the new location:
    
    stsadm -o restore -directory -restoremethod new -backupid -newdatabaseserver
    
    Again, Ensure the shared drive ("UNC Path") is available to the new SQL server - i.e. in the format (\\server name\folder name) as opposed to a directory local to the server. This process took around 10 minutes to complete. This should complete with 0 warnings and 0 errors. Note that this process can be rolled back (by deleting the restored SSP) using:
    
    stsadm -o deletessp -title -deletedatabases -force
    
16. Remove backup SSP directories.

5. Summary

In this post, we took a look at the process for migrating all databases from a SQL 2005 x86 server to a SQL 2008 x64 server. In future posts, I may go into more detail around the specification of the new server including decisions about the hardware choices and RAID configuration. Please post a comment if this post helped you!

6. Appendix: How to resolve SSP backup error "Object Shared Search Index failed in event OnPrepareBackup."

This is only appicable if you receive the error above when trying to backup a SSP. Note that this takes a LONG time if the SSP backup fails for multiple Web applications - you may decide to simply recreate the SSP rather than carry out this process but I chose this option as it seems to me like the least risky choice. In my case, this process took 10 hours for 23 Web applications but this time will vary depending on your experience with MOSS and number of Web parts you have.

1. Document the following MOSS Central Administration settings for each Web application that needs to be recreated to allow an SSP backup to succeed: "Policy for web application", "authentication providers", "Web application general settings" (timezone) settings, which Web parts that are deployed to which Web application, any people picker settings that need to be restored using STSADM.
2. Take the farm offline - in my case I simply created a rule to block all connections using ISA server 2006.
3. Backup IIS for all WFE servers.
4. Backup all MOSS DBs including content and configuration databases (you never know...).
5. Delete the Web application in SharePoint that failed to backup - do NOT delete the IIS Web site or content DB.
6. Ensure you have a full backup of the IIS Web site content - I used NTBackup. In particular, ensure you have a backup of the Web.config file as SharePoint overwrites this when you recreate the Web app.
7. Delete IIS Web site
8. Create new Web application with new IIS site & note down default (temporary ) content DB name (to be deleted)
9. IISreset /noforce
10. Remove temporary content db in MOSS Central Administration
11. Delete temporary content DB in SQL Server Management Studio.
12. Reattach correct content DB and confirm no. sites > 0- restore web.config from NTBackup-
13. Repeat the above for all sites that are reported when attempting to backup the SSP.
14. Backup IIS on all WFE servers again in case a manual virtual directory restore is required
15. Restore IIS backup on all WFE servers
16. Reconfigure the "Policy for web application", "authentication providers" and "Web application general settings" (timezone) settings for each restored Web application in MOSS central administration.
17. Redeploy web parts to all restored web applications
18. People picker settings for all restored web applications using STSADM.

Subscribe to the RSS feed

Follow me on Twitter

Follow my Networked Blog on Facebook

Add this blog to your Technorati Favourites