I am helping develop some custom Remote Blob Storage services for SQL Server 2008 and SharePoint 2010 (Stepwise Enterprise Storage – imminent release!) and I wanted to be able to use the ULS log viewer, but I didn’t want to spam the SharePoint logs and our services don’t need to be installed on SharePoint itself.

I set up the Microsoft Enterprise Library Logging Application Block and the Rolling Flat File Trace Listener to do the job for me. The rolling flat file trace listener automatically creates a new file when the current file meets certain criteria, such as size or age.  Note the hard-coded [fileName="MyService-20200101-0000.log"], this is required because the ULS Log Viewer is hard-coded to look for files matching the format *-????????-?????.log i.e. MyService-yyyymmdd-HHmmss.log. You can set this to something different, but I thought this would be fairly safe.

First I added a Rolling Flat File Trace Listener:

<add name="Rolling Flat File Trace Listener"
fileName="MyService-20200101-0000.log"
footer=""
formatter="ULS SingleLine Text Formatter"
header=""
rollFileExistsBehavior="Increment"
rollInterval="Day"
rollSizeKB="2048"
timeStampPattern="-yyyyMMdd-HHmm"
traceOutputOptions="None"
listenerDataType="Microsoft.Practices.EnterpriseLibrary.Logging.Configuration.RollingFlatFileTraceListenerData, Microsoft.Practices.EnterpriseLibrary.Logging, Version=4.1.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35"
filter="All"
type="Microsoft.Practices.EnterpriseLibrary.Logging.TraceListeners.RollingFlatFileTraceListener, Microsoft.Practices.EnterpriseLibrary.Logging, Version=4.1.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35"
/>

then I need to make sure the format for the output was correct. ULS wants the following fields supplied:

•Timestamp: Equivalent to the “TimeGenerated” field in the “Application” event Log

•Process: the image name of the process logging its activity followed by its process ID (PID) between parentheses.

•TID: Thread ID

•Area: This maps the “Source” field in the “Application” event Log

•Category: this maps the “Category” field in the “Application” event Log

•EventID: A unique internal Event ID (which you can generate)

•Level: Logging level i.e. Information, Verbose, Error, etc.

•Message: Text message

•Correlation: may contain a link to the the EventID of another logged event, which again you can generate

so this is the formatter I used:

<add
template="{timestamp(local:dd/MM/yyyy HH:mm:ss.fff)} {tab}{appDomain} {tab}{win32ThreadId} {tab}{source} {tab}{category} {tab}{eventid} {tab}{severity} {tab}{message} - {priority} - {dictionary({key} - {value}|)}"           
type="Microsoft.Practices.EnterpriseLibrary.Logging.Formatters.TextFormatter, Microsoft.Practices.EnterpriseLibrary.Logging, Version=4.1.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35"           
name="ULS SingleLine Text Formatter" />

This can be done via Windows 2008 DNS server when creating a default primary authorative zone. Normally you would create a zone for mydomain.com and then add a host entry for “www”. This would however have the effect that the DNS server became authoratative for the entire mydomain.com domain, which I did not want as there are other mydomain.com addresses that our environment does not host.

The solution is to create a primary DNS zone matching the full URL, www.mydomain.com.

DNS primary zone entry

Then, you create a blank A record pointing to the internal IP address of your website:

Create a blank A record

This A record becomes the default entry for the zone, so internal requests for www.mydomain.com will resolve to your A-record, but any requests for other URLS in the mydomain.com space will be forwarded via normal DNS processes.

1. SharePoint central admin site (PUB01)
2. 2 x SharePoint web front-ends serving content (WFE01 and WFE02)
3. Radware load balance (LOAD01)

I’m mentioning these items because they impacted how my content deployment was not working, and also the resolution required.

What You Absolutely Need To Have

There is ample documentation on setting up Content Deployment, but from a high level you must have the following in place:

1. Your source Central Admin website must be able to access the destination Central Admin website, including setting the destination CA to accept content deployment. This includes the IP address and port of course!
2. The account you use to deploy content on the destination server must have the appropriate permissions
3. Your source and destination farms must be the same SharePoint version (right down to the Cumulate Update if applicable!)
4. The destination CA must have enough room to store the content in a temporary location (check hard drive space on the destination server that hosts the Central Admin site)
5. The destination CA must have the Microsoft SharePoint Foundation Web Application service “Started” on the server (destination CA uses this to check IIS configuration – more on this later)

Content Deployment Process

Content deployment goes through several high-level stages when it deploys content:

1. The Source CA packages up the content into one or more CAB files
2. The Source CA pushes the CAB files to the Destination CA, which stores them in a temporary location
3. The Destination CA validates the IIS site and destination site collection
4. The Destination CA unpacks the cab files and individually pushes the content into the destination site collection

The import log file shows the results of the attempted import and will display any warnings or errors that occurred.

System.NullReferenceException: Object reference not set to an instance of an object

In my environment, this error occurs very quickly after the data has been transferred to the destination farm. I dialed up the ULS diagnostic monitoring by editing the Monitoring, “Configure diagnostic logging”, “Category – Web Content Management/Content Deployment” setting to Verbose. Your ULS log file will then show much more detail about the process, including whether the CAB content files are being copied up properly from the source to the destination CA. Look for entries such as:

Upload file 'C:\SPDeploy\1c8b92e5-975c-4d30-b38e-04c25975e2bb\ExportedFiles9.cab' succeeded

The “System.NullReferenceException” error occurs very soon after the final CAB file has been deployed.

The thing that tripped me up was the load-balanced url I had set up in the Radware load-balancer and DNS settings. My destination URL DNS entry was pointing to the load-balancer virtual IP address, which then load-balanced requests to my two web front-end servers. My suspicion is that the destination Central Admin content deployment process checks IIS to see if it is configured correctly, and because it is traveling via a virtual IP address it doesn’t resolve to a local IIS metabase entry. In the ULS logs I saw entries such as:

Saving CachingSettings for SiteUrl 'http://your.sharepoint.url/'

immediately followed by the dreaded:

Failed import operation for Content Deployment job 'Remote import job for job with sourceID = 409e7be7-7694-496d-8469-eb472e5070f6'. Exception was: 'System.NullReferenceException: Object reference not set to an instance of an object.
at Microsoft.SharePoint.SPSite.PreinitializeServer(SPRequest request)
at Microsoft.SharePoint.SPWeb.InitializeSPRequest()
at Microsoft.SharePoint.SPWeb.get_AllProperties()
at Microsoft.SharePoint.Publishing.SiteCacheSettings..ctor(SPSite site)
at Microsoft.SharePoint.Publishing.SiteCacheSettingsWriter..ctor(SPSite site)
at Microsoft.SharePoint.Publishing.SiteCacheSettingsWriter.SaveCacheSettingsBeforeImport(String importSiteUrl)
at Microsoft.SharePoint.Publishing.Administration.ContentDeploymentJob.DoImport()'

I checked the “PreInitializeServer” code using ILSpy (an improved, and more importantly free, replacement for Reflector), which wasn’t doing anything particularly complex but does use the SPRequest object. My thinking at this point was that the CA was unable to connect to IIS – it possibly does an IIS metabase lookup to get configuration settings, and because it was being bounced to the load-balancer and then down to the web front-ends, this step was failing. This occurs immediately after the cab files have been uploaded to the destination server.

The fix I put in was to edit the “hosts.” file (C:\Windows\System32\Drivers\etc\hosts.) and add an entry for the CA destination server IP address against the URL I was publishing to. This forces the destination CA to use the local server instead of trying to go via the load-balancer. This won’t affect the content being served, the web front-ends are configured via the load-balancer, and the destination CA doesn’t actually participate in this process. If your destination CA is also a web front-end, this would also still work as the URL resolves properly anyway.

After I set up my hosts file and re-ran the content deployment, it all worked perfectly! At least, on this occasion

A second “fix” I have had to use is to create another host-header site collection on my destination CA. I didn’t need to use this new site collection – I just created a blank one (see powershell script below). I’m not sure what happens in the background, but when I next tried my content deployment, it immediately changed from FAILED to SUCCESS. It’s possible that the act of creating another site collection resets Something(tm) so that content deployment can connect correctly.

Additional Content Deployment Troubleshooting Tips

While trying to diagnose the problem I also came up against the following issues (in no particular order):

1. When creating a destination site collection, do not add a template to the site. In essence, you are creating an empty site collection, not a site collection with the blank or blank internet template. This breaks content deployment. I use the following Powershell script to auto-create a host-header (multi-tenancy) site collection:

   $contentDbName = “Your.Database.Name”
   $webName = “Your web app name”
   $url = “http://your.url”
   # Get the web application
   $webApp = Get-SPWebApplication –Identity $webName
   # Create content database
   New-SPContentDatabase –Name $contentDbName -WebApplication $webApp
   # Create the site
   $contentDb = Get-SPContentDatabase –Identity $contentDbName
   New-SPSite –Url $url –OwnerAlias “DOMAIN\gavin.mckay” –OwnerEmail “gavin.mckay@domain.local” –ContentDatabase $contentDb –HostHeaderWebApplication $webApp

   The important point here is not to use the “-Template <template name>” parameter in New-SPSite. This will break content deployment and you will get a stack of errors about being unable to overwrite content. Deployment failure!

2. If you recreate your site collection, you will need to delete your content deployment job and path from the source CA server. The destination site collection id is stored as part of the job/path, and if you recreate your destination site collection it will have a new id. Deployment failure!
3. If you delete your content database and recreate it, you will have to do an IISRESET on your CA and web front-ends to get them to resolve properly. You will recognise this by trying to connect to the destination URL and getting the following in your browser:
   “HTTP/1.1 200 OK Server: Microsoft-IIS/7.5 Date: Tue, 08 Nov 2011 02:41:04 GMT Connection: close”
   SharePoint is confused. Deployment failure!
4. If you have Office Web Apps enabled on your source site collection, but not on the destination site collection, you will get:’Microsoft.SharePoint.SPException’ : ‘Could not find Feature MobilePowerPointViewer.’
   Deployment failure! Use Site settings, Site collection features, to disable Office Web Apps in the source site collection, then start your deployment job again.
5. “The exception thrown was ‘Microsoft.SharePoint.SPException’ : ‘Unable to import the folder _catalogs/fpdatasources. There is already an object with the Id <Guid> in the database from another site collection.’”
   Deployment failure! This seems to occur if you have previously had a failed content deployment for a new site collection/content database.
   Using your destination CA remove the content database then re-add it again. This will restore the previous empty site collection ready for content deployment.
6. Warning “A minor version has been exported with no corresponding major version. It is possible that this item was unpublished then published again. If this item is meant to be published, publish a new version and export/import it again”
   This is reasonably straight-forward. Either indivudally (or via a batch job) publish a major version, or turn off major/minor version control for the list/library.
7. Warning “Cannot revert to the site definition version of this file”
   Currently unknown how to fix this…
8. Error “A file with the name [filename] already exists”
   Currently unknown how to fix this…

More (Excruciating) Detail

The Content Deployment jobs are stored in the source farm SharePoint_Config database. You can view the details in a SQL query via:

Select top 100

id

,classid

,parentid

,name

,status

,version

,properties

FROM [dbo].[Objects]

WHERE Properties like ‘%ContentDeploymentJobDefinition%’

The Properties field is an XML data string that defines the job information.

The requirements for enabling this are:

- Kerberos is enabled in your domain (Windows Server 2003 and up general support this automatically)

- You have application pools that use Domain accounts, insted of Local accounts

In this configuration, Kerberos uses the application pool identity to decrypt Kerberos tickets.

Viewing Logon Sessions in Event Viewer

You can check what authentication is being used by viewing the logon sessions in the Security event log. You should see entries logged with Event ID 4624 and a Task category of Logon. If NTML is being used, in the details you will see an entry like:

Detailed Authentication Information:
	Logon Process:		NtLmSsp
	Authentication Package:	NTLM
	Transited Services:	-
	Package Name (NTLM only):	NTLM V1
	Key Length:		0

Whereas if Kerberos is being used, the entry will be similar to:

Detailed Authentication Information:
	Logon Process:		Kerberos
	Authentication Package:	Kerberos
	Transited Services:	-
	Package Name (NTLM only):	-
	Key Length:		0

Viewing Kerberos Tickets

When a session is created to a Kerberos-enabled application, the client receives a Kerberos ticket that is used to authenticate the user’s account. In Windows 7 and Windows Server 2008, you can use the KLIST application to view Kerberos tickets from the command line. A typical output would be something like:

Current LogonId is 0:0x2510zzz

Cached Tickets: (2)

#0>     Client: gavin.mckay @ YOUR.DOMAIN.LOCAL
        Server: krbtgt/YOUR.DOMAIN.LOCAL @ YOUR.DOMAIN.LOCAL
        KerbTicket Encryption Type: RSADSI RC4-HMAC(NT)
        Ticket Flags 0x40e00000 -> forwardable renewable initial pre_authent
        Start Time: 11/4/2011 8:02:36 (local)
        End Time:   11/4/2011 18:02:36 (local)
        Renew Time: 11/11/2011 8:02:36 (local)
        Session Key Type: AES-256-CTS-HMAC-SHA1-96

#1>     Client: gavin.mckay @ YOUR.DOMAIN.LOCAL
        Server: ldap/SERVER1.YOUR.DOMAIN.LOCAL/YOUR.DOMAIN.LOCAL @ YOUR.DOMAIN.LOCAL
        KerbTicket Encryption Type: AES-256-CTS-HMAC-SHA1-96
        Ticket Flags 0x40a40000 -> forwardable renewable pre_authent ok_as_deleg
ate
        Start Time: 11/4/2011 8:02:36 (local)
        End Time:   11/4/2011 18:02:36 (local)
        Renew Time: 11/11/2011 8:02:36 (local)
        Session Key Type: AES-256-CTS-HMAC-SHA1-96

As Kerberos tickets are cached, you may need to use the command:

KLIST purge

to remove any current tickets and force new ones to be generated.

Create Service Principal Names (SPNs)

NOTE: In the commands below, the HTTP refers to the service type, NOT the protocol type. So if you have https://your.intranet.local, the entry below would be HTTP/your.intranet.local:443. The HTTPS is not included in the service name, but the port number is.

1. Use the command:

(Windows Server 2008)
setspn -S HTTP/your.intranet.local DOMAINNAME/AccountName

(Windows Server 2003)

setspn -A HTTP/your.intranet.local DOMAINNAME/AccountName

Enable Kernel Mode Authentication in IIS 7

From the IIS Manager description:

By default, IIS enables kernel-mode authentication, which may improve authentication performance and prevent authentication problems with application pools configured to use a custom identity. As a best practice, do not disable this setting if Kerberos authentication is used in your environment and the application pool is configured to use a custom identity.

1. Open IIS Manager
2. Select the website you want to enable Kernel Mode authentication for
3. In the IIS group, select Authentication
4. Select Windows Authentication
5. In the Actions area, select Advanced Settings
6. Ensure “Enable Kernel-mode authentication” is ticked
7. Click OK

Update the Authentication in applicationHost.config

1. Using Notepad, open the applicationHost.config (by default this is in C:\Windows\System32\inetsrv\config)

2. Find your website entry, generally this should be similar to:

<location path="your.sharepoint.url">

i.e. if you access your site via http://your.sharepoint.url, look for the location path as above

3. Update the tag <windowsAuthentication…> to:

<windowsAuthentication enabled="true" useKernelMode="true" useAppPoolCredentials="true">

4. Save the file

Additional Troubleshooting

Configuring Kerberos authentication: Core configuration (SharePoint Server 2010)
Kerberos Delegation Configuration Reporting Tool (ASP.NET web app)

.\ReplaceDefaultCertificate.ps1 -generateNewCertificate $true

When I then tried to secure the FAST search connector via the Powershell command:

.\securefastsearchconnector.ps1 -certPath "" -ssaName "FASTSSA" -userName "<user name for osearch14"

I got an error when the script was trying to set Read/Execute permission on the certificate private key:

Could not set access rights on certificates private keys. Script can be rerun to only set access rights when reason for error is detected.

as well as:

Set-Acl : Attempted to perform an unauthorized operation

I tried several different methods to resolve, including setting the ownership of the file in script use TAKEOWN, but it wouldn’t work. Eventually I had to hack the securefastsearchconnector.ps1 file and comment out the line that set the ACL, as well as printing out what it was trying to do:

write-host "setting ACL for " $keypath$keyname " to " $script:userName
#set-acl -aclobject $acl $keypath$keyname

the output was then:

setting ACL for [full path to cert private key] to [user account of osearch14]

This effectively bypasses the setting of the ACL, and I then used the full path of the cert private key to manually add Read/Execute permission to my osearch14 account.

Instead of:
HKEY_CURRENT_USER > Software > Policies > Microsoft > Communicator

I had to use:
HKU > S-1-5-21-[more long code here] > Software > Policies > Microsoft > Communicator

I discovered this by using the Windows Process Monitor and tracking what registry key it was looking for. A quick search for DisableServerCheck in the monitor found the appropriate key.

HTH

I tried to deploy FIM 2010 Cumulate Update 1 and it failed, citing “Cannot connect to SQL Server”. I had the nasty suspicion that while FIM 2010 seemed to support SQL Server aliases, the Cumulate Update process did not.

FIM 2010 maintains information on SQL Server connections in the Registry under the hives:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\FIMService
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\FIMSynchronizationService\Parameters

The FIM services use these keys to establish database connections. Likewise, the Cumulative Update process accesses these locations to perform database updates.

A final note – after changing to a SQL Server instance instead of an alias, I got some errors when trying to upgrade the database via the Cumulative Update 1:

25070 Invalid class string

Installing the latest SQL Server Native Client from the Microsoft SQL Server 2008 R2 SP1 Feature Pack fixed these errors.

We wanted to be able to search these site collections, but host header site collections cannot be searched on their own. Your content indexer has to access the site collections via the web application itself. As an example, lets assume you have a web application that resolves to “my.sharepoint.local” and you create a host-header site collection “site.sharepoint.local” using the following powershell script:

$formsContentDbName = “WSS_Content_SiteLocal”
$webName = “my.sharepoint.local”
$url = “http://site.sharepoint.local”

# Get the web application
$webApp = Get-SPWebApplication –Identity $webName

# Create content database
New-SPContentDatabase –Name $formsContentDbName -WebApplication $webApp

# Create the site
$contentDb = Get-SPContentDatabase –Identity $formsContentDbName
New-SPSite –Url $url –OwnerAlias “myalias” –OwnerEmail “my.email@somewhere.com” –ContentDatabase $contentDb –HostHeaderWebApplication $webApp –Template “STS#0”

You will now be able to access the host-header site collection via http://site.sharepoint.local.

If you create a content source however using the url “http://site.sharepoint.local” and run a full crawl, you’ll get the following warning in your crawl log if you don’t have a site collection attached to http://my.sharepoint.local:

This URL is part of a host header SharePoint deployment and the search application is not configured to crawl individual host header sites. This will be crawled as a part of the host header Web application if configured as a start address

As the warning states, the trick here is to create a site collection for the web application, in this case my.sharepoint.local. Then in the content source add only the url http://my.sharepoint.local. Your content source will then happily index the in your host header site collection, as well as the base site collection.

This however has some implications. Lets assume you have a large site collection (say 100GB) and a smaller site collection (say 10MB). They are both attached to a web application. You cannot do a full reindex of the smaller site collection without also reindexing the larger site collection.

I’ve raised a support incident with Microsoft to try and find a resolution. Stay tuned…

There are a number of solutions to this, but we felt the easiest way was to create replacement host-header site collections instead i.e. http://author-web1.local/ instead. This lets us create as many site collections as we want while still using the same physical web application http://author.local.

I backed up the site collection http://author.local/sites/web1 using Backup-SPSite, and then tried to restore the site collection using Restore-SPSite via the following:

Restore-SPSite -identity http://author-web1.local -path <physical path> -contentdatabase <contentdb> -hostheaderwebapplication http://author.local/

However I got the following error from powershell:

Restore-SPSite : The site collection could not be restored. If this problem persists, please make sure the content databases are available and have sufficient free space.
At line:1 char:15
+ Restore-SPSite <<<<  -identity http://author-web1.local -path 'C:\site.bak' -contentdatabase MyDb -hostheaderwebapplication http://author.local
+ CategoryInfo          : InvalidData: (Microsoft.Share...dletRestoreSite:
SPCmdletRestoreSite) [Restore-SPSite], SPException
+ FullyQualifiedErrorId : Microsoft.SharePoint.PowerShell.SPCmdletRestoreS
ite

ULS reported the following error in the log files:

System.Data.SqlClient.SqlException: Violation of UNIQUE KEY constraint 'Docs_IdLevelUnique'. Cannot insert duplicate key in object 'dbo.AllDocs'.  The statement has been terminated.

at Microsoft.SharePoint.Utilities.SqlSession.ExecuteReader(SqlCommand command, CommandBehavior behavior, SqlQueryData monitoringData, Boo

It turned out that I had a previous site collection from a failed restore attempt before. No problem, I deleted the site collection so there were none in there at all, then tried again. Same result. I checked in the content database, and there were indeed files in the AllDocs tables even though the site collection had been deleted. I went the Big Bang approach and deleted the content database. After creating a new content database, the restore worked perfectly.

My guess at this stage is that I may have been a victim of the site collection recycle bin! SP1 includes the ability to restore deleted sites and site collections via PowerShell, so there may still have been deleted references in the content database. Regardless, it works now. I don’t know how to fix this issue if you only have the one content database, but I always try and split out site collections in host-header web applications to different databases so this didn’t affect me in this case.

List-based workflows are tied to three different lists – the “main” source list where the data is held (such as a Forms library or custom list), a task list, and a workflow history list. The latter in particular is tricky, because it is a hidden list and cannot be viewed via the normal interface.

The fix for this is to modify the source workflow files to force the workflow to reattach to the list. The following method assumes that your new site/location that you are moving the workflow too does not yet have any workflows already attached to any lists on the site. When a workflow is created in a site for the first time, a Tasks list and a Workflow History list are automatically created/used by the workflow. These lists must exist before you can force your migrated workflow to attach to the list.

Step 1 – Create a Blank Workflow

1. Using SharePoint designer, connect to your destination site
2. Create a new list-based workflow called “test” (or whatever), attaching it to your migrated list
3. Create one condition (i.e. if 1 = 1)
4. Create one action (i.e. add comment “hello”)
5. Save and publish the workflow

After this step has been completed, your site will now have a Tasks list and Workflow History list.

Step 2 – Get the list ids for your new workflow

The list ids are required to configure your migrated workflow.

1. Using SharePoint Designer, connect to your destination site
2. in the left-hand side, in Site Objects, select All Files
3. in the All Files list, select Workflows
4. Select your new workflow that you just created (in this example, “test”)
5. You should see (at least) four files:
   text.xoml
   test.xoml.rules
   test.xoml.wfconfig.xml
   test.xsn
6. Right-click on test.xoml.wfconfig.xml and select Open With, Notepad
7. Look for the <Association…> tag in the xml:
   <Association ListID=”{7DC232FD-4D0B-4F7B-AC72-3D4D6399147C}” StartManually=”true” TaskListID=”{CB551A8B-F1E1-49F9-A8F8-A2C8EDC241C7}” HistoryListID=”{04B19F9D-5A47-4E14-B85E-53FFF06CFA63}” StartOnCreate=”true” StartOnChange=”true”/>
8. Record the Guid entries for ListID, TaskListID, and HistoryListID

Step 3 – Update the migrated workflow

1. Still using SharePoint Designer, All Files, Workflows, this time select your migrated workflow
2. You should again see (at least) four files. Right-click on “your-workflow.xoml.wfconfig.xml” and select Open with SharePoint Designer (As XML)
3. Find the Association tag and very carefully change the Guids so that your migrated List IDs are the same as the “test” workflow List IDs.
4. Save the file
5. Close SharePoint Designer
6. Open SharePoint Designer again and open your site
7. Instead of All Files, this time click on Workflows
8. Select your workflow
9. Save the workflow
10. Publish the workflow

Your workflow should now be reassociated with your list on your new site.