Archive for the ‘FAST for SharePoint 2010’ Category

The FAST Search for SharePoint Sam Worker service entered the stopped state

November 2, 2012

We recently stumbled on a Windows Update patch that knocked out our FAST Search for SharePoint 2010 farm. We got the resolution from the “Install a software update (FAST Search Server 2010 for SharePoint)” technet page, and we needed to run the script:

<FASTSEARCH>\installer\scripts\psconfig.ps1 –action p

This script stops the FAST for SharePoint 2010 services, upgrades the FAST database schema, and restarts the services.

Advertisements

Error replacing self-signed search certificate for FAST – Could not set access rights on certificates private keys

October 31, 2011

I have a test environment with SharePoint 2010 and FAST installed, and am using the self-signed certificates to enable FAST to communicate with SharePoint and vice-versa. My self-signed certificate expired, so I generated a new one using:

.\ReplaceDefaultCertificate.ps1 -generateNewCertificate $true

When I then tried to secure the FAST search connector via the Powershell command:

.\securefastsearchconnector.ps1 -certPath "" -ssaName "FASTSSA" -userName "<user name for osearch14"

I got an error when the script was trying to set Read/Execute permission on the certificate private key:

Could not set access rights on certificates private keys. Script can be rerun to only set access rights when reason for error is detected.

as well as:

Set-Acl : Attempted to perform an unauthorized operation

I tried several different methods to resolve, including setting the ownership of the file in script use TAKEOWN, but it wouldn’t work. Eventually I had to hack the securefastsearchconnector.ps1 file and comment out the line that set the ACL, as well as printing out what it was trying to do:

write-host "setting ACL for " $keypath$keyname " to " $script:userName
#set-acl -aclobject $acl $keypath$keyname

the output was then:

setting ACL for [full path to cert private key] to [user account of osearch14]

This effectively bypasses the setting of the ACL, and I then used the full path of the cert private key to manually add Read/Execute permission to my osearch14 account.

HOWTO Create a Content Index for a Host Header Site Collection

September 20, 2011

We have an environment with several web applications that have a number of host header site collections attached to them. This reduces the resources required by your server (you have fewer physical IIS web applications) while still allowing you to have 100s/1000s/lots of site collections each with their own URL.

We wanted to be able to search these site collections, but host header site collections cannot be searched on their own. Your content indexer has to access the site collections via the web application itself. As an example, lets assume you have a web application that resolves to “my.sharepoint.local” and you create a host-header site collection “site.sharepoint.local” using the following powershell script:

$formsContentDbName = “WSS_Content_SiteLocal”
$webName = “my.sharepoint.local”
$url = “http://site.sharepoint.local”

# Get the web application
$webApp = Get-SPWebApplication –Identity $webName

# Create content database
New-SPContentDatabase –Name $formsContentDbName -WebApplication $webApp

# Create the site
$contentDb = Get-SPContentDatabase –Identity $formsContentDbName
New-SPSite –Url $url –OwnerAlias “myalias” –OwnerEmail “my.email@somewhere.com” –ContentDatabase $contentDb –HostHeaderWebApplication $webApp –Template “STS#0”

You will now be able to access the host-header site collection via http://site.sharepoint.local.

If you create a content source however using the url “http://site.sharepoint.local&#8221; and run a full crawl, you’ll get the following warning in your crawl log if you don’t have a site collection attached to http://my.sharepoint.local:

This URL is part of a host header SharePoint deployment and the search application is not configured to crawl individual host header sites. This will be crawled as a part of the host header Web application if configured as a start address

As the warning states, the trick here is to create a site collection for the web application, in this case my.sharepoint.local. Then in the content source add only the url http://my.sharepoint.local. Your content source will then happily index the in your host header site collection, as well as the base site collection.

This however has some implications. Lets assume you have a large site collection (say 100GB) and a smaller site collection (say 10MB). They are both attached to a web application. You cannot do a full reindex of the smaller site collection without also reindexing the larger site collection.

I’ve raised a support incident with Microsoft to try and find a resolution. Stay tuned…

Error administering FAST Search Server: QI for IEnumVARIANT failed on the unmanaged server

June 14, 2011

This error started hitting my SharePoint 2010 Central Admin servers in one of our DMZ environments. As soon as I clicked on the “Content Sources” link in my FAST Connector Search Service Application (SSA) this error popped up almost immediately:

Error QI for IEnumVARIANT failed on the unmanaged server.

I checked to see if my Central Admin server could connect to one of the FAST admin service URLs that is used to manage content collections:

http://%5Byour fast server]:13257/contentcollectionservice.svc

and I got a standard WCF “I’m here!” service summary page response. So connection to the FAST server was OK.

I then checked my FAST server logs in:

[path to FAST]\var\log\syslog\lasterror.log

and found an entry for a timeout connecting to the database server.
I then checked the database server and found 1,000s of event log errors (a bit of background – it’s a brand new environment and had been installed/configured a month ago, then not touched since then). The errors in the log file were:

SSPI handshake failed with error code 0x80090311, state 14 while establishing a connection with integrated security; the connection has been closed. Reason: AcceptSecurityContext failed. The Windows error code indicates the cause of failure. [CLIENT: ].

I found out that the Security Configuration Wizard had been run on all servers, and Group Policy was in effect on the whole environment. This had closed off some required ports and caused everything to fall in a heap. I disabled the group policies and used the command:

gpupdate /force

to apply the policy. This stopped the errors occurring and returned everything back to normal again. Once I get it to a completely stable point, i’ll start to adjust and reapply the security policies.

SharePoint 2010 Search Administration Correlation Error when Accessing Content Sources

February 15, 2011

This occurred on a medium SharePoint 2010 web farm using FAST for SharePoint 2010. When clicking on the Content Sources I would get an error with a correlation id but no matching entry in any of the log files. I used the ULS log viewer to check real-time events that were being created and found the following entry:


0x80040d1b000007FEE0FB3B26d:\office\source\search\native\gather\gthrsvc\cgatherprj.cxx3278

This points to an error in the Gatherer service of SharePoint. I used Service Manager in Windows to stop the following services:

SharePoint Server Search 14
SharePoint Foundation Search V4

and then restarted them in the following order:

SharePoint Foundation Search V4
SharePoint Server Search 14

The errors disappeared from my ULS log viewer and I was able to access the search administration components again, in particular the content sources.

Microsoft FAST for SharePoint 2010 Local Security Groups

January 31, 2011

Microsoft FAST for SharePoint 2010 creates a local group called “FASTSearchAdministrators”. Members of this group are able to fully administer the FAST server farm.

An additional local group that can be created manually is the “FASTSearchKeywordAdministrators” group. Members of this group are able to administer FAST keywords.

SharePoint 2010 Content Source loading very slow in Central Admin Website

January 25, 2011

******************
UPDATE JUNE 10TH 2011

This issue has been resolved and confirmed – included in April 2011 Cumulative Update, as well as Service Pack 1.

The following PowerShell property has been added to the Search Service Application. This property will allow you to set the number of days to retain the crawl log history:

CrawlLogCleanUpIntervalInDays

Example PowerShell to set the property:

1. Run the following to get a list of the search service application Ids.

Get-SPServiceApplication | Where {$_.TypeName -eq "Search Service Application"}

2. Run the following command to get the Search Service Application Object. Replace the Id with the Id of your search application that was output in the previous PowerShell command.

$searchApp = Get-SPServiceApplication | Where {$_.Id -eq "<your content service application id>"}

3. Set CrawlLogCleanUpIntervalInDays to 10

$searchApp.CrawlLogCleanUpIntervalInDays = 10
$searchApp.Update()

******************
UPDATE MAY 5TH 2011

MS Support have been in touch to say that the issue has been resolved as part of the April 2011 Cumulative Update, however I cannot see any mention of this issue being resolved in the Knowledge Base article itself. I have contacted them to confirm.

******************
UPDATE APRIL 6TH 2011

I believe the stored procedure in the Search Service Application DB (on my wizard-built install this is Search_Service_Application_DB_<GUID>) called proc_MSS_CleanupWithInterval is the procedure responsible for removing old history data after the hardcoded 90 days.

USUAL CAVEAT/NOTE/WARNING – MAY NOT BE DIRECTLY SUPPORTED BY MS – DANGER WILL ROBINSON etc etc

Running the following command in SQL Server on the Search Service Application DB:

exec [proc_MSS_CleanupWithInterval] @CleanupInterval=60

will remove all search crawl history items older than 60 days. I’ve tested this on one environment and it worked fine (and check that stored proc for the strangest bit of SQL I think I’ve seen… you have to see it to believe it!). IMHO you would be pretty safe running this manually as it goes through and deletes corresponding entries from:

MSSCrawlHistory

MSSCrawlComponentsState

MSSCrawlComponentsStateLog

MSSCrawlComponentsStateTemp

MSSCrawlHistoryLog

MSSCrawlStoresState

which neatly reduces the size of your crawl history table to a much more managable, and more importantly speedier, size. I still prefer the option of adding an index to the table myself – I like my crawl history! And besides which you may need to run analysis over crawl history data at some stage for troubleshooting purposes. But the proposed fix sounds like it will help with the central issue – slow loading of content source history.

******************
UPDATE 31-March-2011
My MS Support case has been updated – the issue will be addressed in the April 2011 CU for SharePoint 2010.
“The fix is to expose crawl log cleanup interval as property of a Search Service Application via powershell. Previously it is hardcoded to 90 days, which causes the slow loading of the listcontentsources.aspx when there is a large crawlhistory table. ”
I prefer the table index fix myself as that would cater for more aggressive search interval requirements, but I guess this works too.
******************

When using Central Administration to view content sources for our FAST Search Connector (Search Service Application) the content source history and content source pages takes a very long time to load. In November it took 40 seconds, in December it took 60 seconds, and now at the end of January it takes 100 seconds. I have five content sources made up of three Intranet sites and two very small static websites. Total content is around 20,000 items (pages/data/list items/etc). Two of the Intranet sites are set to index incrementally every 15 minutes, and the third is set to every 5 minutes (it needs to most up-to-date search results and changes the most).

Looking in the SharePoint 2010 log files there were several entries similar to the following:

12/24/2010 10:02:36.61 w3wp.exe (0x1B6C) 0x16F8 SharePoint Server Database fa43 High Slow Query Duration: 33749.6701396407 4fb4bc60-f4a4-4467-9d63-03dee577b04b

which isn’t hugely helpful but does point to the fact that something in SQL Server is taking A Long Time ™.

Using SQL Server Management Studio – Manager Activity Monitor (available via right-clicking on the Server Name in Ent Manager and selecting “Activity Monitor” underneath “Recent Expensive Queries”) you should find any long-running queries that SQL Server has logged. You can right-click on these queries to view the query text. On our server I found the following:

SELECT
max(a.CrawlID) as CrawlID ,
max(a.CrawlType) as CrawlType ,
max(a.ContentSourceID) as ContentSourceID ,
max(a.Status) as Status ,
max(a.StartTime) as StartTime ,
max(a.EndTime) as EndTime ,
sum(c.SuccessCount) as Success ,
sum(c.WarningCount) as Warning ,
sum(c.ErrorCount ) as Errors ,
sum(c.DeleteCount) as Deletes ,
sum(c.NotModifiedCount) as NotModified,
sum(c.SecurityOnlyCount) as SecurityUpdates ,
sum(c.LevelHighErrorCount) as TopLevelError
FROM dbo.MSSCrawlHistory a INNER JOIN @TEMP b
ON a.ContentSourceID = b.id INNER JOIN
(
select CrawlID,sum(SuccessCount)as SuccessCount,sum(WarningCount)as WarningCount,sum(ErrorCount)as ErrorCount,sum(DeleteCount) as DeleteCount,sum(SecurityOnlyCount) as SecurityOnlyCount,sum(SecurityOnlyErrorCount) as SecurityOnlyErrorCount,sum(NotModifiedCount) as NotModifiedCount,sum(LevelHighErrorCount) as LevelHighErrorCount from dbo.MSSCrawlComponentsState GROUP BY CrawlID
union
select CrawlID,sum(SuccessCount)as SuccessCount,sum(WarningCount)as WarningCount,sum(ErrorCount)as ErrorCount,sum(DeleteCount) as DeleteCount,sum(SecurityOnlyCount) as SecurityOnlyCount,sum(SecurityOnlyErrorCount) as SecurityOnlyErrorCount,sum(NotModifiedCount) as NotModifiedCount,sum(LevelHighErrorCount) as LevelHighErrorCount from dbo.MSSCrawlComponentsStateLog GROUP BY CrawlID
) c
ON a.CrawlID=c.CrawlID
AND
a.CrawlID IN
(SELECT TOP (@Count) e.CrawlID FROM dbo.MSSCrawlHistory e
WHERE e.ContentSourceID = a.ContentSourceID
ORDER BY e.StartTime DESC
)
WHERE a.StartTime IS NOT NULL
GROUP BY a.CrawlID,a.ContentSourceID,a.StartTime
ORDER BY a.StartTime DESC

You can also view the execution plan by right-clicking on the query as well:

Expensive sort taking up precious query time

Expensive sort operation taking up precious query time

The MSSCrawlHistory table stores a record for each successful/unsuccessful crawl attempted/completed on a content source, and is set to store a maximum of 90 days of crawl history. The MSSCrawlHistory table in the FAST Content database has no indexes on the field “StartTime”. As per the query above, the ORDER BY StartTime DESC hammers this query, and the situation gets worse as the size of data in the MSSCrawlHistory table increases. As an example, I have about 20,000 entries in this table and the query takes 100 seconds+. I should add I have a very aggressive content source incremental indexing happening (once per 5 minutes 12 hours per day) because one of our sites relies heavily on accurate data for searching.

PLEASE NOTE THE FOLLOWING CHANGES WERE MADE IN A TEST ENVIRONMENT – I have not cleared this with Microsoft as yet (support call in place) as the following scripts adds an Index to the MSSCrawlHistory table which in theory invalidates your warranty for SharePoint 2010. SO ONCE AGAIN DO NOT IMPLEMENT THIS IN PRODUCTION WITHOUT REALISING THE CONSEQUENCES!

While I can’t change the query that is being executed, I can add an Index to the SQL table to help speed things up a lot. This SQL script will add a DESCending index on StartTime to the MSSCrawlHistory table. This improved my query times from 100seconds to <1 second. This works a treat in my test environment, I am currently checking with Microsoft to see if this would become a supported fix/hotfix. Adding an index would only impact a table when there is a lot of data being added to it – however in this case the Crawl History table logs the results of crawls and is quite a small table. It should be reasonably safe to run – in theory 🙂

/* To prevent any potential data loss issues, you should review this script in detail before running it outside the context of the database designer.*/
BEGIN TRANSACTION
SET QUOTED_IDENTIFIER ON
SET ARITHABORT ON
SET NUMERIC_ROUNDABORT OFF
SET CONCAT_NULL_YIELDS_NULL ON
SET ANSI_NULLS ON
SET ANSI_PADDING ON
SET ANSI_WARNINGS ON
COMMIT
BEGIN TRANSACTION
GO
CREATE NONCLUSTERED INDEX IX_MSSCrawlHistory_StartTime ON dbo.MSSCrawlHistory
(
StartTime DESC
) WITH( STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
GO
ALTER TABLE dbo.MSSCrawlHistory SET (LOCK_ESCALATION = TABLE)
GO
COMMIT

Microsoft FAST Search Administration – Unexpected error occurred while communicating with Administration Service

September 13, 2010

UPDATE: 6-Oct-2011 Added info on installing FAST certificate on SharePoint 2010 servers

I have a SharePoint 2010 Enterprise installation that is using Microsoft FAST for SharePoint 2010 to index Intranet content. The search content is working successfully and content is being returned, but trying to administer any of the managed properties or crawled properties always displays an error. The errors occur when I visit the Search Administration area for FAST property management:

  1. Open the Central Admin website
  2. in Application Management, Service Applications, select “Manage service applications”
  3. Locate your FAST Query search service application and click the manage link (i.e. click on “FAST Query SSA”)
  4. In the left-hand menu, under Administration, click FAST Search Administration
  5. Clicking any of the links on this page may show the error “Unexpected error occurred while communicating with Administration Service”

Further investigation does not provide much additional support beyond this error. The areas that I got errors for were:

Property Management
Managed properties
Crawled property categories

Property Extraction
Manage property extraction

Spell Checking
Spell checking management

Additionally, under the SharePoint site, Site Settings, Site Collection Administration, the following also gave the same error:

FAST Search keywords
FAST Search site promotion and demotion

There are a couple of areas to check.

SharePoint Application Pool Account Permissions on FAST Admin Servers

The Central Admin site application pool account requires permission to connect to the FAST admin web service on your FAST admin server. To resolve this:

  1. On your FAST admin server, check that you have a local security group “FASTSearchAdministrators”
  2. Add the active directory account for your central admin app pool to the local security group on your FAST admin server “FASTSearchAdministrators”

FAST Service Account Permissions on FAST Service Applications in SharePoint

The Central Admin service applications may require permissions to be allocated to the app pool accounts.

  1. Open Central Admin website
  2. In Application Management, Service Applications, select Manage Service Applications
  3. Hilight your FAST Search Connector service application and click the “Permissions” button
  4. Check that the FAST service account has “Full Control” permissions assigned
  5. Click “OK”

SharePoint Certificate Installed on FAST

Follow the instructions in Enable queries from Microsoft SharePoint Server (FAST Search Server 2010 for SharePoint) to ensure the SharePoint certificate used to communicate is trusted by the FAST admin web service. Note that it states “Configure claims authentication”, but I have had to do this on claims-based sharepoint sites as well as “classic” windows authenticated sites. Thus I believe this is a necessary step to remove this error.

FAST Certificate Installed on SharePoint

On installation, FAST creates a PFX certificate that is used to secure communication between the SharePoint server and the FAST Admin server. To install this certificate on your SharePoint server(s):

  1. Copy the script securefastsearchconnector.ps1 from the FAST Search Server 2010 for SharePoint server to the SharePoint 2010 Server. The securefastsearchconnector.ps1 script is in the installation folder, under installer\scripts\
  2. Copy the certificate file FASTSearchCert.pfx from the FAST Search Server 2010 for SharePoint admin server to the SharePoint 2010 Server. The certificate file is in the installation folder, under data\data_security\cert\
  3. Open a Microsoft SharePoint 2010 Administration Shell with the Run as administrator option on the SharePoint 2010 Server. Navigate to the directory where you copied the securefastsearchconnector.ps1 script and run it, replacing the necessary parameters with the values for your environment. The domain and username should reflect the details of the user running the SharePoint Server Search 14 (OSearch14) service.
    .\SecureFASTSearchConnector.ps1 -certPath "path of the certificate\certificatename.pfx" -ssaName "name of your content SSA" -username "domain\username"

When prompted to enter the certificate password, enter the certificate password that you supplied when you ran the post-setup configuration of FAST Search Server 2010 for SharePoint.