Result Source

1. Select the search service application, site collection, or site you wish to create the result source on (Figure 5-1).

   

   Figure 5-1. Result source—service application (left), site collection (center), site (right)

2. On the Manage Source page, click New Result Source.

3. On the Add Result Source page (Figure 5-2), complete the following:

   1. In the Name text box, enter a name of your choice for the result source (example: Office 365).

   2. For the Protocol, select Remote SharePoint.

   3. For the Remote Service URL, type the address of the root site collection of the Office 365 SharePoint Online tenant whose results should be included (example: https://Contoso.SharePoint.com ).

   4. For Credentials Information, ensure that the Default Authentication option is selected. This will pass the user’s UPN as a claim, along with the query for the result source.

   5. Leave the remainder of the option with their default settings.

   

   Figure 5-2. Outbound query result source

4. Click Save to save the new result source.

Query Rule

Triggering parallel queries for one, multiple, or all result sources allows you to promote results, add additional result locks, and change the results ranking.

1. Select the search service application, site collection, or site your wish to create the Query Rule on. If the result source was created at the Search Service Application, it can be created there or on a site collection or site (Figure 5-3).

   

   Figure 5-3. Result source—service application (left), site collection (center), site (right)

2. On the Manage Query Rules page, in the For what context do you want to configure rules? list box, select Local SharePoint Results (Figure 5-4).

   

   Figure 5-4. For what context do you want to configure rules?

3. Click New Query Rule.

4. On the Add Query Rule page, do the following:

   1. In the General Information section, in the Rule Name box, type a name for the new query rule (example: Office 365).

   2. If you would like the Query Rule to trigger for additional contexts, click the Context link to expand it.

      • After the Query is performed on these sources you can select All Sources, or add additional Context source with the Add Source Link

      • Leave the remainder of the settings in this section set to their defaults.

   3. In the Query Condition section, click the Remove Condition, to have the query rule run for every query.

   4. In the Action section, under Result Blocks, click Add Result Block.

   5. In the Add Result Block dialog box, complete the following:

      • In the Query section, in the Search this Source list box, select the name of the result source that you created earlier (example: Office 365). In the items list box section, the default number of items that will be displayed is 2. You can increase or decrease this value to control the number of items that will be displayed.

      • Click the Settings link to expand the section.

      • Ensure that This block is always shown above core results is selected.

      • No changes need to be made in the Routing section.

      • Click OK to add the result block.

   6. Back on the Add Query Rule page, click the Publishing link, to expand the section. Ensure the Is Active check box is selected.

   7. Click the Save button to save the result source.

Testing

To test this setup, be sure that you are using a user account that has permissions to the on-premises content and is licensed and permissioned on content in SharePoint Online and take the following steps:

1. Using this account, go to the site collection or search center that you configured your query rule on and issue a query (Figure 5-5).

   

   Figure 5-5. Outbound query results

   1. If you receive results for on-premises content and SharePoint Online content, you have set everything up successfully.

   2. If no results return for SharePoint Online, proceed to the next step.

2. Browse to a SharePoint Online search center and confirm that you get results.

   1. If you don’t get any results, be sure you have permissions on the content you are trying to query. If you don’t, update the permissions and wait a while for the content to be re-indexed.

   2. If do you do get results, proceed to the next step.

3. Navigate back to your result source page and from the result source context menu, click the test button (Figure 5-6).

   

   Figure 5-6. Outbound result source test

   1. If the test fails, troubleshoot the contents of the Test details section.

   2. If the test succeeds, proceed to the next step.

4. Navigate to your query rule and ensure that the settings are correct.

Inbound Hybrid Search

The Inbound Hybrid Search model is much like the Outbound Hybrid model. The difference is this time, instead of setting up the Results Source and Query Rule on the on-premises SharePoint farm, it will be set up in SharePoint Online. This setup is more complex than the Inbound Query Hybrid. This is because SharePoint Online must authenticate to the on-premises farm. In addition to configuring SharePoint, an Internet-facing end point for the on-premises farm must be created for SharePoint Online to be able to connect. A SharePoint Web Application must be published through a reverse application proxy. If you are using AD FS you can reuse the same Reverse Application Proxy server. A Secure Store Target Application will also be needed.

$config = Get-SPSecurityTokenServiceConfig
$config.AuthenticationPipelineClaimMappingRules.AddIdentityProviderNameMappingRule("OrgId Rule", [Microsoft.SharePoint.Administration.Claims.SPIdentityProviderTypes]::Forms, "membership", "urn:federation:microsoftonline")
$config.Update()

$ExternalCertPath = "c:CertsSharePoint.Contoso.com.pfx"
$ClientCertPath = "c:Certsuserauth.Contoso.com.pfx"
$ExternalURL = "https://SharePoint.Contoso.com"

<# If External and Internal URLs Please change the URL below
   are different please add
   -DisableTranslateUrlInRequestHeaders:$False and
   -DisableTranslateUrlInResponseHeaders:$False to the
   Add-WebApplicationProxyApplication cmdlet
 #>

$InteranlURL = $ExternalURL

#Get the thumbprint of the External URL Certificate

# Prompt for Certicate passwords
$ExternalCertPassword = Read-Host -Prompt "External Certificate Password" -AsSecureString
$ClientCertPassword = Read-Host -Prompt "Client Certificate Password" -AsSecureString

# Imports Certificates to the Local Computer Personal store
Import-PfxCertificate -FilePath $ExternalCertPath -Password $ExternalCertPassword -CertStoreLocation Cert:LocalMachineMy
Import-PfxCertificate -FilePath $ClientCertPath -Password $ClientCertPassword -CertStoreLocation Cert:LocalMachineMy

#Publish the Web Application
Add-WebApplicationProxyApplication `
-Name "Hybrid Inbound Rule" `
-BackendServerUrl $InteranlURL `
-ExternalUrl $ExternalURL `
-ExternalCertificateThumbprint $ExternalCert.Thumbprint `
-ExternalPreauthentication "ClientCertificate" `
-ClientCertificatePreauthenticationThumbprint $ClientCert.Thumbprint

Get-WebApplicationProxyApplication -Name "Hybrid Inbound Rule" | fl

SharePoint Online Result Source

To complete the setup of the inbound query, we will create a result source and query similar to the Outbound Query Rule. As it can on-premises, your result source can be created at three levels: Tenant Administration, Site Collection, and Site. Two changes must be made in this process.

The first is to the Remote Service URL. Instead of using the Office 365 root URL, you will use the URL of the externally published SharePoint Web Application (Figure 5-7).

Figure 5-7. SharePoint Online result source protocol

The second change required is to the Credentials Information section. Select the SSO Id radio button and in the Reverse proxy certificate (Secure Store Id) enter the ID of the secure store target application created in the preceding step (see Figure 5-8).

Figure 5-8. SharePoint Online result source credentials

Two-Way Hybrid Search

The Two-Way Hybrid Search model is a simplified way of seeing your on-premises farm has been configured with Outbound Query to SharePoint online and SharePoint Online has been configured with Inbound Queries to your on-premises farm. By completing the two previous configuration sections, you have set up Two-Way Hybrid Search.

Create Cloud Search Service Application

A Cloud Search Service is created in a similar manner to an Enterprise Search Service Application. The first change is when creating the Search Service Application: the CloudIndex property must be set to true. This property disables the ContentPlugin in the Crawler component that is responsible for routing items to the Content Processing component for indexing. After the Search Service is created, the IsHybrid property of the Search Service Application must be initialized. The initialization process will activate AzurePlugin to replace ContentPlugin. AzurePlugin batches crawled content and prepares it to be pushed to Azure.

Because content processing is done in the cloud, and the index is stored there as well, there is no reason to scale out the Content Processing and Index components in the Search Service Application. However, you must have one of each of these components in the Search Service topology. You have the option to scale out the Crawling, Query, Admin, and Analytics Processing components. You must have one of each of these components. Scaling them out to different servers in the on-premises farm will create high availability for their services.

The following script creates a Cloud Search Service Application with one of each component required for a Search Service topology. The script assumes the Service Application pool already exists.

# Ensure SharePoint PowerShell Snapin is loaded
Add-PSSnapin Microsoft.SharePoint.PowerShell -ErrorAction SilentlyContinue

# Set varabiles up for Service Application

$SSAName = "Cloud Search Service Application"
$SVCAcct = "ContosoSP_Farm"
$indexLocation = "E:CloudSearch"
$AppPoolName = "ServiceApplicationPool"
$DatabaseName = "SP2016_CloudSearch"

# Get the Service Application pool
$AppPool = Get-SPServiceApplicationPool -Identity $AppPoolName

# Get the local Search Service Instance for additional servers can be set to other vaiables
$SSI = get-spenterprisesearchserviceinstance -local

# Create the directory to store the index
New-Item $indexLocation -ItemType Directory

# Create Cloud Search Service Application
$SearchApp = New-SPEnterpriseSearchServiceApplication -Name $SSAName -applicationpool $AppPool -databasename $DatabaseName  -CloudIndex $true

# Create Service Application Proxy for Search Service Application
$SSAProxy = new-spenterprisesearchserviceapplicationproxy -name $SSAName -SearchApplication $SSAName

# Get the default Topology Object and create a new Topology object
$initialSearchTopology = $searchApp | Get-SPEnterpriseSearchTopology -Active
$newSearchTopology = $searchApp | New-SPEnterpriseSearchTopology

# Create Admin component, more then one component can be deployed to additional servers
New-SPEnterpriseSearchAdminComponent -SearchTopology $newSearchTopology -SearchServiceInstance $SSI

# Create Analytics component
New-SPEnterpriseSearchAnalyticsProcessingComponent -SearchTopology $newSearchTopology -SearchServiceInstance $SSI

# Create Crawl component, more then one component can be deployed to additional servers
New-SPEnterpriseSearchCrawlComponent -SearchTopology $newSearchTopology -SearchServiceInstance $SSI

# Create Content Processing Component
New-SPEnterpriseSearchContentProcessingComponent -SearchTopology $newSearchTopology -SearchServiceInstance $SSI

# Create Query Processing Component, more then one component can be deployed to additional servers
New-SPEnterpriseSearchQueryProcessingComponent -SearchTopology $newSearchTopology -SearchServiceInstance $SSI

# Create Index Component
New-SPEnterpriseSearchIndexComponent -SearchTopology $newSearchTopology -SearchServiceInstance $SSI -RootDirectory $indexLocation -IndexPartition 0

# Active the new Topology Object
Set-SPEnterpriseSearchTopology $newSearchTopology

# Default Topology Object
Remove-SPEnterpriseSearchTopology -Identity $initialSearchTopology

Onboarding the Search Service to SharePoint Online

Microsoft has made the onboarding process to connect a Cloud Search Service Application to your SharePoint Online service very simple. The script Onboard-CloudHybridSearch.ps1 is available on the Microsoft download site. Before you run this script, you must install two additional components on that server. If they are not present, the script will fail, requesting that the components be downloaded and installed. Most administrators will install these modules on their Central Administration Server(s), as these are the servers that do most of the administration of the farm. The prerequires, which can be downloaded from the Microsoft download site, are

1. Microsoft Online Services Sign-In Assistant

2. Microsoft Online Services Module

The file name you will want to search for on the Microsoft download site is CloudHybridSearchScripts.zip. It contains the Onboard-CloudHybridSearch.ps1, along with the CreateCloudSSA.ps1 script. The CreateCloudSSA.ps1 is another way to create a single server Cloud Search Service Application.

To complete the onboarding process, open a new PowerShell console and set the location to the directory containing the Onboard-CloudHybridSearch.ps1 script. You will have to provide the SharePoint Online portal URL or the Cloud Search Service Application name or GUID, if the farm has more than one Search Service Application.

The script starts by determining if a Search Service Application was provided. If not, and the service has only one Search Service Application, it will use it. The selected Search Service Application CloudIndex property will be confirmed, and the IsHybrid property will be set to true.

The tenant admin credentials are requested, if not provided at execution, and a connection is made to Microsoft Online. The tenant information and Azure Active Directory ID are retrieved from Microsoft Online.

The local SharePoint farm is configured next. The script determines if the Azure Access Control Service Application Proxy exists, and the script exits if it does, prompting for it to be manually removed. If it does not exist, the Azure Access Control Service Application will be created with the name ACS, and it is added to the default proxy group. The ACS Service Application Proxy is required to be in the default proxy group for authentication to work. The S2S Trust Broker is configured between the proxy and Azure Active Directory. A SharePoint Online Application Principal Management Service Application Proxy, named SPO App Management Proxy, is created, with a connection to the tenant SharePoint portal.

The security token local login certificate is uploaded to the Microsoft Online tenant as a new service principal credential for SharePoint Online. The Azure Search Connector Service URLs are added to a new Service Principal name in the Microsoft Online tenant as well. Then, the on-premises farm is connected to SharePoint Online.

The Microsoft Online Tenant is prepared for the Cloud Hybrid Search Service. This process can take up to four minutes. Service information is collected and validated and then configured on the Cloud Search Service Application. Finally, the SharePoint Timer Service is restarted.

On-Premises Content Source

The configuration of the content sources in a Cloud Search Service Application is the same as for a regular Enterprise Search Service Application. When crawling large start addresses, the status polling for the batches that have been submitted to Azure can be overly aggressive and trigger throttling responses from Azure. Setting the EnableNoGetStatusFlight property on the Search Service Application will stop the pooling and prevent you from getting throttled. This can be done with the following code. You must restart the SharePoint Host Controller for this property change to take effect.

$ssa=Get-SPEnterpriseSearchServiceApplication
$ssa.SetProperty("EnableNoGetStatusFlight",1)
$ssa.update()

The Search Gatherer Azure Plugin performance object in Performance Monitor can also be used to monitor the progress of your crawls. Another common bottleneck that can reduce crawl performance is the server’s uplink to the Internet. An example of this would be the web front end rendering the content for the crawler.

Testing

Once your first full crawl has completed, you are ready to test SharePoint Online. As in the case of federated query hybrid tests, you must ensure that you employ a user account with permission for on-premises content and that is licensed and has permissions for content in SharePoint Online. Navigate to your SharePoint Online Search Center and issue a query. You should see a mix of on-premises and SharePoint Online content returned (Figure 5-10).

Figure 5-10. Cloud Search Service query results

Document Previews

In SharePoint 2013, with the Office Web Application Server, and in SharePoint 2016, with the Office Online Server, the query result page will show document previews for Office documents. These previews are available for results from SharePoint Online. However, for on-premises content, they are not displayed, unless the Office Web Application Server or Office Online Server are published externally to the Internet. The proper steps to publish these servers has been published on Microsoft’s TechNet site, in the article “Plan to Publish Applications through Web Application Proxy.”

On-Premises Query Options for Cloud Hybrid Search

The Enterprise Search Service Application, with the CloudIndex property set to true, will switch the plug-ins used by the crawl components to redirect the parsed content to Azure. No changes are made to the query process for these Service Applications; they are still directed to the local index, which will have no content. The parsed content that is directed to Azure is stored in the same index locations that are used by SharePoint Online for its query process. No further changes are needed to allow it to return cloud and on-premises content.

In terms of queries issues on on-premises sites, there are a few configuration changes that can be made to ensure that your user’s queries are processed by SharePoint Online. The first option can be used independently or in conjunction with one or both the other options. This option is to set up the outbound federated hybrid query result source as described earlier in this chapter. After the result source is saved, I recommend creating the result source at the Search Service Application level and making it the default result source from its context menu on the result sources page, after it has been saved. Look for the local SharePoint results. Each result source and query rule should also be duplicated and targeted at SharePoint Online. You will also require multiple query components on different servers, if you want high availability for your search queries.

The second option you can configure is to set the Global Search Center URL in the Cloud Search Service Application to a search center site collection in SharePoint Online. You can use this in conjunction with the first or third options. This will redirect the Everything, People, and Conversations search box option to the SharePoint Online Search Center. This can be set up by browsing to the Search Service Administration page and clicking the URL to the right of Global Search Center URL. The contextual options of This Site and This List will still be directed to the site’s current default result sources (Figure 5-11).

Figure 5-11. Global Search Center URL

The third option is to redirect the Contextual Search result page on a site-by-site level to SharePoint Online. It is a good idea to use this in conjunction with the second option. It can be manually set or confirmed with the following steps.

1. From an on-premises site collection, navigate to Site Settings.

2. On the Site Settings page, click Search Settings, under Site Collection Administration (Figure 5-12).

   

   Figure 5-12. Site collection search settings

3. On the Site Collection Administration Search Settings page, under Which search results page should queries be sent to? (see Figure 5-13),

   1. Disable Use the same Results Page Setting as my Parent.

   2. Send queries to a custom results page.

   3. In the results page URL, enter the following text: {SearchCenterURL}/Results.aspx.

   4. * {SearchCenterURL} is a token that can use the value of Search Center URL from the top section of the page. If that value is null, it will inherent the Global Search Center URL for the Search Service Application.

   5. Click OK to save the settings.

   

   Figure 5-13. Site collection search settings page

4. It can take up to 30 minutes for this setting change to take effect.

The following PowerShell code can also be used to update a site collection. It can be nested in a foreach loop for all site collection.

$Site = Get-SPSite <URL>
$Web = $Site.RootWeb
$Web.SetProperty("SRCH_SB_SET_SITE", "{`"Inherit`":false,`"ResultsPageAddress`":`"{SearchCenterURL}/results.aspx`",`"ShowNavigation`":false}")
$Web.Update()

Search Schema Changes

SharePoint 2013 introduced local search schemas at the site and site collection levels. These are present in SharePoint 2016 as well. However, when using a Cloud Search Service Application, it’s access is limited to the crawled properties. It cannot update any of the mappings between crawled and managed properties. Any modifications you make must be updated from the SharePoint Online Tenant Manage Search Schema page in the Search section.

The Cloud Search Service Application has also brought a new crawled and managed property, both called IsExternalContent. This is a Boolean property and is set to true for all content crawled from the Cloud Search Service Application, as it is external to the index location of SharePoint Online. This property can be used in a query or result source, to filter between SharePoint Online content and on-premises content.

People Search

SharePoint Online is configured to automatically import your users’ information from Azure Active Directory into the SharePoint Online User Profile Service. The User Profile server is also crawled by SharePoint Online and can present that information in the People Search results. If your organization has used line-of-business applications to augment your users’ profiles, you will most likely want to index the on-premises User Profile Service. This can be done by adding the URL of a web application connected to the User Profile Service Application Proxy replacing http with sps3 or https with sps3s.

Indexing this content will have duplicate profiles displayed in results. A new Results Source will be created that uses the IsExternalContent property set to 1 or true, to filter results to on-premises users only, with a query transformation of

{?{searchTerms} ContentClass=urn:content-class:SPSPeople IsExternalcontent:1}

SharePoint Online profiles can be displayed with a similar query transformation that has a NOT before the IsExternalContent, as shown following:

{?{searchTerms} ContentClass=urn:content-class:SPSPeople NOT IsExternalcontent:1}

Removing On-Premises Content from SharePoint Online

Special consideration needs to be taken when removing content indexed through the Cloud Search Service Application. You should never use the Reset Index option in the service application. This will orphan the on-premises content in the cloud index. When using incremental crawls, the Cloud Search Service Application can remove content from the index, by deleting it on-premises and waiting for the next incremental crawl. You can set the NoCrawl flag at the list or site level and wait for the next incremental crawl. You can also create a crawl rule to exclude the content and then start a full crawl of just that URL from the crawl log. The final option to remove all content is to delete the start address from the content source. When a start address is removed from a content source, SharePoint automatically starts a delete crawl and removes all content from the index that was crawled because of that start address. If you plan on deleting the content source, you should allow the delete crawl to finish before removing the content source.