Point-in-Time Recovery

This is the concept of recovering content across time. While you might expect to be able to restore a piece of content from any time in the past, the reality is very different. Here are some points to consider in this regard:

• How long in the past do you really need to recover a piece of content, and from where?

• What type of content requires PTR?

• Differentiate data loss recovery type—deletion operation vs. alteration

• What is the PTR coverage you currently offer on your existing infrastructure?

• What are the current tool(s) and methods you use to achieve the different PTR objectives?

• What does O365 offer in terms of PTR?

As I mentioned in the introduction, in SharePoint, you must consider these specifications for every layer of content. On many occasions, I have seen backups duplicated for the sole purpose of recovering many layers. For example, to recover a document vs. a site collection vs. a content database, I would see three different backup processes implemented to achieve these goals. While this is very efficient to facilitate the recovery process, it has a very large backup footprint for both storage requirements and backup window time.

This is a very important aspect that you must discuss with your enterprise backup vendor. While they will tell you that you are covered for every layer, ask how the backup process is done. Basically, if you back up a database, can you easily restore a site collection or a document?

It is important to understand that when you back up a SharePoint content database, no matter which method you use, you are actually backing up every piece of content contained in it: site collections, sites, lists/libraries, and items/documents.

But a backup is only as good as the restoration process that comes with it. While all content is part of a content database, your expectations may not be met, as your recovery may require more granularity than can be achieved through content database recovery.

As I stated previously, out of the box, the smallest full-fidelity restoration in SharePoint is a site collection. And it is also bound to a supported limit of 100GB that may impact your backup and restore scenarios. Let me detail the capabilities, based on the level and service and the proper approach for large-site collections (Table 6-1).

14-Day SharePoint Online

This is an important aspect differentiating usual on-premises expectations and SharePoint Online. If you have to recover a faulty artifact in SharePoint, it can be restored for any point time in the preceding 14 days, as long as it’s a database or a site collection. Outside of that scope, no official support is offered.

Let’s look at some hypothetical scenarios in which the expectations of a user may not be met after transitioning to SharePoint Online.

1. I am looking for a document that I created three months ago, and it is not present in either

   1. My recycle bin

   2. My previous versions

2. I need to restore my site collection to the state it was in a month ago.

3. I want to restore a specific library on SharePoint Online.

4. I have a compliance that requires that I keep deleted and altered content for a specific period.

While this is not the subject of this chapter, it is important to differentiate between a restore requirement vs. compliance. There are many features in SharePoint Online and in the security and compliance center of Office 365 that are designed to address specific concerns about retention outside of a backup. Although activation and configuration of these features may result in content being retained indefinitely, this is not a one-size-fits-all solution.

If your compliance requires you to keep everything, altered or deleted, for, let’s say, seven years, this is achievable with a retention rule in Office 365. Note that this will impact the size of your site and your tenant quota.

Alternatives on the Cloud

The 14-day PTR in the cloud is very similar to a well-balanced PTR on-premises. The difference is only in long-term recovery. On-premises, you will not have a full PTR. Instead, you will probably have a monthly PTR, and it is very possible that a piece of content will not exist on an archived tape, for example. If a document or a version was created/deleted within a month, chances are that it will not be captured. Because there is no equivalent of something like a tape retention, what are the other options available, if required?

The most popular are cloud-based enterprise tools. As you would probably do on-premises to an archiving area, the vendor will store your PTR content on the cloud. There are a few advantages to this approach.

• Much less storage requirement on-premises

• No archiving management

• No traffic from O365 to your network and much faster transfer time, especially if you are storing your PTR content on Azure, which can be in the same data center as O365

You can still back up the content to your data center, but that would defeat the benefits of using SharePoint Online, and you must manage bandwidth and storage requirements.

Limitations

You have to understand that these cloud backups are only for granular content. It will not do a database or a site collection backup. It is a limitation but, at the same time, a benefit. Let me explain.

At Microsoft, we will cover a recovery of a site collection or a database within 14 days. This is sufficient in any corruption scenarios in which your site is defective. The missing piece is granular PTR, and this is where we rely on great partners to provide a solution, if you require it.

Capabilities may differ from vendor to vendor, but you can at least expect the capability of recovering a document at a point in time based on your retention rules. But another benefit is that vendors can also provide more services, such as site reconstruction. Let me explain where site reconstruction/reorganization fits into a recovery requirement.

Site Reconstruction/Reorganization

Many times, when organizations start to use SharePoint, they do so without a proper governance and taxonomy (site and content structure). On top of this, they mix or aggregate too much content within a site collection, which becomes unmanageable over time. While this chapter does not cover site creation governance, you may have to reorganize content at some point, and this will require you to have a recovery strategy that is in line with our discussion.

Another scenario is when a site becomes defective owing to heavy customization or other issues, and you have to extract its content to a new site collection, so that you do not lose content but regain stability. On-premises, you can use the export feature of SharePoint (export-spweb), but it has limitations. It is bound to the version you are running and the site definition it was created with. But you can use an export to, well, export content to a new site collection. If you have a granular backup solution on O365, you could achieve the same goal of exporting content to a new site.

Just be aware that any exportation is not the same as a fidelity backup, and you may have to do some manual work to make your new site behave exactly as its predecessor. Typical limitations in exporting content are

• Last edited author and date replaced

• Workflow state lost and to be rerun

• List lookup relation lost

• Recycle bin content

• Alerts

Content Categorization

Content categorization is very important to help you define your recovery strategy, especially that we now know that, with SharePoint, we must deal with the content and/or the container. A simple strategy, such as what you may be used to with backing up a file share, or a database, may not be sufficient. Let me show you the type of content and high-level categorization that exist in SharePoint.

Whenever I work with information architecture (IA), no matter the size or how mature it is, I provide the basic pyramid that is the foundation of understanding content categorization in SharePoint. No matter how your IA is defined, you will probably find your content in one of these categories and can apply a basic SLA to it (see Figure 6-4).

Figure 6-4. My own extended version of the SharePoint content pyramid

The top layer is the realm of any business-critical application. It may be identified as a portal, but in SharePoint, it defines something that is highly managed. It can be very small or mildly large, but it has a high impact and requires a specific recovery SLA. In any case, it should be isolated in a site collection and, many times, an isolated database.

The second layer is the most complex area to handle. Document management and record management are usually subject to well-defined retention policies, which vary from geography to geography, from industry to industry, and even among customers. It should have, by far, the largest containers and site collections. This is usually where you must decide if you need granular recovery capabilities.

The third layer, while it is the most puzzling category lately, should be the easiest to handle and has the most capabilities. This is the “team,” “social,” and “modern collaboration” category. The technologies used in this category of content extends outside just SharePoint! We now have Outlook groups that use both the best from SharePoint and Exchange, Yammer, and, more recently, Microsoft Teams. You may wonder which one to use, and this is a very hot topic right now, but not within the scope of this chapter. I will limit this layer to the typical SharePoint team site that is not meant for departmental usage. That means that security is not centralized in AD or Azure and that sites are typically small and have a limited lifespan.

Talking about lifespan, a departmental site is not meant to have an expiration. We will manage document retention instead. Collaborative sites, on the other hand, may have expiration and retention rules and can be ad hoc in nature.

Last, but not least, is personal storage—what used to be your personal drive on a network server or local My Documents. Many individuals have moved or are planning to move to either on-premises or O365 One Drive for Business. This can be small or very large and comes with many functionalities, such as versioning, sharing, compliance, and access from anywhere and on different devices. With a capability that can go up to 5TB per user, and even more on demand, compliance may be required, but what about recovery?

If we recapitulate, the following is what categorization and recovery can look like:

• A critical business application site collection has a very rapid SLA covering the site collection.

• A departmental or business unit site collection(s) has (have) a granular document SLA and a database SLA.

• An ad hoc project/team has a site collection SLA without granular level.

• A personal drive is an isolated storage without any SLA besides the Sync Client and/or manager’s retention.

You can now translate this to a basic service definition , as in Table 6-2.

Now that we have a minimal content categorization, we will be able to define a backup retention strategy. I will provide some examples, but this will depend on your policies. Be sure to validate the typical statement whereby an organization thinks it has to keep everything forever. This is where technicalities are important to understand, as they greatly affect the restoration process.

The GUID Story

Other important factors to understand when dealing with site restoration are globally unique identifiers (GUIDs) . GUIDs are unique IDs that keep the consistency in the SharePoint structure, but they can lead to disconnected services and various additional issues. They are the relational aspect in SQL and, by doing some faulty operations, can lead to conflicting artifacts that you often see referenced as orphans. In dealing with recovery of databases and site collections, there are two important GUIDs: the site collection and the content database.

One operation that will lead to issues is the restoration and mounting to SharePoint of a copy of a database. In mount-spcontentdatabase, you have an optional parameter -AssignNewDatabaseId that allows you to have a new database GUID. Unfortunately, this command will not change the site collection’s GUIDs, in it or its URL. So, if you do this, you will end up with orphaned site collections. A copy of a content database that contains site collections can only be restored in a different farm.

The other operation that may lead to some issues is restore-spsite. By default, if you restore and overwrite a site collection using the -force parameter, the site will get a new GUID. Great! That should prevent orphans—but it will cause a disconnection with your Managed Metadata Service (MMS), if you have local terms in that site collection. Local terms are stored in the MMS database and have the site collection GUID to reference them. Change your GUID, and you will have the metadata available in your site collection, as the content is also stored there, but you cannot edit it, as the MMS is now referring to the wrong GUID. If you plan to overwrite a site collection, you should delete it first and use the latest parameter in SharePoint 2013: -PreserveSiteId.

And last, one common restore request is to have a copy of a site collection, sometimes for parallel improvement or granular recovery of some content. Ideally, you want to restore a copy in a different farm. This provides better protection from any mistake by users or administrators. But if you must, you can restore a copy of a site collection, as long as it is in a different content database, has a different URL, and, of course, a different GUID.

Figure 6-5 shows a typical site collection and overwrite restoration process when the database contains more than one site.

Figure 6-5. Site collection operation

Figure 6-6 shows a typical restore operation when dealing with a large and isolated site collection.

Figure 6-6. Large site collection

While Figure 6-6 shows a site larger than 100GB, I suggest that you proactively identify which sites are or will grow larger, isolate them right away, and use the database recovery process.

Another important point is when the SQL team overwrites the database, if any connections are still open, do not forcibly close them. The site may not be functional after restoration. It takes about five minutes for SharePoint to complete an unmount operation. This is the clean way of approaching a large site restoration and will ensure a successful process. And if the site does not respond after restoration (404 error or blank page), you may have to clear the Timer Job Cache and reset IIS.

As for database size, you probably saw the recommended 200GB. This is not a hard limit but a recommendation, owing to these backup/recovery processes. As you just read, in the case of an isolated site collection, all you must do is restore and overwrite a database. But imagine a scenario in which you must restore a copy for a more granular request. In this case, it would not be very practical to have a 2TB content database. The different database size recommendations, based on usage such as archiving and record management while dealing with capacity for IOPS (input/output operations per seconds) are well explained on TechNet. But, as I explained, they are bound by how content is managed in them for operational processes and SLAs.

For example, if you store all your archives in a very large database in one site collection, let’s say, 4TB, you understand that you will not require the same performance access as active sites or will require much higher IOPS. You will also have a different SLA and backup schedule, as the content is, in theory, inactive and mainly for read access. And if granular recovery is necessary, you will require an enterprise backup tool that will capture document changes, as you will not really restore a copy of that database to capture one document.

Recycle Bin and Versioning

These two very important features of SharePoint are often overlooked and not managed. They are also not well understood.

Versioning

One of the first reasons people move from File Share to SharePoint is, of course, versioning. While the recycle bin protects you from accidental deletion, versioning is your first line of defense against undesired change or history tracking. Good recovery planning will include a versioning strategy. Why a strategy? Because versioning has a certain impact on storage requirement, and not every type of content requires the same versioning setting. SharePoint provides not only versioning but also content-approval processes.

When you enable versioning, the default settings will probably not apply to all your content. Depending on the type of site you are working in, versioning may or may not be enabled by default in a specific library (Figure 6-7). As well, when enabled, limitations are optional and inactive by default, which will consume more storage than if you impose limits. Fortunately, since SharePoint 2013, shredded storage was introduced, and only the delta (differences) are saved in the SharePoint database when saving multiple versions of a document, reducing storage requirement quite dramatically, compared to SharePoint 2010 and earlier versions.

Figure 6-7. Versioning settings in a SharePoint library

As with the recycle bin second-stage myth I described previously, versioning also has a setting that is often misinterpreted. If you read the previous screen capture rapidly, you will assume that you can define the number of major and minor versions, basically, how many minor versions you can save between two major ones and how many majors. But go ahead and read again. The reality, and it is well explained in the setting page, is very different.

You should read how many major versions can be stored and how many of them will have minor versions. If I choose to keep five drafts, then only the last five major versions will have minor versions. The previous majors will not. But in these five majors, you can have up to 511 minor versions between each major.

This way of thinking assumes that once a major version is released, keeping track of changes with the last is not required permanently within the drafts. Basically, you will only have to keep track between two major versions. It also assumes that when working with versions, you will at some point publish that document and that you don’t really have to put any limitation between majors.

The reality is that, sometimes, many people work with a minor, without ever publishing a major, version. And some customers will even have high expectations of SharePoint deals with minors, such as being available in a search result. This is not a good working plan, and it should be better defined, depending, again, on the type of content authored.

Versioning requirements will probably be very different in a publishing web site than a collaborative team site. They will also be very different in official document managed sites. For example, in a collaborative site, there may not be much requirement to enable minor versions. In a publishing web site or with a legal department’s document management, where content approval is required, you will probably want to have draft versions enabled. But once a document is approved and final, the need for keeping the draft versions may be less important, at least for a certain time. If change tracking is required, it may be sufficient to keep drafts between two previous major versions.

Services That Fit a Database Strategy

These services, like web applications, can rely on your overall build and SQL backup processes. Basically, if a service must be restored, rebuild the service, using a restored database. Following is a list of services that can use this model:

• Managed Metadata Service

• BCS Service**

• User Profile Service *

• Secure Store *

• Subscription setting

• Apps *

• Machine Translation Service Application

• Performance Point**

  * Requires additional steps to complete the process and may depend on other services

  ** May use Secure Store

As explained in TechNet, you can simply overwrite a service database, but you need, at a minimum, to stop the timer services. You should also never forcibly close the connections when restoring in SQL. In some cases, this is insufficient, and deleting the service application and database(s) is the safest method to recover. In any case, if after a restore the service is not running as expected, you will probably have to reset the IIS service and, in some cases, clear the timer service cache.

Also, restoring only the content of a service application is probably very rare, unless you are copying the service to another farm. Usual restoration requests for a service would probably be due to instability or misconfiguration. In this case, it is preferable to rebuild the service application, using the restored database and reprovisioning the end points (Services on server Stop/Start). Note that deleting the database by removing the service application may not be possible, if it is part of a SQL availability group. Deleting a database may also create overhead on the SQL team.

Additional Steps for Some Services

User Profile Service

In the case of the User Profile Service, you must also restore the Social database. If you are using the FIM engine to synchronize, you are also using the profile Sync database. This database contains encryption, and the key must be restored, if you plan to restore it.

The easiest method to restore the User Profile Service with FIM is to not restore that database.

1. Delete the service and the databases.

2. Restore the Profile and Social databases.

3. Re-create the service. A new Sync database will be created.

Secure Store

On TechNet, the documentation only offers the SharePoint backup as a recovery method, but you can still use the database method. There is one extra element that is very important to consider with this service. This is the Master key that you created originally by using a pass phrase at the service creation. You must store this pass phrase, as it will be required if you re-create your service application using the restored database (Figure 6-8). Note that if you did not delete the service application and only overwrote the database, you do not have to refresh the pass phrase (Figure 6-9).

Figure 6-8. Master key error after restore

Figure 6-9. Master key refresh

App Management Service

The App Management Service has multiple dependencies. If you are restoring within the same environment, consider which component you want to restore.

• App Management Service

• Secure Store service application

• Subscription service application

• Apps Catalog Site Collection

• Any site collection with Apps/Add-in deployed

If you plan to restore the App Service in the same environment, you need only restore the service database. If, in the other end, you want to restore apps in the catalog, the site collection will also have to be restored.

It is also important to note that if you are restoring in another farm, every dependent artifact must also be restored. Otherwise, you may have to reinstall or configure some components, such as trusts, manifest, etc….

For this scenario, because we are only restoring the service application in our current environment, the same process as with the other service applies.

1. Delete the service application.

2. Restore the database.

3. Re-create the service application using the same database name.

Access Service (2013)

This service application relies on the App Service Application and secure store. While Access Service can be backed up using the SP-Farm backup process, restoring it may still lead to issues and errors. Also, re-creating the Access Service application causes the existing Access App to stop working. The newly created Access Service application will be OK.

The problem when re-creating the Access Service application is that it creates a new GUID entry for the hosting SQL server. Any new access apps will use the new GUID, and the previous one will try to connect to the one that does not exist anymore. In the case of a DR farm or recovery of the service within the farm, you will have to reconfigure the service to use the previous GUID. If you are not able to point to the previous GUID, you will have to export the Access data using the Access client, by creating an App Package.

To identify your SQL server reference ID, before any recovery attempt, execute the following script. It may be a good idea to keep this GUID documented in your build documentation.

$ASapp = Get-SPAccessServicesApplication
$app = $Null
if ($ASapp.length -ne $Null) { $app = $ASapp[0] } else { $app = $ASapp }
$context = [Microsoft.SharePoint.SPServiceContext]::GetContext($app.ServiceApplicationProxyGroup, [Microsoft.SharePoint.SPSiteSubscriptionIdentifier]::Default)
Get-SPAccessServicesDatabaseServer -ServiceContext $context

Otherwise, you will probably find it in the ULS log, where after a restore, you may get the following error in the ULS log: “AccessServicesDatabaseServerGroupCollection.GetDatabaseServer: Could not find a server matching reference id…”

To add the previous server entry, you can execute the following script, by inserting the previous GUID in the $ServerRefID and your SQL Server Host Name variables.

$ServerRefID = "Your previous GUID here"
$SqlHost = "Your SQL Host Name"
########
$serverGroupName = 'DEFAULT'
$ASapp = Get-SPAccessServicesApplication
$app = $Null
if ($ASapp.length -ne $Null) { $app = $ASapp[0] } else { $app = $ASapp }
$context = [Microsoft.SharePoint.SPServiceContext]::GetContext($app.ServiceApplicationProxyGroup, [Microsoft.SharePoint.SPSiteSubscriptionIdentifier]::Default)
$newdbserver = New-SPAccessServicesDatabaseServer -ServiceContext $context -DatabaseServerName $SqlHost -DatabaseServerGroup $serverGroupName -ServerReferenceId $ServerRefID -AvailableForCreate $true

Services That Do Not Have a Database

These services have only to be re-created if there are issues and do not possess any recovery steps .

• Excel Services

• Visio Services

• Word Automation

• Work Management

Search

Search is very different and has its own complexity. It contains not only databases but local indexes that need to be in sync. Therefore, there are no complete search database recovery processes. If your SLA requires that you back up search completely, in a scenario in which recovery would be faster than rebuilding the indexes and crawling the content, you will be required to use the SharePoint backup process (backup-spfarm).

Backup-spfarm -Item "FarmShared ServicesShared Services Applications<SearchServiceApplicationName>"

If you desire only to keep the search configuration, your build process and documentation should have the automation and documentation for any configuration, such as custom result sources, content sources, etc. If automation/documentation is not present or not up to date, it is possible to reuse the Admin database to re-create your new Search Service Application, using the same process to create the service via the existing/restored Admin database. You must also ensure that you have a copy of the thesaurus files that you used to import into search, as they are not part of the SharePoint backup.

Configuration Database

By now, you should know that restoration of the configuration database is not supported. The only method by which you can reuse the configuration database is by some replication scenario in which every bit of SharePoint is guaranteed to be backed up at the same time, including VM and databases.

The other typical scenario in which a configuration database will be restored is when a complete backup is taken before a major update, while the farm is offline, and where, in the case of a major issue, it is part of a rollback plan whereby every component, server, and database is restored at the same point in time. This is an all-or-nothing scenario. You cannot restore a configuration database alone from a point in time different from that of any other of its components.

I always tell my customers that the best farm recovery process is a great build process. It is much easier to rebuild than fix. If it is not the case in your organization, you should prioritize testing your build process.

SharePoint Farm Backup

The SharePoint farm backup can be practical and useful in certain situations, but it may not be sufficient in many scenarios. It is also redundant of your actual SQL backup and build strategy. If your farm becomes unstable, unusable, or even if one of the servers is problematic, chances are that using the restore-spfarm operation will not fix your issue. I like to think of this backup as a setting backup that does not replace a good enterprise strategy. Remember that the configuration database is not really restored, and if it was damaged, chances are that the farm backup would not fix it.

The farm backup can be used to reapply settings to a new farm, if you don't have any automation in place. Also, the backup-spfarm configuration can only be useful to recuperate settings, without backing up other components, such as databases that are already taken by SQL backups.