Disks formatted using FAT are organized as shown in Figure 15-1. They have a boot sector that stores information about the disk type, starting and ending sectors, the active partition, and a bootstrap program that executes at startup and boots the operating system. This is followed by a reserve area that can be one or more sectors in length.

Figure 15-1. Here is an overview of the FAT16 volume structure.

The reserve area is followed by the primary file allocation table, which provides a reference table for the clusters on the volume. Each reference in the table relates to a specific cluster and defines the cluster’s status as follows:

If a cluster is in use, the cluster entry identifies the number of the next cluster in the file or indicates that it is the last cluster of a file—in which case, the end of the file has been reached.

FAT volumes also have the following features:

When an application attempts to read a file, the operating system looks up the starting cluster of the file in the file allocation table and then uses the file allocation table to find and read all the clusters in the file.

Although FAT supports basic file and folder operations, its features are rather limited. By using FAT, you have the following capabilities:

However, FAT has the following disadvantages:

In addition, although FAT16 and FAT32 support small cluster sizes, exFAT does not. Table 15-2 provides a summary of FAT16, FAT32, and exFAT.

By default, Windows Server sets the size of clusters and the number of sectors per cluster based on the size of the volume. Disk geometry also is a factor in determining cluster size because the number of clusters on the volume must fit into the number of bits the file system uses. The actual amount of data you can store on a single FAT volume is a factor of the maximum cluster size and the maximum number of clusters you can use per volume. This can be written out as a formula:

ClusterSize × MaximumNumberOfClusters = MaximumVolumeSize

FAT16 supports a maximum of 65,526 clusters and a maximum cluster size of 64 KBs. This is where the limitation of 4 GBs for volume size comes from. With disks less than 32 MBs but more than 16 MBs in size, the cluster size is 512 bytes and there is one sector per cluster with 512b disks. This changes as the volume size increases, up to the largest cluster size of 64 KBs with 128 sectors per cluster on 2 GB to 4 GB volumes.

FAT32 volumes using 512-byte sectors on 512b disks can be up to 2 TBs in size and can use clusters of up to 64 KBs. To control the maximum number of clusters allowed, the Windows operating system reserves the upper 4 bits, however, limiting FAT32 to a maximum 28 bits’ worth of clusters. With a maximum recommended cluster size of 32 KBs (instead of the maximum allowable 64 KBs), this means a FAT32 volume on the Windows operating system can be up to 32 GBs in size. Because the smallest cluster size allowed for FAT32 volumes is 512 bytes, the smallest FAT32 volume you can create is 33 MBs.

Volume Name : LogData
Volume Serial Number : 0x70692a2e
Max Component Length : 255
File System Name : FAT32
Preserves Case of filenames
Supports Unicode in filenames

NTFS volumes have a very different structure and feature set from FAT volumes. The first area of the volume is the boot sector, which is located at sector 0 on the volume. The boot sector stores information about the disk layout, and a bootstrap program executes at startup and boots the operating system. A backup boot sector is placed at the end of the volume for redundancy and fault tolerance.

Instead of a file allocation table, NTFS uses a relational database to store information about files. This database is called the master file table (MFT). It stores a file record of each file and folder on the volume, pertinent volume information, and details on the MFT itself. The first 15 records in the MFT store NTFS metadata as summarized in Table 15-3.

The MFT mirror stores a partial duplicate of the MFT that can be used to recover the MFT. If any of the records in the primary mirror become corrupted or are otherwise unreadable and there’s a duplicate record in the MFT mirror, NTFS uses the data in the MFT mirror and, if possible, uses this data to recover the records in the primary MFT. It is also important to note that the NTFS version that shipped with Windows Server 2003 and later (NTFS 5.1) has a slightly different metadata mapping from the version that originally shipped with Windows 2000 (NTFS 5.0). In NTFS 5.1, the $LogFile and $Bitmap metadata files are located in a different position on disk than they were originally. This gives a performance advantage of 5 to 8 percent to disks that are formatted under NTFS 5.1 and comes close to approximating the performance of FAT.

The rest of the records in the MFT store file and folder information. Each of these regular entries includes the file or folder name, security descriptor, and other attributes, including file data or pointers to file data. The MFT record size is set when a volume is formatted and can be 1,024 bytes, 2,048 bytes, or 4,096 bytes, depending on the volume size. If a file is very small, all its contents might be able to fit in the data field of its record in the MFT. When all of a file’s attributes, including its data, can be stored in the MFT record, the attributes are called resident attributes. Figure 15-2 shows an example of a small file with resident attributes.

If a file is larger than a single record, it has what are called nonresident attributes. Here, the file has a base record in the MFT that details where to find the file data. NTFS creates additional areas called runs on the disk to store the additional file data. The size of data runs depends on the cluster size of the volume. If the cluster size is 2 KBs or less, data runs are 2 KBs. If the cluster size is larger than 2 KBs, data runs are 4 KBs.

Figure 15-2. This figure is a graphical depiction of the MFT and its records.

As Figure 15-3 shows, clusters belonging to the file are referenced in the MFT, using virtual cluster numbers (VCNs). VCNs are numbered sequentially, starting with VCN 0. The Data field in the file’s MFT record maps the VCNs to a starting logical cluster number (LCN) on the disk and details the number of clusters to read for that VCN. When these mappings use up all the available space in a record, additional MFT records are created to store the additional mappings.

Figure 15-3. This figure shows a graphical depiction of a user file record with data runs.

In addition to the MFT, NTFS reserves a contiguous range of space past the end of the MFT called the MFT zone. By default, the MFT zone is approximately 12.5 percent of the total volume space. The MFT zone enables the MFT to grow without becoming fragmented. Typically, the MFT zone shrinks as the MFT grows.

The MFT zone is not used to store user data unless the remainder of the volume becomes full. Fragmentation can and still does occur, however. On volumes with lots of small files, the MFT can use up the MFT zone, and as additional files are added, the MFT has to grow into unreserved areas of the volume. On volumes with just a few large files, the unreserved space on a volume can be used up before the MFT, and in this case, the files start using the MFT zone space.

Several versions of NTFS are available. NTFS 5.1 is the version of NTFS that was first included in Windows XP and Windows Server 2003. NTFS 5.1 with Local File System (LFS) 2.0 was first included with Windows 8 and Windows Server 2012.

You have the following capabilities when you use NTFS 5.0:

NTFS 5.1 provides some additional enhancements, primarily the ability to use shadow copies. In similar fashion, NTFS 5.1 with Local File System (LFS) 2.0 also provides some additional enhancements, primarily related to self-healing technology used with Check Disk (Chkdsk.exe) and the update sequence number (USN) change journal. Specifically, NTFS and ReFS use version 2.0 change-journal records by default, which contain 64-bit identifiers. ReFS also implements version 3.0 records, which contain 128-bit identifiers.

Windows Server automatically sets the size of clusters and the number of sectors per cluster based on the size of the volume. Cluster sizes range from 512 bytes to 64 KBs. As with FAT, NTFS has the following characteristics:

Thus, although volumes have a specific maximum size, the cluster size used on a volume can be a limiting factor. For example, a dynamic volume with a 4 KB cluster size can have dynamic volumes up to 16 TBs, which is different from the maximum allowed dynamic volume size on NTFS.

If you want to examine the structure of a volume formatted using NTFS, you can use the FSUtil FSinfo command to do this. Type fsutil fsinfo ntfsinfo DriveDesignator at the command prompt, where DriveDesignator is the drive letter of the volume followed by a colon. For example, if you want to obtain information on the C drive, you type

fsutil fsinfo ntfsinfo c:

The output would be similar to the following:

NTFS Volume Serial Number :       0xbcf4c873f4c82125
NTFS Version   :                  3.1
LFS Version    :                  2.0
Number Sectors :                  0x000000001d3c57ff
Total Clusters :                  0x0000000003a78aff
Free Clusters  :                  0x00000000035e477d
Total Reserved :                  0x000000000001e2b0
Bytes Per Sector  :               512
Bytes Per Physical Sector :       512
Bytes Per Cluster :               4096
Bytes Per FileRecord Segment    : 1024
Clusters Per FileRecord Segment : 0
Mft Valid Data Length :           0x0000000004e00000
Mft Start Lcn  :                  0x00000000000c0000
Mft2 Start Lcn :                  0x0000000000000002
Mft Zone Start :                  0x00000000000c4e00
Mft Zone End   :                  0x00000000000cc820
Resource Manager Identifier :     CBDD98AD-E33F-11E1-95F2-C407271F80D4

As Table 15-4 shows, FSUtil FSinfo provides detailed information on the NTFS volume structure, including space usage and configuration.

Using FSUtil, you can also obtain detailed statistics on NTFS metadata and user file usage since a system was started. To view this information, type fsutil fsinfo statistics DriveDesignator at the command prompt, where DriveDesignator is the drive letter of the volume followed by a colon. For example, if you want to obtain information on the C drive, you type:

fsutil fsinfo statistics c:

The output is shown in two sections. The first section of the statistics details user file and disk activity and the overall usage of NTFS metadata. As shown in this example, the output shows the number of reads and writes and the number of bytes read or written:

File System Type :     NTFS
UserFileReads :        31441
UserFileReadBytes :    857374720
UserDiskReads :        31584
UserFileWrites :       6302
UserFileWriteBytes :   197198336
UserDiskWrites :       6505
MetaDataReads :        3168
MetaDataReadBytes :    21770240
MetaDataDiskReads :    4165
MetaDataWrites :       3883
MetaDataWriteBytes :   16805888
MetaDataDiskWrites :   4644

The second section of the statistics details usage of individual NTFS metadata files. As shown in this example, the output details the number of reads and writes and the number of bytes read or written for each NTFS metadata file:

MftReads :             2962
MftReadBytes :         12132352
MftWrites :            2460
MftWriteBytes :        10465280
Mft2Writes :           0
Mft2WriteBytes :       0
RootIndexReads :       0
RootIndexReadBytes :   0
RootIndexWrites :      0
RootIndexWriteBytes :  0
BitmapReads :          8
BitmapReadBytes :      8388608
BitmapWrites :         847
BitmapWriteBytes :     3796992
MftBitmapReads :       1
MftBitmapReadBytes :   65536
MftBitmapWrites :      107
MftBitmapWriteBytes :  442368
UserIndexReads :       1086
UserIndexReadBytes :   4448256
UserIndexWrites :      711
UserIndexWriteBytes :  3153920
LogFileReads :         8
LogFileReadBytes :     32768
LogFileWrites :        5895
LogFileWriteBytes :    36777984
LogFileFull :          0

Every file created on a volume has a hard link. The hard link is the directory entry for the file, and it is what enables the operating system to find files within folders. On NTFS volumes, files can have multiple hard links. This allows a single file to appear in the same directory with multiple names or to appear in multiple directories with the same name or different names. As with file copies, applications can open a file by using any of the hard links you’ve created and can modify the file. If you use another hard link to open the file in another application, the application can detect the changes.

Wondering why you’d want to use hard links? Hard links are useful when you want the same file to appear in several locations. For example, you might want a document to appear in a folder of a network share that is available to all users but have an application that requires the document to be in another directory so that it can be read and processed on a daily basis. Rather than moving the file to the application directory and giving every user in the company access to this protected directory, you decide to create a hard link to the document so that it can be accessed separately by both users and the application.

Regardless of how many hard links a file has, the related directory entries all point to the single file that exists in one location on the volume—and this is how hard links differ from copies. With a copy of a file, the file data exists in multiple locations. With a hard link, the file appears in multiple locations but exists in only one location. Thus, if you modify a file by using one of its hard links and save the file, and then someone opens the file using a different hard link, the changes are shown.

Because there is only one physical copy of a file with multiple hard links, the hard links do not have separate security descriptors. Only the source file has security descriptors. Thus, if you were to change the access permissions of a file by using any of its hard links, you would actually change the security of the source file and all hard links that point to this file would have these security settings.

You can create hard links by using the FSUtil Hardlink command. Use the following syntax:

fsutil hardlink create NewFilePath CurrentFilePath

Here, NewFilePath is the file path for the hard link you want to create, and CurrentFilePath is the name of the existing file to which you are linking. For example, if the file ChangeLog.doc is found in the file path C:CorpDocs and you want to create a new hard link to this file with the file path C:UserDataLogsCurrentLog.doc, you would type

fsutil hardlink create C:UserDataLogsCurrentLog.doc C:CorpDocsChangeLog.doc

Hard links can be created only on NTFS volumes, and you cannot create a hard link on one volume that refers to another volume. Following this logic, you couldn’t create a hard link to the D drive for a file created on the C drive.

Every file created on a volume has a data stream associated with it. A data stream is a sequence of bytes that contains the contents of the file. The main data stream for a file is unnamed and is visible to all file systems. On NTFS volumes, files can also have named data streams associated with them. Named data streams contain additional information about a file, such as custom properties or summary details. This enables you to associate additional information with a file but still manage the file as a single unit.

After you create a named data stream and associate it with a file, any applications that know how to work with named data streams can access the streams by their names and read the additional details. Many applications support named data streams, including Microsoft Office, Adobe Acrobat, and other productivity applications. This is how you can set summary properties for a Microsoft Word document—such as Title, Subject, and Author—and save that information with the file.

In fact, if you press and hold or right-click any file on an NTFS volume, select Properties, and then tap or click the Details tab, you can see information that is associated with the file using a data stream, as shown in Figure 15-4.

Figure 15-4. Information entered on the Details tab is saved to a named data stream.

Generally speaking, the named data streams associated with a file are used to set the names of its property tabs and to populate the fields of those tabs. This is how other tabs can be associated with some document types and how the Windows operating system can store a thumbnail image within an NTFS file containing an image.

The most important thing to know about streams is that they aren’t supported on FAT. If you move or copy a file containing named streams to a FAT volume, you might see the warning prompt labeled “Confirm Stream Loss” telling you additional information is associated with the file and asking you to confirm that it’s okay that the file is saved without this information. If you tap or click Yes, only the contents of the file are copied or moved to the FAT volume—and not the contents of the associated data streams. If you tap or click No, the copy or save operation is canceled.

In a file’s Properties dialog box on the Details tab, you also have the option of removing properties and personal information associated with a file. You do this by tapping or clicking the Remove Properties And Personal Information link and then selecting a Remove Properties method. Windows accomplishes this task by removing the values from the related data streams associated with the file.

An NTFS volume can use an update sequence number (USN) change journal. A change journal provides a complete log of all changes made to the volume. It records additions, deletions, and modifications regardless of who made them or how the additions, deletions, and modifications occurred. As with system logs, the change log is persistent, so it isn’t reset if you shut down and restart the operating system. The operating system writes records to the NTFS change log when an NTFS checkpoint occurs. The checkpoint tells the operating system to write changes that would enable NTFS to recover from failure to a particular point in time.

The change journal is enabled when you install certain services, including distributed file system (DFS). Domain controllers and any other computer in the domain that uses these services rely heavily on the change journal. The change journal enables these services to be very efficient at determining when files, folders, and other NTFS objects have been modified. Rather than checking time stamps and registering for file notifications, these services perform direct lookups in the change journal to determine all the modifications made to a set of files. Not only is this faster, it also uses system resources more efficiently.

You can gather summary statistics about the change journal by typing fsutil usn queryjournal DriveDesignator at the command prompt, where DriveDesignator is the drive letter of the volume followed by a colon. For example, if you want to obtain change journal statistics on the C drive, you type:

fsutil usn queryjournal c:

The output is similar to the following:

Usn Journal ID   : 0x01cd77459da4462a
First Usn        : 0x0000000000000000
Next Usn         : 0x0000000002573bf8
Lowest Valid Usn : 0x0000000000000000
Max Usn          : 0x7fffffffffff0000
Maximum Size     : 0x0000000020000000
Allocation Delta : 0x0000000000400000
Minimum record version supported : 2
Maximum record version supported : 2

The details show the following information:

Individual records written to the change journal look like this:

File Ref#       :                                  0x18e90000000018e9
ParentFile Ref# :                                  0x17c00000000017c0
Usn             :                                  0x0000000000000000
SecurityId      :                                  0x00000119
Reason          :                                  0x00000000
Name (024)      :                                  ocmanage.dll

The most important information here is the name of the affected file and the security identifier of the object that made the change. You can get the most recent change journal entry for a file by typing fsutil usn readdata FilePath, where FilePath is the name of the file for which you want to retrieve change information. For example, if you want to obtain the most recent change journal information on a file with the path C:DomainComputers.txt, you type:

fsutil usn readdata c:domaincomputers.txt

The output is similar to the following:

Major Version    :                                 0x2
Minor Version    :                                 0x0
FileRef#         :                                 0x000800000001c306
Parent FileRef#  :                                 0x0005000000000005
Usn              :                                 0x00000000237cf7f0
Time Stamp       :                                 0x0000000000000000
Reason           :                                 0x0
Source Info      :                                 0x0
Security Id      :                                 0x45e
File Attributes  :                                 0x20
File Name Length :                                 0x26
File Name Offset :                                 0x3c
FileName         :                                 domaincomputers.txt

This data shows the file’s reference number in the root file index and that of its parent. It also shows the current USN associated with the file and the file attributes flag. The File Name Length element shows the total length in characters of the file’s long and short file names together. This particular file has a file name length of 38 (0×26). That’s because the file name has more than eight characters followed by a dot and a three-letter extension. This means the file is represented by NTFS, using long and short file names. The long file name is domaincomputers.txt. This is followed by an offset pointer that indicates where the short file name, domain~1.txt, can be looked up, which is where the total file name length of 38 characters comes from.

Another feature of NTFS is the ability to use object identifiers. Object identifiers are 16 bytes in length and are unique on a per-volume basis. Any file that has an object identifier also has the following:

These values are also 16 bytes in length. If a file is moved within a volume or moved to a new volume, it is assigned a new object identifier, but information about the original object identifier assigned when the object was created can be retained by using the birth object identifier.

Several system services use object identifiers to identify files uniquely and identify the volumes with which they are associated. The Distributed Link Tracking (DLT) Client service uses object identifiers to track linked files that are moved within an NTFS volume, to another NTFS volume on the same computer, or to an NTFS volume on another computer.

Any file the DLT Client service uses has an object identifier field set containing values for the object ID, birth volume ID, birth object ID, and domain ID. The actual field set looks like this:

Object ID :                            52eac013e3d34445334345453533ab3d
BirthVolume ID :                       a23bc3243a5a3452d32424332c32343d
BirthObject ID :                       52eac013e3d34445334345453533ab3d
Domain ID :                            00000000000000000000000000000000

Here, the file has a specific object ID, birth volume ID, and birth object ID. The domain ID isn’t assigned, however, because this is not currently used. You can tell that the the DLT Client service uses the file because the birth volume ID and birth object ID have been assigned and these identifiers are used only by this service. Because the birth volume ID and birth object ID remain the same even if a file is moved, the DLT Client service uses these identifiers to find files no matter where they have been moved.

If you are trying to determine whether the DLT Client service uses a file, you could use the FSUtil ObjectID command to see whether the file has an object identifier field set. Type fsutil objectid query FilePath at the command prompt, where FilePath is the path to the file or folder you want to examine. If the file has an object identifier field set, it is displayed. If a file doesn’t have an object identifier field set, an error message appears, stating, “The specified file has no object ID.”

On NTFS volumes, a file or folder can contain a reparse point. Reparse points are file system objects with special attribute tags that are used to extend the functionality in the I/O subsystem. When a program sets a reparse point, it stores an attribute tag and a data segment. The attribute tag identifies the purpose of the reparse point and details how the reparse point is to be used. The data segment provides any additional data needed during reparsing.

Reparse points are used for directory junction points and volume mount points. Directory junctions enable you to create a single local namespace by using local folders, local volumes, and network shares. Mount points enable you to mount a local volume to an empty NTFS folder. Both directory junction points and volume mount points use reparse points to mark NTFS folders with surrogate names.

When a file or folder containing a reparse point used for a directory junction point or a volume mount point is read, the reparse point causes the path to be reparsed and a surrogate name to be substituted for the original name. For example, if you were to create a mount point with the file path C:Data that is used to mount a hard disk drive, the reparse point is triggered whenever the file system opens C:Data and points the file system to the volume you mounted in that folder. The actual attribute tag and data for the reparse point would look similar to the following:

Reparse Tag Value :  0xa0000003
Tag value : Microsoft
Tag value : Name Surrogate
Tag value : Mount Point
Substitute Name offset :   0
Substitute Name length :   98
Print Name offset :  100
Print Name Length :  0
Substitute Name :  ??Volume{3796c3c1-5106-11d7-911c-806d6172696f}
Reparse Data Length : 0x0000006e
Reparse Data :
0000 : 00 00 62 00 64 00 00 00  5c 00 3f 00 3f 00 5c 00  ..b.d....?.?..
0010 : 56 00 6f 00 6c 00 75 00  6d 00 65 00 7b 00 33 00  V.o.l.u.m.e.{.3.
0020 : 37 00 39 00 36 00 63 00  33 00 63 00 31 00 2d 00  7.9.6.c.3.c.1.-.
0030 : 35 00 31 00 30 00 36 00  2d 00 31 00 31 00 64 00  5.1.0.6.-.1.1.d.
0040 : 37 00 2d 00 39 00 31 00  31 00 63 00 2d 00 38 00  7.-.9.1.1.c.-.8.
0050 : 30 00 36 00 64 00 36 00  31 00 37 00 32 00 36 00  0.6.d.6.1.7.2.6.
0060 : 39 00 36 00 66 00 7d 00  5c 00 00 00 00 00        9.6.f.}......

The reparse attribute tag is defined by the first series of values, which identifies the reparse point as a Microsoft Name Surrogate Mount Point and specifies the surrogate name to be substituted for the original name. The reparse data follows the attribute tag values and, in this case, provides the fully expressed surrogate name.

Reparse points are also used by file-system filter drivers to mark files so that they are used with that driver. When NTFS opens a file associated with a file-system filter driver, it locates the driver and uses the filter to process the file as directed by the reparse information. Reparse points are used in this way to implement Remote Storage, which is discussed in the next section.

Often, scientific or other data collected through sampling is stored in large files that are primarily empty except for sparsely populated sections that contain the actual data. For example, a broad-spectrum signal recorded digitally from space might have only several minutes of audio for each hour of actual recording. In this case, a multiple-gigabyte audio file such as the one depicted in Figure 15-5 might have only a few gigabytes of meaningful information. Because there are large sections of empty space and limited areas of meaningful data, the file is said to be sparsely populated and can be referred to as a sparse file.

Figure 15-5. This figure shows sparse file usage.

Stored normally, the file would use 20 GBs of space on the volume. If you mark the file as sparse, however, NTFS allocates space only for actual data and marks empty space as unallocated. In other words, any meaningful or nonzero data is marked as allocated and written to disk, and any data composed of zeros is marked as unallocated and is not explicitly written to disk. In this example, this means the file uses only 5 GBs of space, which is marked as allocated, and has unallocated space of 15 GBs.

For unallocated space, NTFS records only information about how much unallocated space there is, and when you try to read data in this space, it returns zeros. This enables NTFS to store the file in the smallest amount of disk space possible while still being able to reconstruct the file’s allocated and unallocated space.

In theory, all this works great, but it is up to the actual program working with the sparse file to determine which data is meaningful and which isn’t. Programs do this by explicitly specifying the data for which space should be allocated. In Windows Server 2012 R2, several services use sparse files. One of these is the Indexing Service, which stores its catalogs as sparse files.

Using the FSUtil Sparse command, you can easily determine whether a file has the sparse attribute set. Type fsutil sparse queryflag FilePath at the command prompt, where FilePath is the path to the file you want to examine, such as:

fsutil sparse queryflag c:datacatalog.wci0010002.ci

If the file has the sparse attribute, this command returns:

This file is set as sparse

You can examine sparse files to determine where the byte ranges that contain meaningful (nonzero) data are located by using FSUtil Sparse as well. Type fsutil sparse queryrange FilePath at the command prompt, where FilePath is the path to the file you want to examine, such as:

fsutil sparse queryrange c:datacatalog.wci0010002.ci

The output is the byte ranges of meaningful data within the file, such as:

sparse range [0] [28672]

In this particular case, the output specifies that there’s meaningful data from the start of the file to byte 28672. You can mark files as sparse as well. Type fsutil sparse setflag FilePath at the command prompt, where FilePath is the path to the file you want to mark as sparse.

Windows Server 2012 R2 supports transactional NTFS and Self-Healing NTFS. Transactional NTFS allows file operations on an NTFS volume to be performed transactionally. This means programs can use a transaction to group sets of file and registry operations so that all of them succeed or none of them succeed. While a transaction is active, changes are not visible outside the transaction. Changes are committed and written fully to disk only when a transaction is completed successfully. If a transaction fails or is incomplete, the program rolls back the transactional work to restore the file system to the state it was in prior to the transaction.

Transactions that span multiple volumes are coordinated by the Kernel Transaction Manager (KTM). The KTM supports the independent recovery of volumes if a transaction fails. The local resource manager for a volume maintains a separate transaction log and is responsible for maintaining threads for transactions separate from threads that perform the file work.

By using the FSUtil Transaction command, you can easily determine transactional information. You can list currently running transactions by typing fsutil transaction list at the command prompt. You can display transactional information for a specific file by typing fsutil transaction fileinfo FilePath at the command prompt, where FilePath is the path to the file you want to examine, such as:

fsutil transaction fileinfo c:journalls-dts.mdb

Traditionally, you had to use the Check Disk tool to fix errors and inconsistencies in NTFS volumes on a disk. Because this process can disrupt the availability of Windows systems, Windows Server 2012 R2 uses Self-Healing NTFS to protect file systems without having to use separate maintenance tools to fix problems. Because much of the self-healing process is enabled and performed automatically, you might need to perform volume maintenance manually only when the operating system notifies you that a problem cannot be corrected automatically. If such an error occurs, Windows Server 2012 R2 notifies you about the problem and provides possible solutions.

That said, with Windows 8.1 and Windows Server 2012 R2, self-healing has been enhanced and extended to work better with Check Disk. These improvements enable you to use Check Disk to correct many types of inconsistencies and errors on live (online) volumes, whereas Check Disk previously could perform these types of corrections only with offline volumes.

By using Self-Healing NTFS, the file system is always available and does not need to be corrected offline (in most cases). Self-Healing NTFS does the following:

Although Self-Healing NTFS can correct many types of inconsistencies and errors automatically, some issues can be resolved only by running Check Disk (Chkdsk.exe) and allowing Check Disk to work with NTFS to resolve the problems, as discussed earlier in this chapter under “NTFS features.”

As Table 15-5 shows, ReFS maintains compatibility with key aspects of NTFS, particularly when it comes to security features such as access permissions and share permissions. However, ReFS diverges when it comes to extended features, including support for compression, encryption, and disk quotas. Furthermore, you cannot boot from ReFS or use ReFS with removable media.

Not only are the transactional and self-healing features of NTFS important components of ReFS, but ReFS extends these features in several ways to allow for the automatic verification and online correction of data. ReFS avoids the possibility of torn writes by not writing metadata in place and optimizes for extreme scale by using scalable structures. To provide full end-to-end resilience, ReFS integrates fully with Storage Spaces. This integration does the following:

ReFS reuses the code that implements the file-system semantics of NTFS to ensure compatibility with existing file-system application programming interfaces (APIs). This ensures that the core of the file-system interface is the same and that file operations—including read, write, open, change notification, and close—work in exactly the same way. When working with ReFS, Windows maintains the in-memory file and volume state, enforces security, and maintains memory caching and the synchronization of file data in exactly the same way as with NTFS.

Where NTFS and ReFS differ greatly is in the on-disk store engine underneath the file-system interface. The on-disk store engine is what implements the on-disk structures such as the MFT. As discussed earlier in the chapter, the MFT represents files and directories by storing a file record of each file and folder on the volume along with pertinent volume information and details on the MFT itself.

The on-disk store engine for NTFS is NTFS.SYS. The on-disk store engine for ReFS is REFS.SYS. REFS.SYS was designed specifically for ReFS.

ReFS uses B+ tree structures to represent all information on the disk. B+ trees scale well from very small, compact structures to very large, multilevel structures, and using B+ trees simplifies the architecture and reduces the size of the code base.

The on-disk store engine uses enumerable tables with sets of key-value pairs. Access into most tables is provided by a unique object identifier, which is stored in a special object table that forms the base of the B+ tree.

The object table at the base of the B+ tree contains a disk offset and checksum for each unique object ID. This makes the object table the root of all structures within the file system. The entries in the object table refer to directories and global system metadata.

As shown in Figure 15-6, directories are represented as tables rooted within the object table. Each directory has an object identifier that acts as a key in the object table, and it has a corresponding value that provides a disk offset for where the table is found on the volume along with a checksum. The directory table contains rows that identify the files in the directory by file name and metadata. File metadata, in turn, identifies file attributes and their actual values. Among these values is a table of offset mappings to file extents. This table contains rows identifying file extents, paired with values that provide the disk offset location for each file extent and an optional checksum. Each file extent contains a section of the data for the parent file.

Figure 15-6. This figure shows file structures in ReFS.

Put another way, directories are represented as tables in the file structure. Files are embedded within rows of a directory table and are themselves tables containing rows of file metadata. The file metadata also represented as a table has a row for each file attribute paired with the related value. Within the file metadata is an embedded table containing rows that identify file extents and provide offset locations to the extents on the volume along with optional checksums.

Other global structures are represented within the file system as tables as well. As an example, ACLs are represented as tables rooted within the object table.

ReFS supports file sizes up to 264 – 1 bytes, 264 files in a directory, 264 directories on a volume, and volume sizes up to 278 bytes using 16 KB cluster sizes (in contrast, Windows stack addressing allows 264 bytes). Because B+ trees scale with extreme efficiency, ReFS volumes can perform well whether they contain very large directories, very large files, or both. Disk space allocation is managed using a hierarchical allocator. This allocator represents free space as tables of free-space ranges. Each table has a different level of granularity so that large free-space ranges can be allocated as easily as medium or small free-space ranges, and all are relative to the volume size and available free space.

One of the disadvantages of NTFS is that metadata is maintained in place, and this can result in writes that are randomized and in torn writes. ReFS improves reliability and eliminates torn writes by using an allocate-on-write approach. Here, rather than updating metadata in place, the file system writes it to a different location. This technique sometimes is also referred to as shadow paging. The transaction architecture, derived from NTFS, is built on top of the allocate-on-write framework to provide failure recovery.

ReFS allocates metadata by using B+ tree structures that allow for fewer, larger reads and writes. It does this by combining related data, such as stream allocations, file attributes, file names, and directory pages. The approach offers read/write efficiencies whether hard disk drives or solid-state drives are used.

ReFS and Storage Spaces were designed to work together. Using mirroring or disk striping with parity, Storage Spaces can safeguard data against disk failures by maintaining copies of data on multiple disks. Whether you are using NTFS or ReFS, these multiple copies of data enable Storage Spaces to correct read failures by reading alternate copies of data, to correct write failures by reallocating data transparently, and to correct complete media loss on read/write. Storage Spaces gains efficiencies with ReFS when it comes to detecting data corruption and lost and misdirected writes. Here, ReFS can detect metadata corruption and lost and misdirected writes by using its checksums and then interface with Storage Spaces to read all the available copies of metadata and choose the correct one by validating the checksum. Next, ReFS instructs Storage Spaces to fix the bad metadata by using the good copies of the metadata. The error detection and correction happens transparently. When integrity streams are enabled for files, this automatic error detection and correction process is applied to each individual extent of a file as well.

ReFS supports two types of data streams: conventional streams and integrity streams. Conventional streams behave identically to NTFS streams but might have metadata associated with them that is integrity protected. Integrity streams are streams that are integrity protected, meaning data is checksummed and updates to data are handled using copy-on-write.

With ReFS, keep in mind that integrity is an attribute that can be applied to files and directories. When a file or directory has the (FILE_ATTRIBUTE_INTEGRITY_STREAM) integrity attribute, it uses integrity streams to protect against data corruption. Only Storage Spaces with redundancy has integrity streams enabled by default.

The integrity attribute is inheritable. When you enable the integrity attribute on a directory, the attribute is inherited by all files and directories created in the directory. Because of this, if you enable the integrity attribute on the root directory of a volume, you can ensure that every file and directory on the volume uses integrity streams by default.

You can enable integrity streams on the root directory of a volume when you format it. Use the following command syntax:

format /fs:refs /i:enable Volume

Here, Volume is the drive designator for the volume to format, such as:

format /fs:refs /i:enable m:

For empty files, the integrity attribute can be set and unset. For nonempty files, the integrity attribute can be removed only by moving the file to a file system that doesn’t support integrity, such as NTFS.

ReFS safeguards against data loss as a result of parts of a volume becoming corrupted over time by periodically scrubbing all metadata and integrity stream data. Data is scrubbed by reading all the redundant copies and validating their corrections by using checksums. If checksums do not match, bad copies are repaired using good copies. Typically, this automatic process occurs only with Storage Spaces that have redundancy enabled.

If metadata or data corruption cannot be automatically repaired, ReFS performs a salvage operation to remove the corrupt metadata or data from the namespace. The salvage operation ensures that irreparable corruption cannot adversely affect sound data. For example, the file system cannot open or delete a corrupt file or directory. By removing the corrupt file or directory, ReFS ensures that an administrator can recover the file or directory from backup or have an application re-create it. When ReFS is running on top of redundant Storage Spaces with integrity streams, an automatic error-detection and correction process, applied to each extent of a file, can recover file and directory data.

Windows allows you to enable compression when you format a volume by using NTFS. When a drive is compressed, all files and folders stored on the drive are automatically compressed when they are created. This compression is transparent to users, who can open and work with compressed files and folders just as they do regular files and folders. Behind the scenes, Windows expands the file or folder when it is opened and compresses it again when it is closed. Although this can decrease a computer’s performance, it saves space on the disk because compressed files and folders use less space.

You can turn on compression after formatting volumes as well or, if desired, turn on compression only for specific files and folders. After you compress a folder, any new files added or copied to the folder are compressed automatically. If you move a compressed file to a folder on the same volume, it remains compressed. If you move a compressed file to a folder on a different volume, it inherits the compression attribute of the folder.

Moving uncompressed files to compressed folders affects their compression attribute as well. If you move an uncompressed file from a different drive to a compressed drive or folder, the file is compressed. However, if you move an uncompressed file to a compressed folder on the same NTFS drive, the file isn’t compressed. Finally, if you move a compressed file to a FAT16, FAT32, exFAT, or ReFS volume, the file is uncompressed because NTFS compression is not supported.

To compress or expand a drive, follow these steps:

You can selectively compress and expand files and folders as well. The advantage here is that this affects only part of a disk, such as a folder and its subfolders, rather than the entire disk. To compress or expand a file or folder, follow these steps:

Windows Server 2012 R2 also provides command-line utilities for compressing and expanding your data. The compression utility is called Compact (Compact.exe). The expansion utility is called Expand (Expand.exe).

You can use Compact to determine quickly whether files in a directory are compressed. At the command line, change to the directory you want to examine and type compact without any additional parameters. If you want to check the directory and all subdirectories, type compact /s. The output lists the compression status and compression ratio on every file, and the final summary details tell you exactly how many files and directories were examined and found to be compressed, such as:

Of 15435 files within 822 directories
0 are compressed and 15435 are no compressed.
2,411,539,448 total bytes of data are stored in 2,411,539,448 bytes.
The compression ratio is 1.0 to 1.

Compressed (zipped) folders are another option for compressing files and folders. When you compress data by using this technique, you use zip compression technology to reduce the number of bits and bytes in files and folders so that they use less space on a disk. Compressed (zipped) folders are identified with a zipper on the folder icon and are saved with the .zip file extension.

Compressed (zipped) folders have several advantages over NTFS compression. Because zip technology is an extension of the operating system rather than of the file system, compressed (zipped) folders can be used on any type of volume. Zipped folders can be password protected to safeguard their contents and can be sent by email. They can also be transferred using File Transfer Protocol (FTP), Hypertext Transfer Protocol (HTTP), or other protocols. An added benefit of zipped folders is that some programs can be run directly from compressed folders without having to be expanded. You can also open files directly from zipped folders.

You can create a zipped folder by selecting a file, folder, or group of files and folders in File Explorer, pressing and holding or right-clicking it, pointing to Send To, and tapping or clicking Compressed (Zipped) Folder. The zipped folder is named automatically by using the file name of the last item selected and adding the .zip extension. If you double-tap or double-click a zipped folder in File Explorer, you can access and work with its contents. As shown in Figure 15-10, the zipped folder’s contents are listed according to file name, type, and date. The file information also shows the packed file size, original file size, and compression ratio. Double-tapping or double-clicking a program in a zipped folder runs it (as long as it doesn’t require access to other files). Double-tapping or double-clicking a file in a zipped folder opens it for viewing or editing.

Figure 15-10. Compressed (zipped) folders can be accessed and used like other folders.

While you’re working with a zipped folder, you can perform tasks similar to those you can do with regular folders. You can do the following:

You can also perform additional tasks that are unique to zipped folders. Press and hold or right-click and then choose Extract All to start the Extraction Wizard, which you can use to extract all the files in the zipped folder and copy them to a new location.