Virtual File System

Much like the unified file system concept common to Unix and Linux systems (as discussed in Chapter 5), PyFLAG features a unified Virtual File System (VFS) for all objects under examination. PyFLAG refers to each of these items as inodes. Each item loaded into the PyFLAG database receives a PyFLAG inode, in addition to the internal file system metadata address it may already have. This means that any number of file system images, network traffic captures, standalone log files, and even streams of unstructured data may be loaded under the same virtual root and subsequently processed with PyFLAG.

Additionally, as an examination using PyFLAG proceeds it is not uncommon to discover nested files (such as files inside of archives, or files recovered via carving or extracted from a network packet capture). These nested files will be allocated a PyFLAG inode chained under their parent item. These chained inodes will be indicated by a pipe character (|).

Scanners

PyFLAG’s scanners are discrete modules with a specific task. A scanner will run over objects loaded into the VFS and perform a function. For example, the PstFile.py scanner will run over inodes loaded into the VFS, determine if they are an Outlook PST, and process them accordingly. This processing populates the VFS with inodes referencing any discovered discrete items inside of the PST, which are then rescanned with any additional loaded scanners. Using discrete scanners coupled with the inode creation and nesting mentioned earlier can enable an examiner to fairly rapidly gain deep knowledge of the true contents of the file system in question.

SQL Queries

The bulk of the PyFLAG GUI is simply a front end to submitting SQL queries to the back-end database, and then formatting and displaying the results. One nice aspect of this division is that the database can be queried and manipulated directly, without the necessity of the browser. Indeed, many of the more advanced functions of PyFLAG require direct database interaction.

If you are not familiar with SQL queries, however, PyFLAG should still be usable and may, in fact, assist you in learning SQL. On each set of responses displayed by the PyFLAG GUI that are results of a SQL query, there will be a “SQL Used” Icon that resembles a hand holding some traditional database icons. Clicking this will display the SQL query the GUI issued to the database to generate the page in question.

Additionally, PyFLAG uses SQL to do all of its display filtering. In any of the filtering dialogues, you can enter SQL directly or you can use the provided “helper” functions that display available tables and operators.

Keyword Searching and Indexing

PyFLAG does not offer “on demand” keyword scanning. This isn’t necessarily a shortcoming, as performing a full file system scan for each keyword is incredibly time-consuming on large file systems. Instead, PyFLAG offers indexed searching. PyFLAG builds an index based on a dictionary file, which allows for very fast searching. However, it only builds this index when a new source is loaded into a case. As the PyFLAG dictionary is empty after installation, this effectively means that unless the examiner takes steps to set up a dictionary, keyword searching won’t be possible.

The PyFLAG FAQ contains a handy set of commands to populate the index with a large number of keywords sourced from the wordlist file included for spellcheck support on most Linux distributions:

~/pyflag$ grep -E ...+ /usr/share/dict/words > keywords.txt

~/pyflag$ pyflag_launch ./utilities/load_dictionary.py keywords.txt

This retrieves all three-letter or longer words and loads them into the PyFLAG database. Note that this technique can be extended to load any arbitrary set of keywords for indexing.

Carving and Advanced Carving

PyFLAG supports file carving in a fairly novel way, through use of the VFS. Using simple header/footer signature carving, PyFLAG carvers generate VFS inodes corresponding to the location and run length of the corresponding hit. The only overhead from carving is the time and cycles required to perform the carving rather than the heaps of disk space required for a standard carving operation. Additionally, PyFLAG carvers will run over all inodes present in the VFS, to include inodes generated by scanners. Given this, it is possible to extract embedded files from within archives without performing the intermediary extraction and decompression phases out to disk.

PyFLAG also supports a handful of advanced carvers that use built-in knowledge of the file structures they are carving. The three advanced carvers currently supplied with PyFLAG are for PDFs, JPEGs, and ZIP archives.

Log Analysis

As the name belies, PyFLAG is equally intended to be used for log analysis as well as file system forensic analysis. Because logs entries are stored in the PyFLAG database and are indexed in the same manner as all other content, keyword searches across the database will produce log results in addition to file system results. PyFLAG currently has native knowledge of the Windows binary event log format, IIS logs, Apache Logs, and Linux IPTables logs, but also supports generating log templates based on a source log file.

Network and Memory Forensics Support

While an exploration of these capabilities is outside the scope of this book, it is important to be aware that one of PyFLAG’s greatest strengths is its ability to intelligently parse various application-layer networking protocols. Network protocols currently supported in PyFLAG include HTTP Web traffic, SMTP and POP mail transfer, IRC chat, and VOIP telephony.

PyFLAG also interfaces with the Volatility framework, an open source framework for performing forensic analysis memory images from Windows and Linux systems. Items recovered from network or memory captures will be populated into the VFS and can be processed further using any of the appropriate scanners.

An excellent walkthrough of performing a unified investigation using PyFlag is available in Michael Cohen and AAron Walters’ write up from the 2008 Digital Forensics Research Workshop Forensics challenge [2]. To demonstrate basic usage of PyFLAG for file system forensic analysis, we will install the application and explore some of the features available using Lance Mueller’s first Forensic Practical Image, available at http://www.lancemueller.com/blog/evidence/WinXP2.E01.

Installation

PyFLAG installation is a bit more involved than many of the other packages we have used up to this point. It has many components and is not currently included in any package management systems. In addition, the most recent version is only available via the darcs source code management system. We will need to start out by making sure we have all the dependencies required for PyFLAG’s operation. The install document (available online at http://www.pyflag.net/pyflag/INSTALL or with the PyFLAG source) states the following packages are required and handily provides the Debian/Ubuntu package names:

Additionally, we will need the “mysql-server,” “python-sqlite,” and “python-pyparsing” packages. We have some of these already, but requesting them again will not hurt anything, so we can install all of these (and their dependencies, and so on) in one shot with the following command:

sudo apt-get install build-essential python-dev libz-dev libmagic-dev python-mysqldb python-imaging python-pexpect python-dateutil python-urwid mysql-server python-sqlite python-p6

PyFLAG can provide additional, optional functionality if the following packages are installed:

Because we installed AFFLIB and LibEWF from source in Chapter 2, we can skip these and only install the other packages using:

sudo apt-get install libgeoip-dev libjpeg62-dev clamav-daemon

With all the prerequisites installed, we can fetch the current development branch of the PyFLAG source code. PyFLAG uses darcs revision control software. If you don’t have this installed, you can rectify this with apt-get. We can fetch a copy of the current source to work with using the following command:

darcs get --partial http://www.pyflag.net/pyflag

Once the command completes we will have a new directory named pyflag in our current directory. We will need to move into this directory and execute the included “autogen.sh,” which will set up the build environment.

sh autogen.sh

This will create the configure script and associated files, so once this is completed a standard build will complete the process:

./configure

make

sudo make install

The next step is to start up PyFLAG.

user@ubuntu:~/source/pyflag$ sudo pyflag

Checking schema for compliance

31172(Warning): Error: Unable to connects - does the DB Exist?: (1049, "Unknown database ’pyflag’ ")

31172(Warning): Error: Unable to connects - does the DB Exist?: (1049, "Unknown database ’pyflag’ ")

31172(Infomation): Serving PyFlag requests on http://127.0.0.1:8000

Note that the errors are completely normal—the “pyflag” database does not exist the first time you run PyFLAG. Setting this up is the first thing we have to do.

Once we open a Web browser and navigate to the provided URL, we will be prompted to initialize the database. If we check the “Upgrade the database?” box and click Submit, PyFLAG will build the “pyflag” database using the appropriate schema.

Before continuing further, we need to confirm (and possibly change) some PyFLAG configuration data. There are two primary configuration directives of concern: uploaddir and resultdir. These determine where PyFLAG looks for input files and where it places output files. Because PyFLAG will not be able to read or write to the file system outside of these directories, ensure that they exist and make sure to place any data you plan on analyzing with PyFLAG in the uploaddir directory.

Usage

Once the database is created and our configuration options are acceptable, the next step is to start a new case (Case Management -> New Case). Give your case a memorable name and click Submit.

Next, we will need to add an image file to process using PyFLAG. Select Load Data -> Load IO Data Source. You should see the screen shown in Figure 9.1.

Using our example image shown in Figure 9.1, we will select “EWF” as the IO Subsystem. After hitting Submit, we see the screen in Figure 9.2. Note that PyFLAG performs some preliminary analysis of the loaded data, hinting at the content in the red highlighted text.

Because this is a file system image, we will select “Sleuthkit” as the file system type. This tells PyFLAG what driver to use to analyze the structure of the data source we are loading. Before proceeding, we will need to provide a VFS Mount Point for the data source. This is like creating a mount point when mounting a volume on a Linux system. We can put whatever we would like here, but it is best to choose something relevant to the source material, such as “WinXP2.”

Once the file system driver loads, you should get a file browser view similar to Figure 9.3.

From here we can browse through the file system using the GUI, export files, and so on. Before we do that, though, we can exercise some of PyFLAG’s more interesting capabilities. Selecting Load Data -> Scan Filesystem takes us to the screen shown in Figure 9.4.

This is the main configuration panel that determines which scanners are run against the VFS (or a subsection defined under the “Scan Files” entry). Each entry shown is a category, and clicking on the tool icon to the left of each lets an examiner drill down and enable or disable specific scanners. Once we are done setting which scanners we want, we can click “Submit” to kick off the scanning process. This yields the display seen in Figure 9.5.

After scanning is complete, you are returned to the file system browser. Any new inodes generated by the scanner will be populated into the VFS automatically.

A frequent task in forensic analysis is examination of files of a specific type—only images, for example. PyFLAG has a function that displays all files by type under Disk Forensics -> Browse Types. This simply displays all files on the system for which a type was able to be determined. We can utilize the filtering capabilities of PyFLAG to restrict this to only JPEG images, for example. Selecting the Filter icon brings up the page shown in Figure 9.6.

We can use the “Column” and “Operators” drop-downs to generate a Filter without knowing any SQL. Simply select the column or operator of interest, position the cursor in the Search Query field, and select “Insert.” When the query is completed, click “Submit.” See Figure 9.7 for a subset of the resulting JPEG-only output.

PyFLAG has another method for viewing files of a specific type under Disk Forensics -> Feeling Lucky. These are canned sets of queries useful for retrieving specific sets of data. In Figure 9.8, this was used to retrieve all graphics files from the file system.

As mentioned before, we can use the “SQL Used” button to force PyFLAG to “show its work,” so to speak. The segment that follows is the SQL statement used in the graphics query in Figure 9.8.

select inode.inode_id as ’Thumbnail’,’inode’.’size’ as ’Size’,file.link, concat(file.path,file.name) as ’Filename’ from ’inode’ join ’file’ on ’inode’.inode_id = ’file’.inode_id where ((1) and (( ’inode’.’inode_id’ in (select inode_id from type where type like ’%image%’)) and ’inode’.’size’ > ’20000’)) order by ’inode’.’size’ desc

Note that this is but a small sampling of what you can do with PyFLAG. Many more extensive tutorials are available on the PyFLAG wiki.

Virtual File System

Much like PyFLAG, DFF also creates a Virtual File System under which all subsequent items are loaded. DFF refers to unique items as nodes. The core of the DFF application is tasked with handling the creation of VFS nodes in memory. All display tasks are handled by a GUI layer, and populating nodes with data is handled by modules. The Virtual File System is viewed as a tree structure in the DFF GUI, with all nodes appearing as children under the root (/ ) or as children of other nodes.

Modules and Scripts

DFF modules are similar to PyFLAG’s scanners. A module performs a discrete task and populates a node with the output of that task. The current version of DFF (0.8 at the time of this writing) comes with many modules that perform various tasks, including processing cell phone memory dumps, performing hash comparisons, viewing movies and images, and generating statistics about a node or set of nodes. Because all “analysis tasks” in DFF are performed via modules, the exposed application programming interface (API) has a great deal of functionality available. Additional modules can be written in C++ or Python. In addition, scripts can be generated on the fly using the built-in Python scripting engine.

GUI and Command-Line Capabilities

One of the strongest features of the Digital Forensics Framework is the clean, simple GUI. The familiar three-pane layout is a paradigm used by many GUI applications so even a novice should not have much trouble getting oriented in the application. Additionally, all functions performed via modules from the GUI can also be performed via the DFF shell. The DFF shell can be launched from within the GUI or can be used standalone. This allows the examiner to use the same tool whether she is connecting via a remote shell or sitting at a local desktop.

Installation and Prerequisites

DFF is available via a binary installer package for most Linux distributions and Windows and can be compiled from source on Windows, Linux, and OS X systems. In this example, we compile DFF 0.8.0 from source.

As always, first we need to install the program’s dependencies. DFF’s GUI is based on the QT4 library, so we will need to install a host of packages related to this. The following command will get us all the packages we need to build DFF successfully.

user@ubuntu:~$ sudo apt-get install swig1.3 cmake python-qt4-dev pyqt4-dev-tools qt4-qmake libqt4-dev libqscintilla2-5 libqscintilla2-dev python-qscintilla2 python-magic python-qt4-phonon

DFF uses the cmake build system, so to start the process off we run “cmake” in the DFF’s unpacked source directory.

user@ubuntu:~/source/dff-0.8-src$ cmake .

This will generate the makefiles we need to compile. Once complete, we can run make and then make install.

user@ubuntu:~/source/dff-0.8-src$ make

...

user@ubuntu:~/source/dff-0.8-src$ sudo make install

Usage

We will use the same image as before to explore the usage of DFF (WinXP2.E01). At this point, DFF does not support Expert Witness Format images; however, as demonstrated in Chapter 2, we can create a virtual mount point for the EWF container and expose the content as a “raw” image, which DFF can process using the following series of commands.

First we’ll create a mount point for the container:

user@ubuntu:~$ mkdir ~/mnt

user@ubuntu:~$ mkdir ~/mnt/ewf

Next we will mount the EWF container:

user@ubuntu:~$ mount_ewf.py ~/images/WinXP2.E01 ~/mnt/ewf/

Using libewf-20100226. Tested with libewf-20080501.

Finally, we can confirm a proper mount by listing the directory we mounted the EWF to:

user@ubuntu:~$ ls -lath /home/user/mnt/ewf/

total 2.0G

drwxr-xr-x 3 user user 4.0K 2010-11-26 09:26 ..

dr-xr-xr-x 2 root root 0 1969-12-31 16:00 .

-r--r--r-- 1 root root 2.0G 1969-12-31 16:00 WinXP2

-r--r--r-- 1 root root 284 1969-12-31 16:00 WinXP2.txt

With this set up, we can start up DFF. We will pass the “-g” argument to tell DFF to start the GUI—without this flag, DFF defaults to shell operation.

user@ubuntu:~$ dff -g

Once the GUI comes up, the first thing we need to do is have it read in our image file. To do so, we will select “Add Dump” from the File menu (Figure 9.9). DFF uses “Dump” to refer to a bitstream image of a device, as opposed to a physical device.

After browsing to and opening our FUSE-mounted image file (/home/user/mnt/ewf/WinXP2 in our example), we get Figure 9.10. Note that adding the dump did just that, nothing more. We need to tell DFF to process the dump file further.

We can examine the attributes of the file on the right to determine that it is an NTFS volume. To process the NTFS structures in the dump, we can select Modules -> File System -> NTFS (Figure 9.11).

The NTFS parsing module has a few configurable options, as seen in Figure 9.12. Optional fields are off by default—the checkbox to the right enables editing them.

We provide a name for the node we are creating with this module (/WinXP2) and hit “OK.” The Task Manager pane at the bottom of the window should provide a status update as the NTFS processing tasks proceed. Once completed, we are presented with a standard tree file browsing structure, which we can use to examine the file system interactively. For an example of navigating DFF using the thumbnail view, see Figure 9.13.

One of the interesting modules included in DFF is the Fileschart module (Modules -> Statistics -> Fileschart). This performs cursory file analysis of all nodes and displays a graphical overview of the types of files present in the VFS (Figure 9.14).

DFF also includes a carving module. Currently the carving mechanism used is a simple header search, and the types of files supported are fairly limited. Figure 9.15 shows an active file carving process.

Carved files will be added to the VFS under the node they were found in under a new directory named “carved” (Figure 9.16).

DFF contains a number of native viewer modules—an image viewer for graphics files, a movie player for video files, and a text viewer for text files. Unknown or binary files will be displayed using the hexedit module. Among hexedit modules features is the ability to search the file content for hexadecimal, ASCII, or Unicode strings. In Figure 9.17 we searched for “hbin” in the Adminstrator’s NTUSER.DAT file and received 91 results. Clicking on the address of each results in the Offset pane in the lower right will jump to that hit.

The hexedit module also allows the user to highlight and bookmark selected segments of data. In Figure 9.18 we have highlighted and bookmarked a registry key of interest.

The DFF unzip module (under archive) allows an examiner to examine the contents of a zip archive without extracting data first. In Figure 9.19 we launched the unzip module by right clicking on a Zip archive and selecting the “unzip” module from the context menu.

The archive content is added as a child node of the Zip archive under the VFS. We can now navigate down into the archive and examine the content, including the embedded time stamp (shown in Figure 9.20).

Fiwalk-based Tools

The main fiwalk program is written in C in the interest of speed. However, the fiwalk project also provides a Python module (fiwalk.py) so that forensic applications leveraging fiwalk’s capabilities can be developed quickly and simply. To demonstrate this, several demonstration programs that use fiwalk.py are included. These can be used as is or as a basis for additional programs leveraging the fiwalk core.

One example is iredact.py, which is a disk redaction program designed to enable the intelligent removal of files meeting specific criteria from a disk image. The idea is to remove sensitive or otherwise nonshareable information from a disk image prior to making this image available to third parties. Altering a forensic image is always a fraught with peril, but because iredact.py is designed with forensic needs in mind, it avoids oversterilization or disturbing unallocated data.

Redaction is controlled by sets of condition:action pairs. Conditions can be a filename, a wildcard filename, a directory name, an MD5 or SHA1 hash, or file or sector content. Actions can be to fill the redacted content with a specific hex value, to encrypt the content, or to “fuzz” the content, which makes executables nonoperational but does not alter their strings.

Before and After

Before and after are incredibly simple concepts, but can be applied to forensic examinations in interesting ways. Any discrete point in a timeline can be described as having occurred before or after any other point or event. Of course the converse is also true—an event (event B) dependent on another event (event B) occurring could not occur prior to that event. The canonical example of this is the creation of the file system being examined. This is the canonical “start point” for all file system time points. Any time information indicating a date before this is falsified, either intentionally or as a consequence of retaining original times during an archive extraction, as demonstrated in Chapter 8.

Another common example of this type of temporal investigation is examining all activity that occurred after a specific event of interest. In an intrusion investigation, this event will be the initial compromise time. This event can act as a stop point with regards to relevant time information—anything that occurred prior to the initial point of compromise is now irrelevant. This of course assumes you are able to make a definitive judgment about the initial point of compromise. In some investigations, this lower bound on time analysis may shift as new information is uncovered.

As expected, examination of items that occurred before a specific event is the opposite. In an intrusion investigation, the bounding event for this can be the last known traffic from an attacker’s IP address. One of the goals of examining everything that occurred before an event is identification of the initial point of compromise. Given both time points, you can establish a finite window of time data that may be of relevance.

During

This describes a set of time data with a finite beginning and end—the duration of the event. The prime example of this sort of time information is the window of compromise during an intrusion. Given an attacker’s entry, and the time of successful remediation, the time in between is the duration of compromise. This is often the largest time period of immediate relevance. Within this window, additional durations may be uncovered. For example, windows between logins and logouts of compromised accounts may provide additional windows of access to investigate further. This can help reduce the amount of time information of relevance the examiner needs to process.

This can also help reduce the amount of data considered to be compromised. For example, if sensitive data were not on a compromised system outside of the window of access, it may be reasonable to consider that there was no opportunity for data to be compromised. Conversely, identifying the window of access can be used to reduce the scope of investigation to data that existed on the system during this window. Limiting examination to a particular time window can also be useful when attempting to attribute particular activity to a certain user account. Any activity that has specific start and stop times can be used to partition a subset of time data.

1. January 10 2010 2:20:27 pm—File1.doc is created on the system. MFT entry 4067 is populated with metadata relevant to File1.doc. File content for File1.doc is written to a nearly full volume, leading to a 5.9-Megabyte file with 67 fragments.

2. January 10 2010 2:21:58 pm—File1.doc is deleted, freezing the MFT entry. All clusters in the 67 data runs contain File1.doc data at this time.

3. {Sometime after March 10 2010}: The Florida orange crop report is created on this volume. An unknown MFT entry is created for this file, and clusters are allocated. Some or all of these clusters are in a run, which overwrites some of the data runs previously allocated to File1.doc.

4. {Sometime after entry 3}: The Florida orange crop report is deleted. The runs allocated to this file are left intact.

5. {Sometime after entry 4}: The MFT entry created for the Florida orange crop report is reused by a new file. Metadata and data run information contained in the MFT entry are overwritten and lost.