Understanding Document Tracking
Before discussing what you can do with document tracking, it is useful to have a conceptual understanding of how document tracking is implemented in BizTalk Server. The goal of this section is to provide a high-level introduction to the BizTalk Server Document Tracking System.
The Tracking Data Storage and Retrieval System
At the heart of BizTalk Document Tracking is a SQL Server database—called InterchangeDTA by default—where information about documents that have been processed by BizTalk Server is stored. As documents move through BizTalk Server, document data is stored to the InterchangeDTA database by the tracking engine.
All databases have two primary activities: data storage and data retrieval. Your document tracking tasks can therefore be summarized as follows:
Configuring what document data goes into the InterchangeDTA database
Configuring queries to retrieve tracking information from the InterchangeDTA database
Configuring the BizTalk Document Tracking system does not require that a single line of code be written. It is simply a matter of using the various graphical tools and dialog boxes provided by BizTalk Server. If you are disappointed by the lack of coding required, then you will be happy to hear that the BizTalk Document Tracking solution is wide open for extension. You can access the tracking database through Microsoft SQL Server Enterprise Manager in the same manner that you access any other SQL Server database. This means that you can design and write custom queries against the tracking database. BizTalk Server even exposes COM interfaces that allow you to configure and extend the tracking system. The topic of writing custom code for tracking is covered in “Developing Custom Tracking Solutions” later in this chapter.
What Is Stored by the Tracking System
The unit of work in BizTalk Server is the interchange. An interchange consists of the transmission of a document (or several documents packaged together) from one location to another. It should therefore come as no surprise that the primary job of BizTalk tracking is to capture information about interchanges. When the tools provided by BizTalk Server are used to query for tracking data, the result of these queries is always a set of interchanges.
At the highest level, BizTalk Server categorizes an interchange as either an inbound or outbound interchange. Inbound interchanges are documents that have been submitted to BizTalk Server for processing. Outbound interchanges are made up of documents that are leaving the BizTalk messaging system and are on their way to an external destination. The direction of an interchange (inbound or outbound) affects how it is logged by the tracking system.
BizTalk Server stores metadata about interchanges including the organizations involved, processing date and time stamps, and routing information. Specific field values within a document can be selected for tracking. Entire copies of documents can be maintained by the tracking database. If a document is transformed between formats during BizTalk Server processing, you can specify which of the formats to save for tracking purposes.
XML: The Key to BizTalk Document Tracking
You have already learned that all documents processed by BizTalk Server must contain well-formed XML. Documents may be submitted to BizTalk Server in formats other than XML, and documents may leave BizTalk Server in non-XML formats. Nevertheless, while the BizTalk Messaging Engine is processing a document, it requires the document to be in XML format.
There are many reasons why BizTalk Server mandates the use of XML within server boundaries. From the perspective of document tracking, the use of XML allows the tracking engine to parse documents and selectively extract data from the documents. Without having the documents in a standard format such as XML, there would be no way for BizTalk Server to understand enough about the documents to track data at the field level, and there would be no way for you to selectively configure which fields are tracked.
Document Processing Revisited
As mentioned earlier, BizTalk Server can retain entire copies of documents as they move through the server. To understand the capabilities of full document tracking, it is useful to review the stages that a non-XML document goes through during its journey through BizTalk Server.
When a document is submitted to BizTalk Server, it is placed in the work items queue. Placing inbound documents in a work items queue represents a form of load balancing, where the least busy server in your server group processes the inbound document. When the BizTalk Messaging Engine picks up the document, information about the inbound interchange is logged to the tracking database. One of the first steps in processing an inbound document is to check whether it is encrypted or encoded. BizTalk Document Tracking allows you to save MIME-encoded documents to the tracking database in their original format. This is a configurable option that you administer through the BizTalk Server Group properties.
Note
You learn how to configure tracking settings for an entire server group in the next section.
After the document's security requirements have been handled, the document is examined for routing data. If routing information is found in the document instance, this information is logged to the BizTalk Tracking database. Routing data may be specified in the document itself as well as in a receive function or through the parameters specified in IInterchange::Submit(). When routing data is specified in multiple locations, it is important to understand that the tracking engine will log the routing information specified in the document instance. The messaging engine may route the document based on the information specified in the receive function or the IInterchange::Submit() call, but the document's internal routing information is tracked.
A parser is then selected based on the metadata contained in the document's envelope. The job of a parser is to read the inbound document and transform it into XML format so that the BizTalk Messaging Engine can process the document. The parser may also save details about the inbound document to the tracking database.
Besides recording field-level data, you have the option to save the full document, in its original format, to the tracking database. After the document has been transformed into XML, BizTalk Server can once again save the entire inbound document (in its inter-mediate XML format) to the tracking database.
Tip
Having the inbound intermediate XML format saved to the tracking database can be useful for debugging, especially if you have written a custom parser that is not yet production ready.
Next, a channel is selected to process the document. Typical channel processing includes transforming the inbound document to another intermediate XML format. The resulting XML document complies with a BizTalk specification associated with the outbound document. BizTalk Server can save a copy of the outbound document, in XML format, to the tracking database.
The final steps are serialization and delivery. The outbound XML document is serialized to the native format expected by the recipient. After serialization has been completed, you may store the document in its native format to the tracking database. Routing information associated with the outbound interchange is logged for tracking purposes. For outbound interchanges, the tracking engine always logs the routing information contained in the channel.
As you can see, a lot of activity can be recorded by the document tracking engine during an interchange. BizTalk Document Tracking can be a valuable tool for understanding the stages that a document has passed through during its journey through BizTalk Server. Best of all, the logging details are fully configurable through a few simple dialog boxes.
Server Group Tracking Settings
BizTalk Server tracking settings can be configured at three basic levels:
The server group level
The document definition level
The channel level
This section discusses the tracking settings that can be configured at the server group level. Later in the chapter, we will cover document definition and channel tracking settings.
Configuring the Document Tracking and Activity (DTA) Database
All servers in a BizTalk Server group share certain properties relating to the document tracking system. The most obvious of these properties is the tracking database itself. A server group shares one document tracking database that is specified as part of the server group properties.
Server groups are configured and managed through the BizTalk Server Administration tool, shown in Figure 8.1. The tool is located in the Microsoft BizTalk Server 2002 program group.
Figure 8.1. The BizTalk Server Administration tool.
Tip
You can add the BizTalk Server Administration tool to an existing Microsoft Management Console file if the file is configured to allow authoring. If available, select the Add/Remove Snap-in option from the Console menu, click the Add button, and select Microsoft BizTalk Server Administration from the list of snap-ins.
To view the properties of a BizTalk Server group, right-click on the server group and select Properties. A dialog box appears with four tabs labeled General, Connection, Tracking, and Parsers. The first tab that is relevant to tracking is the Connection tab, displayed in Figure 8.2. This tab contains a group box labeled Tracking Database. Within this group box are controls for specifying the name, location, and security credentials for the document tracking database.
Figure 8.2. The Connection tab of the Server Group Properties dialog box.
Connection settings are intended to be configured during the initial setup of BizTalk Server. If you need to modify these settings, you should first stop all servers in the group. To stop a server, right-click on the server name under the BizTalk Server group and choose Stop. After you have finished editing the database connection information, restart each server by right-clicking on the server name and choosing Start.
Also be aware of the following issues when editing the tracking database server and name:
The target database must already exist. The tracking database must be created before you attempt to change the database name in the Server Group Properties dialog box.
When applying changes, a warning may appear stating that you need to shut down the BizTalk Server Interchange Application on all servers in the group before your changes will take effect. If you have already stopped all the servers in your group, then this warning can be safely ignored. When you restart the servers, the new tracking database settings will automatically take effect.
BizTalk Server will not automatically transfer existing tracking data from the original database to the new database. To move the tracking database from one server to another, you must back it up, restore it on the new server, and then follow the previous instructions for changing the connection settings in the Server Group Properties dialog box.
Additional Server Group Tracking Settings
The Server Group Properties dialog box contains a Tracking tab. Clicking on this tab causes the screen shown in Figure 8.3 to be displayed. The tab contains four check boxes for configuring tracking settings at the server group level.
Figure 8.3. The Tracking tab of the Server Group Properties dialog box.
Enabling or Disabling Document Tracking
Although it is useful, document tracking adds overhead to BizTalk Server processing and can dramatically increase disk space requirements. Therefore, BizTalk Server allows you to enable or disable the entire tracking system through the Enable Document Tracking check box. Clearing this box means that tracking is turned off for the selected server group. When the Enable Document Tracking check box is cleared, all other check boxes on the Tracking tab become disabled.
Note
When you toggle tracking on or off, it is a good idea to restart the servers using the BizTalk Server Administration tool. Otherwise, it is possible that new settings may fail to take effect immediately.
Log Incoming/Outgoing Interchange Options
The Tracking configuration tab contains the Log Incoming Interchange and Log Outgoing Interchange check boxes. Despite their labels, these check boxes do not translate directly into turning inbound and outbound interchange logging on and off. Instead, they control whether full copies of inbound and outbound documents are stored in the tracking database. Enabling both options causes the tracking system to store full copies of inbound and outbound documents. You may need to adjust these settings if, for example, you expect large inbound documents but much smaller outbound documents. In this case, it might make sense to only retain copies of the outbound documents. BizTalk Server will still generate a record of the inbound interchange, but it won't save a copy of the document.
Logging MIME-Encoded Messages
BizTalk Server contains built-in support for MIME-encoded messages. MIME is a W3C-approved standard for including multiple objects in a single message and to represent the body of a message in character sets other than US-ASCII. The MIME specification is published as RFC-1521.
You can configure BizTalk Server Document Tracking so that MIME-encoded documents are stored to the tracking database in their original format before decoding takes place. This behavior is controlled through the Log the Original MIME-Encoded Message check box.