Dynamic technologies
This chapter provides insight into software that employs advanced technologies that are used to create dynamic applications such as IBM Taskmaster Accounts Payable Capture and Taskmaster Flex.
This chapter includes the following sections:
For a technical walkthrough of the IBM Taskmaster Accounts Payable Capture product and to see how these technologies are applied in a real application, see Chapter 11, “Technical walkthrough Accounts Payable Capture” on page 339.
10.1 Introduction to dynamic technologies
Dynamic technologies are useful in solving some of the most difficult challenges in data capture. Most of these technologies were designed to capture data from unstructured and semistructured documents. However, many of the technologies can be added to practically any data capture application to make it more powerful or more manageable.
Dynamic technologies include the following methods and techniques:
PageID actions Actions to make page identification easier and more manageable.
FlexID Methods to manually identify pages in the batch.
DNA technology The Dynamic Natural Analysis (DNA) theory and techniques behind fingerprinting.
Sticky fingerprints The method for immediately providing location information to multiple examples of the same new fingerprint in a batch.
Managed recognition A self-monitoring method of calling recognition engines to automatically detect and correct failures in recognition.
CCO merging The technique of combining all of the CCO files of a document into a single CCO to allow document-level searching of data.
FPXML A mechanism for storing zones independent of the Setup Document Hierarchy (DCO).
Line item detection Detecting and displaying repeating data structures on a form.
Enhanced error messaging
An alternative to the standard error messaging employed by validation actions.
Data localization actions
Actions that can detect the localization settings of a machine and programmatically alter the rules execution and data accordingly.
Intellocate The Accounts Payable Capture learning technique.
Flex technology Techniques for creating and using data structures that can locate, display, and validate their own data.
This chapter explains each of these technologies and then takes you rule-by-rule through the current version of the IBM Taskmaster Accounts Payable Capture product to illustrate the use of these technologies in the context of a fully-featured invoicing solution.
10.2 PageID actions and techniques in dynamic applications
Most structured form-driven applications use fingerprinting for the primary page identification technique. However, dynamic applications typically have a document structure that is inherent in the batch composition. There are several reasons for this, but primarily it is because dynamic applications are more document centric in their data structure while structured forms are more page centric. For a given nonstructured document, at application design time, you are unsure whether a piece of data will be found on the first page of the document or on a trailing page.
Dynamic applications usually have three working page types: a required Main_Page and optional Trailing_Page and Attachment page types. Figure 10-1 on page 308 shows the document structure from IBM Taskmaster Accounts Payable Capture.
The Main_Page page type is usually the first page of the document. This page is where all of the data for the entire document resides, regardless of the page from which the data was extracted. Only Main_Page has fields assigned to it.
Other pages of the document are typed as Trailing_Page or Attachment page types. The difference between these additional pages is that pages designated as the Trailing_Page page type are recognized and searched for data as part of the document. Pages designated as an Attachment page type are included in the document, but they are not recognized nor searched by the locate actions.
Figure 10-1 Document structure with Main_Page, Trailing_Page, and Attachment page types
Figure 10-2 shows an example where a vendor sends a three-page invoice and attaches a copy of a purchase order, correspondence, or shipping document.
Figure 10-2 Three-page invoice with a correspondence attachment
The first of the three pages is designated as the Main_Page page type. The second and third pages are designated as the Trailing_Page page type. The page of the attached correspondence is designated as an Attachment page type as shown in Figure 10-3.
Figure 10-3 Page types of a sample invoice
To aid in page identification, IBM Taskmaster Accounts Payable Capture contains a library of page identification actions called PageID.rrx. Inside this library are two actions that are used by dynamic applications to separate the batch into documents.
10.2.1 Page identification by barcode separator
Batches that are scanned into dynamic applications typically need minimal batch preparation.
Most invoices that are processed are a single page. For IBM Taskmaster Accounts Payable Capture and Taskmaster Flex, these single page documents are always placed at the start of a batch. Because every document is a single page, no separation is necessary. When a mail room opens the envelopes that contain the documents, all single page documents go in one stack.
Multiple page documents go into a different stack, with a barcoded separator sheet placed on top of each document. Typically these barcoded sheets are printed on light-colored paper so that, after scanning, the sheets can be easily located and removed for reuse if desired.
You must place an additional separator sheet (Figure 10-4) inside of the document between the actual document pages and any attachment pages that are included.
Figure 10-4 Invoice with document and attachment separators in place
In the example with the three-page invoice and the purchase order and shipping document as attachments, an attachment separator page might be inserted after the third page of the invoice, but before the purchase order or shipping document. These pages are placed in a multipage stack, and a document separator sheet is placed on top of them.
When scanning, the only caveat is that, if a batch contains both single-page documents and multipage documents, the single page documents are the first ones scanned. After the actions encounter a document separator, all documents in the batch that follow must also use document separators.
Sample separator sheets are in the C:datacapsupportseppage directory. This same directory includes the .doc files that you can use to print personalized separator sheets. Be sure that you have a barcode font installed.
The action from the PageID library that is used for such batches is the PageIDbyBCSep action. This action uses two comma-separated parameters, the location of the settings.ini file and the default page type, to name pages in the batch until a document separator is found.
The settings.ini file is typically in the dco_<application name> folder of your project. This action has two pertinent sections as shown in Example 10-1.
Example 10-1 Settings.ini file content related to page identification
[PageID_Barcode-PageType]
'Enter the values of barcodes that appear on separator pages, and the page type to assign to those pages.
ENDDOC=Document_Separator
ATTACH=Attachment_Separator
 
[PageID_LastType-ThisType]
'Enter the values of every page type, and the page type to be assigned to unidentified page types that immediately follow them.
Document_Separator=Main_Page
Attachment_Separator=Attachment
Main_Page=Trailing_Page
Trailing_Page=Trailing_Page
Attachment=Attachment
The first section, [PageID_Barcode-PageType], lists the values in the barcode on the left side, and the right side is the page type to associate with those barcoded sheets.
The second section, [PageID_LastType-ThisType], documents how you want all nonbarcoded pages in the batch to be named.
When a batch is processed, each page is analyzed to see if it contains a barcode that you specified in the first section. Remember that nonbarcoded pages are given the name that you entered as the second parameter of the action. Therefore, a batch with single-page documents at the start of it is given a page type of Main_Page.
After a separator sheet is encountered, that page is named what is specified in the first section of the .ini file segment in Example 10-1. In this case, the page is named Document_Separator. Any nonbarcoded page immediately after that separator is named according to the listing in the second section of the .ini file in Example 10-1. For example, because it is the page immediately following a Document_Separator, it is given the Main_Page page type. If another nonbarcoded page follows the Main_Page page type, the second section of the .ini file specifies that it is given the Trailing Page page type. Also, a nonbarcoded page after that Trailing_Page page type is also given a Trailing_Page page type. This pattern continues until another barcoded separator (either Document or Attachment) is found and the “page-after” naming instructions can continue.
10.2.2 Electronic input of documents
Documents that are received into the system electronically do not need separator pages. These documents are typically received as multipage PDF or TIFF files, and you can employ a different technique to name the pages in a batch.
When a file is received electronically, a variable is placed at the page level of the data structure (runtime DCO) that signifies its origin. For multipage files received by a VScan task, this variable is called ScanSrcPath.
The PageIDbyVariableChange action takes advantage of this feature by using three parameters to name all of the documents in the batch. The first parameter is the variable to “watch” for change. The second parameter is the type you want to set the first page after the watched variable changes. The third parameter is the type you want to use for the remaining pages in the batch.
For example, consider an email message that contains three TIFF files. At the batch level, you place the PageIDbyVariableChange(ScanSrcPath,Main_Page,
Trailing_Page) action.
Each TIFF file is a multipaged invoice. When the images are brought in by actions, the TIFF files are burst into individual TIFF files, and each page has a ScanSrcPath variable that contains a reference back to the source image. If the mail contained three two-page TIFF files, you get a batch with six images, three of them with a ScanSrcPath variable referencing the first multipage TIFF and three images referencing the second multipage TIFF.
The PageIDbyVariableChange action processes the batch. Therefore, when the first source image is referenced by the ScanSrcPath variable, it names the page Main_Page. Then all pages until the ScanSrcPath variable changes are given the Trailing_Page page type.
10.3 FlexID
FlexID is aa way to manually identify pages, but some applications use it to perform image quality assurance. FlexID provides a fast, easy way to look at large thumbnails to identify images that might need to be re-scanned and to quickly and conveniently assign page types. This process can be done before or after a page identification is done to the batch. If the process is done before page identification, you normally name only a subset of the pages in the batch, and page identification can use its rules to complete naming of every page in the batch from there.
FlexID is included in a job in the Flex application and in the IBM Taskmaster Accounts Payable Capture product. When using the demo, choose the FlexID option in the job selection you get when you scan.
For example, consider the common way of naming pages within a document in a dynamic application. Normally the first page of the document is named Main_Page, and the other pages in the document are named Trailing_Page. With FlexID, you can identify the first pages of new documents by naming them Main_Page, and the page identification rules that follow name all of the trailing pages for you.
FlexID also provides a way for a user to clone certain images, such as a fax cover page that is in front of three documents that are faxed together. You can copy the header sheet and place it in front of each document with FlexID.
FlexID is available in a thick client or thin client. Figure 10-5 shows the thick-client version.
Figure 10-5 Thick client FlexID
10.4 DNA technology
DNA describes the method that is used for image identification based on fingerprint technology. Imagine documents that you are familiar with lying on a table at such a distance that you can identify the document by its layout, without reading any text from the document. For example, you might see your telephone bill or a credit card statement that you are looking at from across the room. Without being close enough to read the text from the document, you can identify it because you are familiar with the layout and because the layout is distinctly associated with that particular document.
By using such a natural analysis, the fingerprint technology processes select images from a batch of images. It also provides three vital pieces of information that are useful in processing each image.
First, the DNA technology notifies you if there was a suitable match between the current image and all of the images currently stored in the fingerprint library. The matching process is done by properly setting up the area of the image to be compared against the library and by defining a confidence threshold that must be met to be considered a match. The “dynamic” part of the technology name indicates the capability of the fingerprint mechanism to add new fingerprints to the library during normal processing if such a match is not found. If a match is not found, the fingerprint automatically goes through a process called Intellocate. For information about Intellocate, see 10.12, “Intellocate” on page 335.
Second, if a match is found, the DNA technology returns an identifier, called a TemplateID, to the application. This identifier makes it possible to locate previously identified zones on the image where data was found on the image that is in the fingerprint library.
Finally, the DNA technology also returns an offset to apply to those zones to allow for horizontal and vertical movement of the zones to match precisely to the runtime image. When two images go through a mechanical process, such as faxing or scanning, some shifting of the data on the image is common. This offset indicates how far you need to move the reference zones to align properly on the image that they are now associated with.
10.4.1 Fingerprint matching in dynamic applications
When matching fingerprints in dynamic applications, such as IBM Taskmaster Accounts Payable Capture and Flex, the standard fingerprint actions are used similar to way in which they are used in static applications (structured forms). However the usage is different in three key areas.
First, in learning applications, the first parameter to the FindFingerprint action is set to TRUE. With this setting, the fingerprint action creates a fingerprint and adds it to the existing fingerprint library if a suitable match is not found. More stringent matching parameters are associated with this parameter than might normally be applied to the fingerprint facility than if it was used for another purpose, such as, PageID.
In a typical dynamic application, a matching threshold of 80% or higher is common. A balance must be achieved with a threshold that is high enough that documents are not misidentified. However, the threshold must be low enough so that extra fingerprints for the same document are not created unnecessarily. Also, the zoning the search area is normally restricted to the upper part of the document with a consistent format to allow for variability in the lower portions of the image for line items on invoices, for example.
The second key difference is that the fingerprinting only occurs on key pages of the document. As such, you must override the standard functionality of the fingerprint facility of setting the page type of the matched fingerprint.
With dynamic applications, typically you do a fingerprint match on the first page of a document, to indicate the document type that you are dealing with. Dynamic applications typically name their pages generically, such as Main_Page, Trailing_Page, and Attachment. Typically, only the main page is fingerprinted for identification purposes. The entire data structure for all of the data found on the document is only on the Main_Page of the document, although data can be captured from Trailing_Pages.
Because the PageID of a dynamic application is done before fingerprinting, you must ensure that the page retains its original page type. For example, the rule in Figure 10-6 might run on a page object named Main_Page. The page-level rule resets the page type that is returned from the FindFingerprint action.
Figure 10-6 A page-level rule for FindFingerprint that resets the page type
The FindFingerprint action returns true or false depending on whether it was able to make a match. In either case, you do not want the action to overwrite the page type that you have already assigned in the PageID rule set.
After a new fingerprint is processed, you must classify it and save the information, such as zones, that were captured during the processing. For more information about this topic, see 10.12, “Intellocate” on page 335.
10.4.2 Using the TemplateID
The TemplateID and its associated classification are used to provide valuable information during document processing. The classification is stored in the fingerprint database and is viewable by using the Zones tab in Datacap Studio.
In IBM Taskmaster Accounts Payable Capture, the TemplateID is associated with the vendor that supplies the invoice. In Taskmaster Flex, the TempateID is associated with the Document Class.
The TemplateID also links to systems that indicate you where zonal data is on the form if the fingerprint was previously processed by the system. In cases where a fingerprint match is not found and a new fingerprint is created, no zones are available until it goes through the Intellocate process. For more information about this process, see 10.12, “Intellocate” on page 335.
10.4.3 The offset
The offset returned by the fingerprint mechanism is used to modify the zonal information that is associated with the original fingerprint before it is applied to the runtime image. This offset is derived during the fingerprint matching process to achieve the best possible match of the words and lines on the source fingerprint. Therefore, the offset must be applied to the runtime image before data can be accurately extracted from it.
10.5 Sticky fingerprints
The technology of sticky fingerprints allows for location information. This information is gleaned from processing a new fingerprint at verify time to be applied to other images in the batch that match the same new fingerprint.
For example, the Taskmaster Accounts Payable Capture application contains two different images from the same vendor. If the fingerprint is deleted, the operator encounters the first image and does the data entry on it by using the Click’n’Key capability, supplying the location information for that fingerprint.
When the second image from the same vendor is encountered, a Sticky Available button is displayed. When the operator clicks that button, the location information from the previous invoice is applied immediately to the current invoice. The CCO is searched for the header and line-item data, and the document is processed without further operator interaction.
Figure 10-7 shows the first image from the vendor encountered in the batch. Some fields were prepopulated by locate rules.
Figure 10-7 The first encounter of this invoice
Figure 10-8 shows the image after the operator clicks the image to locate the missing fields and the upper line item and then clicks the FindDetails button.
Figure 10-8 The same invoice after the operator uses Click’n’Key and FindDetails
Figure 10-9 shows the second invoice from the same vendor in the batch. Notice that this invoice is different, including different data and a different number of line items. You must click the Sticky Available button in the upper part of the pane, so that the zonal information from the first invoice can be immediately applied to this invoice.
Figure 10-9 The second invoice from the new vendor in the batch
Figure 10-10 shows this invoice after clicking the Sticky Available button. The location information from the first invoice is applied to this invoice, and all of the fields and line items are populated with data in one click.
Figure 10-10 The second invoice after the Sticky Fingerprint technology is applied
10.6 Managed recognition
Recognition engines are expected to attempt to recognize any image that is sent to them. Taskmaster employs many different recognition engines, some of which are more suited for some types of images than others. Over the years, experience has shown that monitoring the recognition engines closely is prudent. Therefore, Taskmaster employs a technology called managed recognition.
Figure 10-11 illustrates the managed recognition concept.
Figure 10-11 Conceptual drawing of managed recognition
With managed recognition, the recognition engine is placed outside of the current thread and monitored by Taskmaster. When recognition occurs, if the recognition engine is successful in recognizing the image, it returns the data directly to the main application thread for further processing.
If the recognition engine is unable to recognize the image, and exceeds its timeout value, it returns to the main application for another try with a longer timeout.
If the recognition engine hangs and is unable to monitor its time, the Managed Wrapper detects this problem. Then it shuts down the engine immediately, reinitializes it, and tries a second time with a longer timeout.
These timeouts are configurable and might need to be changed based on the speed of the machine running recognition, or the complexity of the images being recognized.
If both attempts at recognizing an image fail, the document is automatically marked for deletion from the current batch. Then a notice is set to administrative personnel along with images for further review.
10.7 CCO Merging
As was mentioned previously, dynamic applications place their data structure only on the Main_Page of the document. However, when searching for data, they search both Main_Page and Trailing_Page page types.
To accomplish this task, you must use a technology called CCO Merging. When you use CCO Merging, the CCO file for the Main_Page is combined with the CCO files from all of the Trailing_Pages. Also, a multipage CCO (MCCO) is created to replace the single-page CCO that is associated with the Main_Page. If the single-page CCO from the Main_Page must be used later in the process, your application must take steps to back it up and restore it.
The merging of CCO files is done by using an action called MergeCCOs_ByType. This action is typically done at the document level after a document structure is created and before the document is searched for data. IBM Taskmaster Accounts Payable Capture and Taskmaster Flex run this action in the Locate rule set.
MergeCCOs_ByType(Main_Page,Trailing_Page) merges all of the Trailing_Page CCO files onto the Main_Page CCO file and replaces the Main_Page CCO file with the result. After this task is done, locate rules search the entire CCO file for the data that you require.
When data is found on a Trailing_Page, the coordinates are in relation to the Main_Page of the document. For example, consider two CCOs for pages that are 3300 pixels in length. The resultant MCCO is 6600 pixels in length. Also, something that is 500 pixels down on the second page shows a vertical coordinate of 3800 pixels (3300 pixels from page 1 and an additional 500 pixels down on page 2).
These coordinates cannot be displayed correctly in a verify panel that shows a single TIFF at a time. Therefore, these raw coordinates must be adjusted back to page coordinates before verification.
This step is done by the MCCOPositionAdjust() action. This action runs after all of the data has been programmatically extracted by Locate rules.
After MCCOPositionAdjust runs, an IMAGEFILE variable is displayed in the field that contains a reference to the actual page on which the data was found. All of the positions are adjusted related to that page. In the example used with the two pages, data found on the second page might have an IMAGEFILE variable referencing the actual TIFF from which the data was extracted. The 3800 vertical coordinate might be adjusted back to 500 pixels.
10.8 FPXML
Taskmaster has two places where it is possible to store the zonal information for a field. By default, applications store the field position information in the Setup DCO. When you have a field that occurs on multiple documents, the Setup DCO stores the position information in tags that reference back to the Template ID, such as in Example 10-2.
Example 10-2 Snippet of the Setup DCO
<V n="Pos1003">2134,1179,2249,1214</V>
<V n="Pos1010">2226,1098,2381,1139</V>
<V n="Pos1016">2177,1571,2315,1615</V>
<V n="Pos1002">2268,1612,2392,1646</V>
<V n="Pos1005">2178,1103,2315,1145</V>
In the snippet of the Setup DCO in Example 10-2, the field in question is on five different documents. The TemplateID of each Main_Page is encoded in the tag itself, such as Pos1003. Also, the comma-separated list contains the coordinates of the field in the sequence X1,Y1,X2,Y2.
As you can imagine, if you have an application such as IBM Taskmaster Accounts Payable Capture, tens of thousands of Template/Position lines might exist for every field that is to be captured zonally. This number of lines can result in excess time in rewriting the Setup DCO each time a new fingerprint is automatically added to the system.
An additional consideration is the portability of the Setup DCO from one system to another. References to the rules are also stored in the Setup DCO. Consider a situation where you change the rules on a test or development system and then move the Setup DCO to a production system. In this case, you lose the field positions of any field that was automatically added to the production system because the file was copied to a test or development system.
For this reason, use the FPXML system to store zonal information when developing dynamic applications. FPXML stores the positions for each fingerprint in a separate XML file. The file is stored in the fingerprint directory alongside of the fingerprint TIFF and the fingerprinted CCO.
When it is time to read the zones for a given image, the ReadZonesFPX action from the FPXML action library is used. This action is the FPXML equivalent to the ReadZones action from the Zones action library, but it reads the zones from the FPXML file instead of the Setup DCO.
When writing the zones with features such as Intellocate, you use a different action other than the ones that write to the Setup DCO. The WriteZonesFPX action is a direct replacement for iloc_SetZones and iloc_SetDetailZones, again with the exception that it writes zones to an FPXML file instead of the Setup DCO.
To set up Datacap Studio to use FPXML for your application, FPXML requires an entry in the .app file of the application such as the one shown in Example 10-3.
Example 10-3 Entry in the .app file
<k name="dco_FlexID">
<k name="UseFPXML" v="true"/>
When this entry is successfully completed, new fingerprints added to the system have a file in the fingerprint directory with the naming convention <TemplateID>.xml. This file includes a tag for each field in your DCO, followed by the position at which the field data can be located for that fingerprint.
A facility is built into the Fingerprint Maintenance Tool to convert positions from FPXML to the Setup DCO and vice versa. The Fingerprint Maintenance Tool also uses FPXML to export fingerprints from a system to be imported into another system.
10.9 Line item detection
Line item detection is used to capture areas of repeating data in a document, such as the line items often found on invoices or purchases on a credit card statement.
Figure 10-12 shows sample line items from an invoice. This invoice has three line items. Taskmaster can capture an unlimited number of line items in the entire document, even multipaged documents, by using the CCO merging technology.
Figure 10-12 Sample line items from an invoice
To understand how Taskmaster deals with line items, you must be aware of how repeating structures are stored, zoned, captured, and filtered.
10.9.1 Storing repeating structures
Repeating structures are stored in a detail field structure. A detail field is typically a field that is placed directly as a child to a page in the DCO, but contains a structure of subfields beneath it to hold the repeating data.
Figure 10-13 shows the detail section from IBM Taskmaster Accounts Payable Capture. In this application, the coordinates of each line of the invoice are stored as a Lineitem type field. An unlimited number of these fields can exist when the repeating structure is filled with an invoice that is being processed. Each line item has six subfields of its own that contain the actual data we want to capture off the line.
Figure 10-13 Detail structure where all of the line item data is stored
The subfield that is immediately under (is a child of) the Details field contains a zone to the entire line that you want to capture. This line is not always a physical line on the page. Some repeating structures use multiple physical lines on the image to describe a single-line item.
Figure 10-14 shows an invoice with multiple physical lines for each line item you will capture. The ItemID is on the second physical line of each line item, but is stored in the same document structure as invoices with single physical line items.
Figure 10-14 Multiple physical lines documenting a single line item
10.9.2 Zoning the detail structure
Each field in the structure needs a zone. With the use of Intellocate, this zoning is initially captured at run time by the verifier clicking the subfield components with the first instance of the document that is processed through the system. From these subfield components, the line item position itself is calculated. Then the detail field is calculated from the top of the line item to the bottom of the CCO.
Under normal conditions with a dynamic application, you only need this automatic zoning of the detail structure. However, for some applications that are form driven, you might choose to zone detail fields manually by using Datacap Studio.
 
Important: Use care when zoning detail structures on the DStudio Zones. Thoroughly test your zones after you complete this task.
To zone the detail structure on the DStudio Zones tab, complete these steps:
1. Zone each subfield by drawing a box around each item as shown in Figure 10-15.
Figure 10-15 Manually zoning the line item subfields
2. Zone the line item field by drawing a box around it, but be sure that it does not overlap any of the subfields. This step can be difficult because you cannot see the subfield zones while you are doing the zoning. Keep the zone tight around the subfields, but allow for more liberal zoning around the entire line item as shown in Figure 10-16.
Figure 10-16 Manually zoning the Lineitem field
3. Zone the detail area. Again, the line item must be inside of the details zone that you set. Therefore, be liberal in your zoning to allow space around where you zoned the line item as shown in Figure 10-17.
Figure 10-17 Manually zoning the Detail field
Notice that you only need to zone the top line item and the subfields, but the detail field must extend over all line items.
If you are working with a project that has a fixed number of line items, such as on a form, the bottom of the details section can be set exactly. However, if you are zoning something that has a variable number of line items, the bottom of your detail zone must be reset programmatically to the bottom of the CCO. This resetting is done at run time by using the ZoneBottom_ImageBottom action. For more information, see 10.9.3, “Capturing the detail structure” on page 329.
10.9.3 Capturing the detail structure
Capturing the detail structure is done in three steps. This task is typically associated with the Locate rule set, where the header data in a dynamic application is found.
At the detail field level, you must first dynamically set the boundaries of the detail zone. If you are working with a form that has a fixed number of detail lines, the zone that you set in Datacap Studio does not need to be programmatically altered. However, if you are working with a document that has an unknown number of line items that can span many pages, adjust your detail zone so that all of the potential line items are contained in it.
This task is done by using the following rules from the Zone action library:
•ZoneTop_ImageTop()
•ZoneBottom_ImageBottom()
•ZoneRight_ImageRight()
•ZoneLeft_ImageLeft()
These four actions extend your detail zone to the entire document. Do not worry if you get additional text that is not a line item in the zone. It is filtered out later.
The action that creates all of the line item child fields is the ScanDetails() action. This action creates a Lineitem for every physical line in the detail zone. If 100 physical lines are in the zone, it creates 100 Lineitem children fields, with a unique ID for each one (LineitemXX), where XX is a sequential number starting with Lineitem1. The actual name represents what you named your child field in the Setup DCO. Each line that is created has its upper-left corner set at the X coordinate that is assigned when zoning the line item. The Y coordinate is the top of the actual line detected by the coordinates of the line in the CCO. The lower-right corner of the line zone is set from this point with the width and height of that the Lineitem zone.
If your Lineitem field is defined to capture more than one physical line, the result is multiple overlapping zones, which are not a problem. Only the zones that have complete line items are kept after the filtering.
After this process is done, you usually have many line-item fields. Although each of these fields has a different ID, they all have the same type. Therefore, a rule defined in Datacap Studio to run on that object type is run for each line item.
The ScanLineItem() action used on each line item. This action creates the subfields for each line item, in relation to the position of each line item. The relative position of each field in the line item is determined by the position in which the field is defined in the initial line-item zone that is defined by Intellocate or by manual zoning in Datacap Studio.
When this task is done, the subfields of each line item must be positioned where you expect line item data to be on each line item. To pull the data from the CCO into the fields, you usually use the PopulateZNLineItemField() action from the Zones action library. In rare circumstances, you can use other locate actions, such as the FindRegExInZone action from the Locate action library. However, normally capturing all of the data in the zone is appropriate.
Again, you can expect many more lines in the processing than you have actual line items in the detail area. All of the header and footer information from all of the pages in the CCO are now zoned and captured as line items. However, only a subset of those zones is retained as line items through a process called filtering.
10.9.4 Filtering line items
Filtering line items is a two-stage process that requires two rule sets. The two rule sets act in concert with each other and are typically called Clean and Filter.
The goal of the filtering process is to remove all of the line items that you do not want from the captured set of line items. To accomplish this task, you must identify certain characteristics about some of the fields of the data set.
At the time of cleaning and filtering the lines, remember that all physical lines are represented in the data structure. Also keep in mind that, in the example in this book, we have zoned each line and pulled any values that were in the zones in which we are looking for particular data.
Figure 10-18 shows the zoned data. Recall that we are zoning every physical line item in the document as though it was a line item. The strategy is to look at key fields where we know that we want a specific data type.
Figure 10-18 Zoned lines
Table 10-1 shows the data that is captured in each of these fields. (Shown in a grid for convenience, the DCO is stored in an XML file.)
Table 10-1 Data pulled from the zoned lines
Qty
Description
ItemID
Price
LineTotal
TX
Dallas,TX 73443
 
 
2 of 2
 
Description
Number
Price
Extended
3
Deluxe First Aid Kit
3332223
128.89
389.67
45
100 Pack Daisy Cups
6423216
7.23
325.35
The cleaning process analyzes the data in each field and then blanks it out if it is not the correct data type. In Table 10-2, the Qty field was analyzed and left blank if it did not contain a number. Also the Price and LineTotal fields were left blank if they did not contain a currency value.
Table 10-2 Data left blank during the cleaning process
Qty
Description
ItemID
Price
LineTotal
 
Dallas,TX 73443
 
 
 
 
Description
Number
 
 
3
Deluxe First Aid Kit
3332223
128.89
389.67
45
100 Pack Daisy Cups
6423216
7.23
325.35
Next, a filter is applied to remove all line items that do not have values in the specified number of fields. In this example, you create filter such as the one shown in Example 10-4.
Example 10-4 Filter to remove all line items that do not have values
CheckSubFields(('Qty' AND 'LineTotal') OR ('Price' AND 'LineTotal') OR ('Qty' AND 'Price')
In this example, we are looking for valid data in only two of the three fields that we analyzed for data types. This way we have some flexibility if we run across an invoice that does not list the quantities or the price, or if the recognition on the field came back incorrectly for some reason.
At the end of this process, you are left with only the last two lines in the example, the nonline items have been filtered out (Table 10-3).
Table 10-3 Lines filtered
Qty
Description
ItemID
Price
LineTotal
3
Deluxe First Aid Kit
3332223
128.89
389.67
45
100 Pack Daisy Cups
6423216
7.23
325.35
10.10 Enhanced error messaging
When a validation action fails, it sets an error message at the field level that is displayed in the status bar of a verification panel. When the action fails, it sets the error message according to the specific action that fails. The result is that, in certain situations, the operator does not have a complete understanding of everything that is wrong with a particular field. Consider the example in Figure 10-19.
Figure 10-19 A validation rule that sets the field-level error messages
This type of rule is common as a validation. The first function checks to see if the data in the field is at least 2 characters long and at least 60% numeric. If this function fails, the second function in the rule indicates that it can also pass validation if the field is blank.
Consider what happens if someone types a value of A1 in the field. It passes the action concerning the minimum length, but fails the action that indicates that it must be at least 60% numeric. The field-level error message is changed to “The data must be at least 60% numeric.”
Then, the rule runs the second function that indicates it can also pass validation if it has a maximum length of 0. This function also fails. Therefore, the value of the field-level error message is changed to “The field must have a maximum length of 0 characters.”
Obviously, the error message can be misleading to a data entry operator.
The solution to this problem is to use the enhanced error message technology in IBM Taskmaster Accounts Payable Capture. With this enhancement, you can control the error message that is ultimately displayed to the user in a dialog box when validation is attempted during verification. For an example, see the AddToErrorMsg action in Figure 10-20.
Figure 10-20 AddtoErrorMsg action
This error message is saved at the page level in a variable that is appended to all other error messages from the document. This way the operator has a complete understanding of all errors that are found during validation for the entire document as shown in Figure 10-21.
Figure 10-21 A page-level enhanced error message
10.11 Data localization actions
Taskmaster has built-in language support for various languages. However, some actions are needed to handle localized dates and numeric formats. This list grows because opportunities are available in various locales and new formats must be handled. The strategy is to detect the localization options of the computer running the application. The strategy includes adjusting the number and date formats from data that is extracted from the document so that the machine on which it is running can process the data correctly.
For example, many European countries use a comma as a decimal separator, but the US and UK use a period as a decimal separator. This practice is normally not a problem unless a machine is trying to process documents generated from locales other than its own. When this case occurs, data localization actions are necessary.
The data localization actions can be placed in two general categories:
•Actions that affect rules execution
•Actions that affect the captured data
10.11.1 Actions that affect rules execution
Many Taskmaster actions require parameters, and sometimes these actions require parameters that must be localized. An example is the SetSearchArea action when you define how much of an image to compare when fingerprint matching.
To accomplish this task, the IsLocalDecimalSeparator() action is used. The parameter that is supplied is tested against the decimal separator that is used by the localization settings of the machine. If you supply a comma, for example, the action returns true if the machine is set to use commas as the decimal separator in the localization settings. Figure 10-22 shows how the action is used.
Figure 10-22 Rule using the IsLocalDecimalSeparator() action
When the localization settings of the machine are set to use a comma, the parameters supplied by the SetProblemValue() and SetFingerprintSearchArea() actions are supplied with commas. If the IsLocalDecimalSeparator(“,”) action returns false, the parameters are supplied with a period as the decimal separator.
10.11.2 Actions that affect the captured data
The data on the form can be a challenging to localize. A date value of 10/20/2011 can always be assumed to be 20 October 2011. However, a date value of 02/04/2011 can be 4 February 2011 or 2 April 2011 depending on the locale of the origin of the document.
Another example is in a field that contains a quantity. Some quantities are fractional. Therefore, it is difficult to know if 6.301 is referring to six thousand, three hundred and one or if it is referring to six and three hundred and one thousandths.
This type of ambiguity must be handled case-by-case, often by looking up the locale of the originator of the document and handling the data based on that locale.
For values such as currency, where you know that there is a decimal separator, the CheckAndFixLocalDecimal() action can be an effective way to convert the data programmatically. This action looks at the rightmost separator, converts it to match the local machine, and removes the thousands separators if present. The currency values then contain only a decimal separator. At export time, those separators can be changed again to match the specified decimal separator of the system to which you are exporting.
10.12 Intellocate
Intellocate is the technology that allows Taskmaster applications to learn. With DNA technology, documents that have never before been processed by the system are added to the fingerprint library. Locate rules are used to automatically find some of the data from these documents by using keyword searches or regular expressions. However, what cannot be automatically found can be identified and captured quickly and easily by a verify operator by using the Click’n’Key capability.
After this task is done, Intellocate saves the zones for the fingerprint. Then the next time a similar document is encountered, the fingerprint is matched, and all of the data is read by the zones.
Before the invention of Intellocate, IT personnel were often asked to set up each document type before a system processed them. Because these documents can come in at any time from various sources without warning, this task did not prove to be an attractive method of dealing with the documents. Others systems only used locate rules to locate data from a form, with no setup, but with limited results. If an unusual keyword was encountered, the document might never be able to be processed correctly.
Taskmaster, with Intellocate, developed a hybrid of the two techniques to provide fast, accurate learning of new documents that are introduced into the application. The added benefit is that this task can be accomplished easily by data entry operators. These operators best know the forms that they are dealing with as opposed to the typically more expensive IT staff.
As explained in 10.4, “DNA technology” on page 314, fingerprints are sometimes added to the system programmatically, but without the zonal information required to extract data zonally. The document goes through the verification process, and an operator clicks the image to populate any fields that were not located programmatically. In this case, the zones of the data are saved so that you can later write them to the Setup DCO or FPXML, depending on the system you selected. You can also classify the fingerprint by the criteria that you choose.
In IBM Taskmaster Accounts Payable Capture, the fingerprint is classified by the vendor who originates the documents. In Taskmaster Flex, the fingerprint is classified by the document type. This process is called Intellocate. It is done in the export process to ensure that the data required to classify the fingerprint is completed. Also all zones are captured in the runtime DCO, so that those zones can be saved to the appropriate location.
To classify the document, use the SetFingerprint(SmartParam) action. The Smart Parameter can refer to a field value or a variable from the page or a field. Alternatively, it can be a value that you specify as text. For example, the SetFingerprint(@P/Vendor) action classifies the current fingerprint in a classification with the name of the value that is currently in the Vendor field, which is a child of the page. If the classification value does not exist, it is created for you. Otherwise, the fingerprint is added to an existing fingerprint class.
To save the zones for the current fingerprint, use the WriteZonesFPX() action from the FPXML action library, or use the iloc_SaveZones() and iloc_SaveDetails() actions from the Intellocate action library.
10.13 Flex technology
In Flex technology, fields are given extra attributes as variables at run time so that they can be located and validated by simple generic rules that are applied to all flex-type fields. With Flex technology, you can create flex-type fields at run time, so that you do not have to explicitly set up all document types in the Setup DCO before processing.
With Taskmaster Flex, for example, the document classes and the fields that you want to capture for each document are stored in a database. They are configured by the Flex Configurator utility. In other projects, Flex technology has been used to read document definition databases that are used to print the documents, capture them using Flex technology, and store them in an image repository with a singe source of the document definition.
Taskmaster Flex reads the document database at run time and applies attributes as variables in the runtime DCO. A typical field in the DCO might look similar to the DCO field shown in Figure 10-23.
Figure 10-23 A DCO field with flex attributes
These additional variables contain all of the information needed for special flex actions to locate and validate the field:
LocateFlexField() Uses the variable attributes on a field to programmatically locate the data on a document.
ValidateFlexField() Uses the variable attributes on a field to programmatically validate the data in a field.
10.14 Conclusion
This chapter described the dynamic technologies that can be useful in solving some of the most difficult challenges in data capture.
Chapter 11, “Technical walkthrough Accounts Payable Capture” on page 339, guides you through the IBM Taskmaster Accounts Payable Capture product and shows you how to use these technologies in the context of a fully featured invoicing solution.
..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset