Production imaging functionality
This chapter highlights the functionality provided in the IBM Production Imaging Edition offering. It focuses on the capabilities, user interfaces, and internal processes of IBM Datacap Taskmaster Capture (Taskmaster).
This chapter focuses on Taskmaster functionality, from a high-level gradually toward a more detailed level. Concepts and capabilities are introduced as part of a typical Taskmaster process, from digitization, to data capture, to committing contents and data to the back-end systems.
This chapter includes the following sections:
3.1 Functionality highlights of Taskmaster
Taskmaster is document capture software. It is bundled into IBM Production Imaging Edition to provide the advanced document capture capability, which is an important requirement in managing the entire document imaging life cycle. The purpose of Taskmaster is to digitize paper documents, extract useful information from them, and feed them into other business processes downstream. Its strength is its ability to perform these tasks with a high degree of automation and accuracy.
Taskmaster provides the following main functions:
•Scans paper documents, straight from a scanner
•Imports electronic documents or existing images from a file system or fax server
•Cleans up images, using such functions as deskewing, removing lines, smears, and borders, to help with the recognition process later
•Classifies and separates types of document
This way it can determine where data needs to be extracted from the paper and which data needs to be extracted for indexing or any other use in the future.
•Extracts data by using recognition technologies
The following recognition technologies can be used:
 – Optical Character Recognition (OCR) for machine-printed characters
 – Intelligent Character Recognition (ICR) for handwriting, typically detached block letters
 – Optical Mark Recognition (OMR) for catching check boxes and other expected marks, such as bubbles in surveys or a signature on a form
Taskmaster can also extract data from one- and two-dimensional barcode types. One-dimensional (1D) barcodes contain simple product or item information such as the barcodes around the corner of product boxes purchased in a store. Two-dimensional (2D) barcodes usually contain more information such as address and shipping information.
•Checks the accuracy of extracted information and corrects errors
Taskmaster can format and validate field values against business rules. It can also automatically look up information in a database from the partial data it has recognized. Taskmaster can trigger verification and validation by a human operator when confidence in the data accuracy is below a set level.
•Learns automatically from the experience of human operators and the processing of documents to improve accuracy over time
•Exports image documents and extracted data to FileNet Content Manager and to a database or a business application
•Organizes the flow of tasks in the capture process from scan to export and exceptions in a workflow
•Controls access to the system and tasks by using functional security
•Monitors progress of capture operations and fixes problems in real time
•Reports on capture operations and provides statistics on how well the system is performing
•Runs unattended tasks in the background by using the Rulerunner engine
•Supports flexible deployment scenarios
Such scenarios can include a central mailroom type of operations and distributed imaging over the web, or regional offices by combining with the distributed deployment capabilities of FileNet Content Manager.
3.2 Taskmaster process
As mentioned in Chapter 1, “Production imaging overview” on page 3, Taskmaster is typically involved in the front end of the production imaging process, or the “precommittal process.” It digitizes and extracts the data that is necessary to classify and index the documents according to the model that has been implemented in the FileNet Content Manager repository and workflow.
To understand the Taskmaster capabilities and how they are used, you must first look at the Taskmaster process and some of its main constructs. At a high level, the Taskmaster process begins with the ingestion of paper documents and finishes with the committal (release) of images and their metadata to the back-end repository and line of business (LOB) databases.
Figure 3-1 shows a typical Taskmaster process.
Figure 3-1 Taskmaster process
The main tasks of the Taskmaster process include the Scan task, background processing, the Verify task, and the Export task. The following sections begin with a brief explanation about batch preparation before the Scan task and then provide details about each task.
3.2.1 Batch preparation for scanning
Typically paper documents are prepared manually for scanning. Then they are assembled into batches based on how they are processed later. For example, loan application forms and their supporting documents might be prepared so that they flow in a predefined order that is repeated within a batch.
In another example, when a batch has high variability and little structure, it is useful to insert document separator sheets with a barcode to mark document boundaries. Separator sheets can contain check boxes or other printed data to facilitate the classification process. You can use separator sheets to split a large batch in smaller ones, and you can use a batch cover sheet to automate the indexing in common to all documents in a batch.
3.2.2 The Scan task
When the manual preparation work is complete, the first task in a Taskmaster process is scanning. The Scan task creates a batch. The batch is immediately recorded in the Taskmaster Engine database, and a workflow job that controls the processing tasks and sequence is initiated. The scanned images are stored in the Taskmaster file server in individual TIFF files along with an XML description of the contents of the batch.
3.2.3 Background processing task
After populating the batch with images through scanning or importing, Taskmaster processes it automatically in the background. Now rule processing comes into play.
Taskmaster processes the batch based on the task definitions or task profiles. A task profile specifies sets of rules, or rule sets, that Taskmaster must apply in a predefined order to the various components of the batch. These components include the batch itself, documents, pages, and fields or zones. The relationship between these various components is defined in the document hierarchy when configuring the Taskmaster application.
As Taskmaster processes the batch, it writes the processing results to the batch directory. It also creates a batch descriptor file that shows the name of the task, which is used to pass on information to the next task.
The background processing task can execute many different sets of rules depending on the specifics of the application. Typically it includes the following rule sets:
•Cleaning and enhancing the images to help the recognition process
•Identifying the class of document by using fingerprints, barcodes, pattern matching, keyword searching, or a combination of these techniques
•Recognizing the data from the zones and fields defined in the document hierarchy
•Validating accuracy, formatting, consistency, and completeness of the recognized data and looking up missing pieces of information from external systems
At the end of the background processing, a batch description file is created that describes the batch, its documents, and the pages. The file also links to the XML files that hold the metadata that is extracted from each image.
3.2.4 The Verify task
The background process flags images that have a low recognition confidence level or validation errors and that need to be verified by an operator. Taskmaster launches the graphical user interface (GUI) that has been configured for the Verify task. It points the operator to the problem fields with the matching image snippet. The operator manually updates the data and submits the changes for validation with the rules that have been associated with the fields. The process is repeated for each problem field until the data is successfully validated or possibly overridden by the operator.
At the end of the Verify task, again, a batch description file is created with the files that contain the verified data.
3.2.5 The Export task
After the images are processed through the background and verification tasks, Taskmaster exports the image files and extracted data to the back-end systems. The Export task includes all the processing that is needed to prepare the documents (possibly converting them to another format), establishes the connection with the back-end systems, and indexes and uploads all the documents in the batch.
Again, the processing is defined as sets of rules, and they are executed on the batch, document, page, and field levels in the background.
3.3 Taskmaster GUI
Taskmaster provides a flexible user interface, especially for highly customized data entry forms, with innovative capabilities to enhance the productivity and experience of an operator.
3.3.1 Productive GUI
You can create GUIs with buttons, hot keys, and labels. You can also have the cursor position itself in the next problem fields. This way the operator does not have to look for where to go next. Figure 3-2 shows the user interface with the image snippets and color-coded fields.
Figure 3-2 Taskmaster image snippets and color-coded GUI
3.3.2 Image snippets
Taskmaster provides the capability to position image snippets next to the matching recognized data fields. This way, operators can easily compare recognition results to the zones from where the data originated.
3.3.3 Color-coded recognition confidence
You can configure color-coded backgrounds and character ink to indicate the confidence levels on recognized data and quickly show validation errors. For example, as shown in Figure 3-2, you can use the following backgrounds:
•Blue background for high-confidence recognition results and no data validation errors
•Yellow background for low-confidence recognition results with red ink for low-confidence characters in the field
•Red background for data validation errors
3.3.4 Click’n’Key capability
With the Click’n’Key capability, you can pull data into a field with a single click. You position the cursor in the destination field and then click or draw a box around the piece of information in the image that you want to grab (Figure 3-3).
Figure 3-3 Click’n’Key capability in Taskmaster
You can also combine this feature with the ability in Taskmaster to learn from the manual input of an operator and record the location where the data was found in the fingerprint that is associated with the type of document. This new information learned by Taskmaster is used to fine-tune and automate the recognition process the next time it processes a similar document. For more information about fingerprints, see 3.6.7, “Fingerprinting” on page 87.
3.4 Taskmaster clients
Taskmaster can be used with a thick client, for centralized or dedicated capture work, or with a thin client, for remote or more occasional use. This section provides an overview of the functionality of these clients and of their usage to provide a better understanding of the dynamics between the Taskmaster process and its clients.
3.4.1 Taskmaster Client (thick client)
Taskmaster Client is a thick client. It is used by operators for launching tasks (such as the Scan and Verify tasks) and by supervisors for monitoring operations. It is also used for administering users, for workstations, for security, and to configure the jobs in the workflow of an application.
From the Taskmaster Job Monitor window (Figure 3-4), a supervisor can monitor the status of each batch in the system and obtain information. For example, a supervisor can gather details about the currently executing task, the elapsed time for the task, who is running the task, on which workstation it is running, and the number of documents and pages in it. The supervisor can also take control of specific problem batches, look up their history, change their status and priority, and assign them a different task, workstation, or user to resolve the issue.
Figure 3-4 Operations and Job Monitor windows in the Taskmaster thick client
When configuring an application, the administrator uses the Administrator window to assemble the set of tasks that were predefined in Datacap Studio (in the task profiles of the workflow of the application). Tasks can run in the background, such as the tasks that are run by Rulerunner. Alternatively, tasks can be user-attended, such as the Verify task. Each user-attended task has a user interface component associated with it, which is specifically for the type of interaction and document in the batch.
A specific arrangement and sequence of elemental tasks constitute a job. Typically each type of scenario has one job. Figure 3-5 shows the Workflow window for Demo with its task hierarchy.
Figure 3-5 Setting up jobs on the Workflow page of Taskmaster Administrator
After the jobs and tasks are defined, they are assigned to users and workstations to provide a tight control over capture operations. The administrator creates workstations, users, and groups and then assigns them privileges (such as monitoring of jobs or workstations) and permissions to run specific tasks as defined in the workflow of the application. Figure 3-6 shows that user Verif1 is given permission to run the VScan and Verify tasks.
Figure 3-6 Assigning users to tasks on the Users page of Taskmaster Administrator
When user Verif1 launches the Taskmaster Client for the Payable application, only two tasks are available, Verify and Scan, as shown in Figure 3-7.
Figure 3-7 Scan and Verify operations assigned to an operator
A task processes an entire batch at a time. The task can be set up to run automatically on the next batch pending in the queue or to allow the user to select it. When a task is set to run automatically, the operator is served with the next batch. Then the process repeats until all outstanding batches for that task have been processed. For more information about tasks, task profiles, and rule processing, see “Rule processing” on page 71.
Batch Pilot for user-attended tasks
User-attended tasks vary depending on the specifics of the application. They are used for scanning, fixing batch issues, and verification.
To design and run the user interface for these tasks, Taskmaster provides a visual tool, called Batch Pilot, as shown in Figure 3-8 on page 62. This tool is used at design time to create forms with buttons, snippets, and fields, and the logic behind them. It is called at run time by the Taskmaster Client, according to the configuration done in the Task module of the administrator of the Taskmaster Client. For Verify tasks, forms reflect the particular data structure of the document hierarchy of the application.
In addition to Batch Pilot, Taskmaster includes a new tool, called Dot Edit, that dynamically generates runtime windows with data fields laid out in a list. Custom Dot Edit forms can also be designed by using Microsoft Visual Studio.
Figure 3-8 Batch Pilot in design mode
Thick Scan task
Taskmaster offers both thick and thin scanning capabilities. The thick client is adapted for high volume scanning, using the Image and Scanner Interface Specification (ISIS) interface. The thick client takes advantage of the rated speed and functions of high-end production scanners, such as duplex and multistreaming scanning. Figure 3-9 shows the Setup window for a thick Scan task.
Figure 3-9 Setup window for a thick Scan task
Taskmaster comes with a sample user interface for the Real Scan (rScan) task (Figure 3-10). This window can be customized to the specific needs of the scan operator, by using Batch Pilot, and inserted as the first step in the workflow of the application.
Figure 3-10 Review windows for a thick Scan task
Thick Verify task
Similarly, user interfaces for the Verify task are provided for each sample application that is delivered in Taskmaster. As shown in Figure 3-11, Batch Pilot runs the Verify form shown earlier in the design mode.
Figure 3-11 Batch Pilot running the Verify task
Figure 3-12 shows the user interface for the same Verify task running in Dot Edit, using a custom form. The customization only applies to the central part of the window where the fields and image snippets are paired up. The other panes of the window, such as the viewer and batch view, are generated automatically.
Figure 3-12 Dot Edit running the Verify task
As indicated previously, Dot Edit can also be used in list mode (Figure 3-13). Dot Edit uses the information of document hierarchy, image zones, and data fields already defined for the application in Datacap Studio.
Figure 3-13 Dot Edit automatically generating the user interface of the Verify task
In this case, contrary to a custom form, all fields are laid out as a list. You can edit them one at a time by selecting the field in the list, which opens the field in edit mode and calls the image snippet associated with it. For more information about Datacap Studio, see 3.7, “Principles and tools of the Taskmaster configuration” on page 94.
3.4.2 Taskmaster Web client
Taskmaster Web client offers the same overall functionality as the thick client. It runs tasks, monitors batches, and administers jobs, users, and workstations. Figure 3-14 shows the Job Monitor page of the Taskmaster Web client where you can monitor Taskmaster jobs.
Figure 3-14 Job Monitor page of Taskmaster Web client
Thin Scan task
One difference between the thick and thin client implementations is the support of scanners. A thick client can drive high-end production scanners through its ISIS interface. A thin client requires an ActiveX component to interface with TWAIN drivers. Another difference is that, for thin scan, you might need to insert an Upload task in the job after scanning. This task transfers the scanned document files from the scan directory to the Taskmaster Server over the web if the web server cannot access the scan directory directly. Figure 3-15 shows the Administrator page of the Taskmaster Web client.
Figure 3-15 Administrator page of Taskmaster Web client
An alternative to driving the scanner directly is to have the scanner write images to an import directory. Then use the Virtual Scan (VScan) task of the thin client to import and manipulate the images in the batch (Figure 3-16) and have the Upload task automatically transfer scanned images to the Taskmaster server.
Figure 3-16 Importing from a local directory using Taskmaster Web
The Taskmaster Web user interface offers generally the same productivity features as the thick client does. It includes image snippets, Click'n'Key, and color-coded backgrounds and inks to show confidence levels. The differences are in the implementation, such as adding buttons for validation instead of key short cuts, to account for the nature of the web browser interface.
Figure 3-17 shows the Verify task user interface in Taskmaster Web.
Figure 3-17 Verify task in Taskmaster Web
3.5 Taskmaster background processes
One of the strengths of Taskmaster is its ability to perform operations on batches in the background. In running the background processes, it relies on the Rulerunner engine and an extensive library of rules and actions.
This section explains how the Taskmaster workflow and rules are processed with the document hierarchy. It also provides an overview of the functionality offered by the Taskmaster actions libraries.
3.5.1 Rule processing
Rulerunner is the process that performs operations, called actions, on the objects of a document hierarchy in the background. It can be invoked manually, such as when a human operator validates a field. However, more typically, it is invoked automatically by Rulerunner. Rulerunner is set up to monitor the Taskmaster job queue and execute tasks automatically on batches as they move forward through the Taskmaster process.
Figure 3-18 illustrates how Rulerunner executes rules as specified in each task.
Figure 3-18 Rulerunner executing rules as specified in each task
Taskmaster has two hierarchies. It has a document hierarchy that describes the relationship between a batch, document, page, and field. In addition, it has a workflow hierarchy that describes the relationship between a job, task, rule set, rule, function, and action. Creating a Taskmaster application entails defining these two hierarchies and the interplay between them, as illustrated in Datacap Studio in Figure 3-19.
Figure 3-19 Datacap Studio user interface
3.5.2 Job, task, and task profile
A job is a particular combination and sequence of tasks in the workflow of an application to address a specific scenario, such as simplex or duplex scanning, thick or web scanning, and so on. When a job is run, Taskmaster creates a batch that is processed from task to task.
When a background processing task is run, Taskmaster invokes the rule sets that were defined in the corresponding task profile. This profile is a template that is used by Taskmaster as an entry point to instantiate a task at run time.
3.5.3 Rule set
A task profile is made up of several rule sets that are arranged in a particular sequence to produce the desired processing results. For example, a task profile can include Cleanup, Identify, Recognize, and Validate rule sets that are run in that order.
3.5.4 Rule
A rule set groups various rules that are bound together to the objects of the document hierarchy of the application and that are executed on demand as Rulerunner walks through the document hierarchy at run time. For example, in Figure 3-19 on page 73, consider the Locate actions that are used to locate words or regular expressions in the recognized data. They can be included in rules and rule sets at the document, page, and field levels, for the specific processing needs of that object. However, the relevant rule of the Locate rule set is only executed upon opening or closing the object of the hierarchy that is associated with it.
More generally, the rules of a rule set can only run when they are mapped to specific objects of the document hierarchy. They can also run only when the rule set they belong to is included in the task profile being executed. The execution order of rules in a rule set is dictated, first, by the order in which the parent rule set appears in the task profile and, second, by the processing sequence of the objects in the runtime document hierarchy.
3.5.5 Processing of the document hierarchy at run time
When a task is invoked, Rulerunner processes recursively each object that is found in the runtime batch (Figure 3-20 on page 75). It starts at the batch level and proceeds to separate the first document, then the first page in it, then all the fields in that first page, then the next page, and so on. It repeats this process with the next document. As it processes each object in this manner, it calls the rule sets that are bound to it. Rule sets can be configured to execute on opening or on closing the object.
Figure 3-20 Workflow and document hierarchies and processing sequence
3.5.6 Function and action
A rule is made up of one or more functions. A function consists of one or more actions. An action represents the code that runs a particular operation on the objects of a document. A function is started in the order in which it appears in the rule. If an action fails, the function that called it exits unsuccessfully, and the next function in the sequence gets executed. If the action succeeds, the next action in the function gets executed. If all actions of a function execute successfully, the rule that called the function exits successfully. By using this approach, you can construct efficient processing rules without coding.
For example, in a rule to identify a type of page (“Page identification” rule), several functions can be assembled in a fallback sequence, from the most to the least processing intensive or efficient. Each function implements a specific recognition technology.
The rule calls the following functions:
•“Identify using fingerprint” function
•“Identify using text match” function
•“Identify using pattern match” function
•“Identify manually” function
Manual identification, in effect only flagging the page for a subsequent user-attended task, is called only after fingerprint, text, and pattern matching all fail. If the fingerprint matching function succeeds, that is, if all the actions in it succeed, the “Page identification rule” exits successfully.
For information about document type identification, see 3.6.6, “Classification” on page 86.
3.6 Taskmaster action libraries
Many features in Taskmaster applications are implemented by using the rich set of action libraries that are available in the product. They are used in the course of processing a batch and its documents. They are also used for export, lookups, integrating with other systems, and system reporting. This section provides a cursory view of these actions before looking at how they are used in the Taskmaster process in the next section.
The action libraries delivered with the base Taskmaster product cover the full spectrum of functionality required in the Taskmaster process. However, by further combining these actions into functions and rule sets, you can implement a much larger functional scope than suggested purely by the elemental actions. The Taskmaster Accounts Payable Capture and Medical Claims Capture add-on applications are examples of this approach. You might want to consider these applications if you intend to implement similar solutions.
 
Important: The use of several action libraries requires additional licensing that is not covered in the license for Production Imaging Edition. See Table 3-1 on page 77 for more information about these libraries.
You can use the actions in Table 3-1 as part of the base license for IBM Production Imaging Edition.
Table 3-1 Summary of Taskmaster actions with the base license of Production Imaging Edition
Actions (library name)
Functionality
Virtual scanning (VScan)
Creates a batch and imports existing images in it from a directory in the file system.
Batch splitting (Split)
Splits a batch into smaller batches that are processed separately.
Color to B&W conversion (ColorToBW)
Changes the color depth of images.
Grayscale to B&W conversion (Grayscale)
Converts grayscale TIFF to bi-tonal TIFF.
TIFF-PDF conversion (DCPDF)
Converts PDF to TIFF and TIFF to PDF.
Image conversion (ImageConvert)
Appends images in a single file, and converts BMP, GIF, and PNG file formats to JPEG or TIFFs and TIFFs to JPEG or vice versa.
TIFF file merging (TifMerge)
Combines individual TIFF images into a multipage TIFF file.
Image clean-up and enhancement (DCImageFix)
Provides filters to despeckle, deskew, remove color, enhance characters, and so on.
Clip images (Dcclip)
Clips a portion of an image and saves it as a separate TIFF file.
Imprinting and redaction (Imprint)
Provides capabilities to overlay text on, or redact part of, an image.
Image overlay (Ioverlay)
Combines the current page image with a background image.
Barcoding (Barcode_x, Barcode_p)
Locates and recognizes 1D and 2D barcodes, and tests barcode values.
ICR processing (ICR_C)
Recognizes hand-written block characters by using the OpenText Recostar engine.
OCR processing (OCR_A, OCR_S, OCR_SR)
Recognizes machine-generated characters by using ABBYY OCR engines, Nuance OCR engines, or a combination for voting purposes. The ABBYY engine can also be used for recognition of certain types of marks (OMR).
Recognition (Recog_Share)
Performs various fingerprint- and recognition-related functions, including recognizing bubble options (OMR) using pixel threshold evaluation.
Runtime Document Hierarchy (DCO)
Sets up, tests, and modifies runtime information associated with each object of the document hierarchy in a batch (batch, document, page, and field). This information can include object statuses, types, recognition confidence levels, object variables, output field values, and fingerprint identifier.
Runtime zones (Zones)
Provides the capability to read and modify, at run time, zone information associated with each field in a fingerprint. It also provides the capability to locate the recognized text of specific zones, assign values to fields, and so on.
Fingerprinting (Autodoc and FingerprintMaintenance)
Provides capabilities to create and maintain fingerprints, match pages to them, and manipulate fingerprint information. Fingerprints are used to automatically classify structured and semistructured documents by using image analysis and field positional information.
Wordfire content-based classification (ICM)
Classifies text-intensive, free-form documents by analyzing their contents, rather than by relying on keyword, physical, or positional information. Requires the IBM Classification Module.
Data validation (Validations)
Extensive library to check and modify the content and format of field values. Also performs arithmetic calculations, assigns and copies values, checks variables, and so on.
Picture string matching (Picture)
Performs field validations by using picture strings that define the permitted character format of a field.
Data entry voting (Vote)
Compares the data of two data entry passes.
Intellocate
Updates the existing field position information in the document hierarchy or adds position information for a new fingerprint.
Locate text and navigate (Locate)
An extensive library used in combination with full text recognition to locate words or regular expressions on a page and to navigate around the page by line or word.
Lookup (Lookup)
Provides the capability to validate field values, by using database lookups, and to populate fields with lookup results.
Pattern matching (PatternMatch)
Provides the capability to search an image for a match to a pattern that is defined in the fingerprint library, for page identification and page registration (alignment).
File system operations (FileIO)
Provides access to the file system to copy, test, rename, and delete files and to set file attributes.
Output to email (Email)
Used to compose and send email from the Taskmaster Client by using CDOSYS (web) or a Simple Mail Transfer Protocol (SMTP) server (requires the Microsoft Outlook user to be logged in).
Export to text file (Export)
Exports and formats document hierarchy information, such as object variables and extracted data fields, to a flat file that can be used to feed external systems.
Export to database (ExportDB)
Exports and formats document hierarchy information, such as object variables and extracted data fields, to a database through an ODBC connection.
Export to XML (ExportXML)
Exports and formats document hierarchy information, such as object variables and extracted data fields, to an XML file that can be used to feed external systems.
Export to FileNet Content Manager (FileNetP8)
Exports document contents and metadata to a FileNet Content Manager repository.
Export to FileNet Image Services (FileNetIDM)
Exports document contents and metadata to a FileNet Image Services repository.
Export to IBM Content Manager (ibmcm)
Exports document contents and metadata to an IBM Content Manager repository.
NENU utility actions (NENU)
These actions are used by the notification utility in Taskmaster for batch monitoring, status notification, and automatic deletion of completed batches.
Rulerunner utility actions (Rrunner)
Performs miscellaneous utility functions, including checking batch integrity, manipulating the values of fields and variables, raising condition flags, and controlling rule execution.
The actions in Table 3-2 require additional licensing.
Table 3-2 Summary of Taskmaster actions requiring additional licensing
Actions (library name)
Functionality
Input from Email (Imail)
Imports image file attachments from an Exchange (through EWS) or Internet Message Access Protocol (IMAP) server into the current batch.
Electronic document conversion (Convert)
Converts Microsoft Excel, Word, Outlook email and attachments, image files, PDF to TIFF, and multipage TIFF to single-page TIFF.
Input from Fax server (OpenTextFaxServer)
Imports faxes from the inboxes of an OpenText Fax Server.
Export to SharePoint (SPExport)
Exports document contents and metadata to a Microsoft SharePoint repository.
Export to Documentum (Documentum)
Exports document contents and metadata to an EMC Documentum repository.
Export to LiveLink (LiveLink)
Exports document contents and metadata to an OpenText LiveLink repository.
The following subsections expand on several key capabilities that are implemented in the action libraries and explain how they are used to get the job done.
3.6.1 Image cleanup and enhancements
The image cleanup and enhancements library contains actions that clean up and enhance images to improve legibility and data capture processing in the later stages of the Taskmaster process. It provides filters and settings to perform the following actions:
•Deskew or straighten a crooked image to improve OCR performance and improve reading.
•Rotate an image typically by increments of 90 degrees when, for example, certain pages that display in landscape mode were scanned as part of a batch that was processed in portrait mode.
•Deshade to increase crispness and better reveal text on shaded areas and graphics.
•Dilate or erode an image to increase or decrease the thickness of the shapes in the image without changing their proportions and to make them easier to process. This action is especially useful for character recognition when characters appear thin, with discontinuities (an “l” looking like an “i”), or conversely, as a mass lacking details (an “i” looking like an “l”).
•Despeckle and reduce noise to remove random specks or background noise typically introduced when the scanner sensitivity threshold is too low in black and white, or by shaded backgrounds in forms.
•Smoothen characters to repair broken segments and smooth ragged edges as occurs when scanning dot-matrix printed documents.
•Inverse text to detect and reverse regions of the image that typically have white text on a black background, such as in table headings.
•Remove horizontal and vertical lines in high-density forms and tables to reduce clutter and enhance recognition of useful data.
•Remove the black borders that typically form on the edges of a black and white image that was scanned with high sensitivity settings.
•Remove streaks, vertical lines, or smears that are sometimes added to an image by the scanning process.
•Fill line breaks to repair broken lines, as in underlined text, for example, and to improve human legibility.
•Create an outline out of shapes on the image to sometimes improve legibility of a busy image by dropping unnecessary details.
These filters are typically used in combination and their proper mix and settings are derived from multiple trials and errors to arrive at the optimal result.
3.6.2 Barcode recognition
The barcode libraries are used to automatically locate and recognize one or several barcodes in an image. Barcodes are used to store data that can be detected with high accuracy, even on poor quality documents.
Many types of barcodes have been developed over the years to serve the needs of specific industries, such as retail, logistics, manufacturing, postal services, healthcare, airlines, and transportation in general. The choice of using a specific barcode in a Taskmaster application can be dictated by the barcode standard in use by the customer. Alternatively, you can decide the choice of barcode arbitrarily to satisfy the internal needs of the application. Such needs might be to identify a type of form, mark document boundaries in a batch, or reconcile incoming documents with a work item or other documents that belong together.
Taskmaster can detect 1D and 2D barcodes. One-dimensional barcodes (Figure 3-21) store a limited amount of data, typically a single alphanumeric string that can be used as a reference. They are coded by using a pattern of vertical lines of varying width read along the horizontal axis of the barcode.
Figure 3-21 One-dimensional barcode Code 39
Figure 3-21 uses the Code 39 barcode to code a claim number that can be used as a reference on outgoing correspondence to a customer to automate the claim identification process when the mail returns. Code 39 can store up to 43 alphanumeric characters.
Taskmaster supports the following one-dimensional barcodes: 2of5, Interleaved 2of5, Airline 2of5, Matrix, Matrix 2of5, Code 32, Code 39, Code 39 Extended, Codabar, Code 93, Code 93 Extended, Code 128, EAN13, EAN8, UPC-A, UPC-E, Addon5, Addon2, UCC128/EAN128, Patch Code, and PostNet.
Two-dimensional barcodes (Figure 3-22) can store up to several kilobytes of data. They are coded by using a matrix that represents information along the vertical and horizontal axes of the barcode.
The PDF417 barcode is a popular two-dimensional barcode that has many uses including to code driver license information in certain states. It codes information in multiple rows (from 3 to 90) that show clusters of bars and spaces. It is described as a “portable data file”, because the barcode itself carries the information, not just a reference to it.
Figure 3-22 Two-dimensional barcode PDF417
Figure 3-22 codes the entire postal address: “International Business Machines, 3565 Harbor Boulevard, Costa Mesa, California, 92626-1405, United States.”
Taskmaster supports the PDF417 and Datamatrix barcodes.
3.6.3 Optical Character Recognition
OCR technology is used to convert machine-printed text in an image back to editable text. It is at the core of the Taskmaster document identification and data capture process and is used for full page recognition and zonal data extraction. Taskmaster includes three OCR engines that can be used individually or in combination to offer the best possible results.
Although implementations differ, the three engines operate in a similar fashion. At a high level, the OCR engine analyzes the image and the textual zones and isolates individual characters. It then compares each character to a collection of template character bitmaps (in various fonts) and selects the closest match. It assigns each character a “recognition confidence level” based on how well it correlates with the template. Then it assembles the characters into words and resolves ambiguities by using dictionaries or lexicons and various other techniques.
The confidence levels are measured against the confidence thresholds set in Taskmaster. The higher the OCR confidence threshold is, the higher the number of errors is. Confidence levels of recognized zones are saved in the document hierarchy at run time to be used to color code the fields and drive the focus to problem fields in the Taskmaster Verify user interface. It is also possible to run multiple OCR engines on the same image to achieve the most accurate results across them all, which is a technique called voting. In that case, in each successive pass, Taskmaster compares the confidence level of the recognized data in the current pass with the one recorded in the previous pass. Then it only updates the field with the new data and the confidence level if it is higher.
Although the accuracy of OCR engines has improved over the years, it is affected by many factors. These factors include the page layout and background, image resolution, color/contrast/brightness, skewing, jagged characters, font type, size, and emphasis, type of compression, language, and character set. However, in most Taskmaster implementations, the following factors can be adjusted to produce the best possible results:
•Resolution, color, brightness, contrast, and skewing can be tuned at the scanner level or corrected by using Taskmaster image cleanup and enhancement actions.
•Character attributes, such as jagging, thickness, or thinness, can be corrected again by using image enhancement actions.
•Page layout, font type, and font size can be adjusted if the organization can influence the design of forms.
•Uncompressed or CITT Group 3/4 TIFF can yield better recognition results. In such case, Taskmaster offers the capabilities to convert to TIFF incoming images with other formats and compression schemes. They can still be carried through the Taskmaster workflow by using a separate stream to the repository. Alternatively Taskmaster can convert the TIFF images to a more compressed format at the Export stage.
•Recognition performance of national languages and character sets (especially accented characters) can be adjusted by selecting the OCR engine that yields the best results or by using voting.
3.6.4 Intelligent Character Recognition
ICR technology converts hand-written characters in an image to editable text. Figure 3-23 shows ICR technology used in a sample application.
Figure 3-23 ICR in a Taskmaster sample application
The overall operating principle of ICR is similar to OCR. However, because of the variability in handwriting, the techniques to separate and classify characters are different. Instead of trying to isolate and match whole characters, individual handwriting strokes are isolated and analyzed spatially to see which strokes belong most likely to which characters. Also, character classification requires a much broader base of shapes and more complex methods, including statistical probabilities, to disambiguate and identify characters and words with confidence.
ICR is affected by the same factors as OCR. However, it is affected by the clutter introduced by handwriting going over the boxes used in form fields and for normalizing character spacing (“constraint boxes”). Line removal or repair and dropout are typically employed to make the text stand out and improve accuracy.
3.6.5 Optical Mark Recognition
OMR is used in Taskmaster to detect whether check boxes, bubbles, or other types of marks have been selected. It is typically used in combination with other Taskmaster functionality that applies processing logic to the basic OMR results to interpret and turn them into actionable data. The results depend on the purpose and type of prompt or answer expected. For example, the answer might be yes or no, yes or no to all that apply, multiple choice, grid to form numbers or words, or add up marks.
Figure 3-24 shows OMR technology used in a sample application.
Figure 3-24 OMR in a Taskmaster Survey sample application
Although this technology has been in use for years, it is not easy to address the many situations that can arise and to determine the confidence levels and errors that trigger the Verify task. Factors that affect accuracy include the type of marks, filling method, variability in the response of the filler (spill over, too light, erasure), and interfering specks and background noise in the image.
Taskmaster provides flexibility to address these cases. When configuring the zones and fields, you define a parent field (for example, “seat”) as an OMR group and subfields (for example, “window” and “aisle”) to host options that belong together. You also specify in the parent field the number of options and whether multiple can be selected.
To do the actual recognition, Taskmaster offers two methods to address various situations. In most cases, when regular check boxes are used, the quickest setup is to use the OCR_A (ABBYY) engine. When the outline of the check box becomes removed through line removal or dropout, which is sometimes necessary to process high density forms, or when using bubbles or unusual check marks, you might need to use the “pixel evaluation method” instead. In this method, the zone defined for the check box is evaluated against a threshold of darkness (black pixel percentage) and a threshold of background noise that you specify to determine what is considered a checked mark. The difficulty with this technique is that it is more sensitive to variations in background noise or specks, which affect the pixel count for the zone. Therefore, adjustments might be necessary to find the appropriate values of these thresholds.
3.6.6 Classification
A great return on investment is derived from the ability of Taskmaster to automatically classify documents and drive the data extraction and indexing process when exporting the documents to the back-end repository. Therefore, it is one of the most critical pieces of functionality.
Given the variability in the types and quality of the documents that are routinely processed, it is often necessary to use a combination of several techniques to reliably identify documents. Taskmaster the following ways, among others, to classify documents:
•Content-based identification with IBM Classification Module (see 3.6.8, “Content-based identification with IBM Classification Module” on page 87)
Manual page identification
Manual page identification is used as a last resort when every other method has failed. With the help of automatic fingerprinting or content-based classification with IBM Classification Module, the number of manual interventions should decrease over time.
Structure-based identification
Structure-based identification is used in cases when the batch is fairly structured, that is, when the succession of pages can be predicted and used to determine the document type. In such case, you can use the actions of the runtime document hierarchy to arbitrarily set the page types.
Text and pattern matching
Text and pattern matching techniques are used on structured and semistructured documents. They are used when the types of documents are so close and the relative positions of zones are constantly changing that fingerprinting is unable to detect the type accurately.
Instead of looking at the overall page, as fingerprinting does, text matching attempts to identify a document based on searching keywords and phrases that unequivocally determine the type of document. It is called after the document has run through recognition, and therefore, typically requires more processing than just image analysis. Pattern matching concentrates on specific graphical marks or anchors in areas of an image.
3.6.7 Fingerprinting
The most innovative Taskmaster feature in the area of document classification is the use of fingerprints. A fingerprint is a unique signature of a page that is saved in the system and used to automatically classify incoming documents against those documents that have been processed before. The idea is that, if the fingerprint of the incoming document matches an existing one, you can safely assume that the incoming document is of the same class as the existing one. This technique is particularly well adapted for structured and semistructured documents that exhibit a fairly constant layout.
A fingerprint is made up of a sample image with a representative layout of the class of document and information that describes its geometric profile based on analyzing its pixel distribution. It can also be complemented with recognition results. The fingerprint is assigned a unique identifier that is saved in the fingerprint database.
Although all the documents of a same class look alike from a distance, every instance of a document is typically unique in that its actual contents are different from the ones before. Therefore, the chances of detecting the exact document are highest when the incoming document is a copy of the original document. Detecting an instance of an already identified class is a matter of measuring the proximity of the fingerprint of the incoming documents to the existing ones. The closest match has the highest probability of the instance belonging to the identified document class.
Fingerprints get perfected over time with additional information when more instances of the same class are routed to be identified by operators in the Taskmaster process. By using the fingerprinting and Intellocate libraries, you can configure your application to automatically create a new fingerprint and zone positions after an unrecognized page has been routed to a Verify task. This way, only verified recognition results are saved in the fingerprint, enhancing accuracy.
3.6.8 Content-based identification with IBM Classification Module
The techniques mentioned previously all have in common that they look for specific features of the document to identify and separate it from others in the batch. However, they work less well in cases when you need to process a mix of mostly text documents. Examples include miscellaneous customer correspondence, complaint letters, policies, statements, or affidavits with no predictable structure, logo, barcode, marks, or keywords. In such cases, you must understand the content in the same way as a person who is unable to recognize a type of document at first glance needs to read its content to make the determination. For this reason, Taskmaster relies on IBM Classification Module.
 
Important: IBM Classification Module connectivity is part of the base Taskmaster product and is included in the license for IBM Production Imaging Edition. However, IBM Classification Module connectivity depends on Classification Module, which requires a specific licensing.
Similarly to fingerprinting, IBM Classification Module creates a unique identity exclusively from the textual contents of documents. It looks for patterns, concepts, and associations and stores the results mathematically. This identity is then associated with a given type of document. Initially, document identification requires human intervention to match a given identity to a document type. However, IBM Classification Module can learn from the processing of a range of sample documents, and over time it requires no manual intervention.
At run time, the IBM Classification Module connector invokes IBM Classification Module and passes to it full-text recognition results. IBM Classification Module analyzes the content and compares it to its collections of identified types of documents. If it finds a match, it returns the type to Taskmaster. Otherwise, Taskmasters assigns a low confidence rating to the document, which causes it to be classified by an operator.
Because IBM Classification Module analyzes documents in their entirety based on concepts, it has a much larger scope to accurately identify a document than other methods can do, based strictly on a linguistic approach. Also the internal representation of information in IBM Classification Module makes it fairly immune to OCR or ICR and manual input errors.
In summary, with the help of ICM, you can perform the following tasks:
•Automatically identify text-intensive, free-form documents.
•Reduce prescan manual sorting and document separating.
•Enable automatic processing of mixed document batches.
•Process noisy OCR or ICR documents without operator intervention.
3.6.9 Language support
Taskmaster supports the several languages in the following ways:
•GUI: Brazilian Portuguese, Dutch, English, French, German, Italian, Polish, Spanish, and Swedish
•Data entry, display, and internal processing of character sets: All Latin 1 to 4 character sets
•OCR/A (ABBYY) engine: Czech, Dutch, Dutch Belgian, English, Finnish, French, German, Hungarian, Italian, Lithuanian, Polish, Romanian, Slovak, Spanish, and Turkish
•OCR/S (Nuance) engine: Nuance Omnipage (OCR/S, ocr_s and ocr_sr libraries): Afrikaans, Albanian, Catalan, Croatian, Czech, Danish, Dutch, English, Estonian, Finnish, French, German, Hungarian, Italian, Icelandic, Latvian, Lithuanian, Maltese, Norwegian, Polish, Portuguese, Slovakian, Spanish, Slovenian, Swedish, Turkish, Esperanto, Romanian, Serbian (Latin), Faroese, Gaelic Irish, Gaelish Scottish, Rhaetic, Sami, Northern Sami, Southern Sami, and Swahili
•ICR/C (RecoStar) engine: Afrikaans, Albanian, Bosnian-Latin, Catalan, Croatian, Czech, Danish, Dutch, English, Estonian, Faroese, Finnish, French, German, Hungarian, Icelandic, Irish, Italian, Latvian, Lithuanian, Norwegian, Polish, Portuguese, Rhaeto-Romanic, Romanian, Serbian-Latin, Slovak, Slovenian, Spanish, Swahili, Swedish, and Turkish
•IBM Classification Module: Dutch, English, French, German, Italian, Norwegian, Portuguese, Spanish, and Swedish
In addition, the following vertical dictionaries can be selected in the following languages for the OCR/S engine:
•Legal and medical dictionaries: Dutch, English, French, and German
•Financial dictionary: English
3.6.10 Imprinting and redaction
Taskmaster provides an imprint library that can be used to overlay text on an image. Alternatively, the library can be used to redact part of the image to protect personal information from public view, which is the case with health records and social security information.
You call the redaction action in a rule, attached to a target field defined in a fingerprint, to black or white it out. Alternatively, you can use text locating actions to find, navigate, and position automatically the redaction zone over the data in unstructured documents. The resulting imprinted content or redaction is flattened and burned in the image.
Typically when redaction is used, two capture streams are implemented in Taskmaster. First, the redacted documents are committed to a repository with access rights for general circulation. Then the originals are committed to a secure repository with restricted access rights for authorized personnel.
3.6.11 Locating text
Taskmaster offers an extensive library of actions that can be used in combination with text recognition to locate words and regular expressions and to navigate around a page. These actions are used in instances when text location cannot be predicted, such as in semi-structured or free forms. They are coupled with actions such as those used to process line items in purchase orders and invoices.
3.6.12 Validations
Taskmaster offers an extensive library of elemental actions. These actions help you to validate and manipulate the data captured in the objects of the runtime document hierarchy and to ensure that they conform to your business rules.
The following actions are possible:
•Data formatting actions to normalize field values or prepare them for calculation or comparison purposes, including the following actions:
 – Padding with zeros or spaces to match the expected number of characters
 – Deleting a specific character in a specific position or all instances
 – Deleting a class of characters from a field (alpha, numeric, punctuation, nonalphanumeric, or system characters)
 – Testing data types (date, currency, alpha, or numeric) and field length
 – Converting the case of characters or value to currency
 – Clearing a field value
 – Inserting a decimal point or a specified character in an existing field value
 – Trimming spaces and truncating and splitting field values
 – Parsing postal addresses and names and populating individual fields
•Manipulation of field values and document hierarchy variables, including the following actions:
 – Assigning default values to fields
 – Copying or appending values between fields
 – Comparing field values and verifying arithmetic calculations between fields
 – Comparing dates and testing them within a range or days
 – Assigning data or a time stamp to a field
 – Testing field content: Filled, empty, max or min length, specified matching value, or percentage numeric or non-numeric data
 – Testing the number of OMR boxes and whether they are selected
 – Testing a regular expression in a field
 – Testing variables and assigning variables to fields
 – Summing up values of subfields
•Invoking a message box to provide guidance in the Verify task
For example, by using combinations of these actions, you can create rules and attach them to any object of the document hierarchy to test the following information:
•Values are within accepted ranges.
•Data is formatted as required.
•Dates are valid, and deadlines are met.
•Numbers add up or are not missing.
•Mandatory fields and check boxes have been completed.
•Dependencies between data are respecte.d
•Data matches sets of permitted values, by combining with lookup actions
Upon failure of any of these rules, Taskmaster flags the associated field and page for manual review in a Verify task, similarly to low confidence recognition results.
Lookups
As a good practice, check and normalize data early in the business process to reduce errors and enforce consistency. To achieve this objective, Taskmaster provides a set of actions that you can use to connect through ODBC or OLE DB to a business database hosting reference information. Such information might include customer names, part numbers, or nomenclature. Then you run an SQL query and pass back a result set to populate fields in the runtime document hierarchy.
By using the rule execution logic outlined in “Rule processing” on page 71, you can handle lookup errors through a function that is called upon failure of any of the lookup functions.
3.6.13 Exports
The Taskmaster process completes the processing of a batch by persisting the captured documents and metadata to the Enterprise Content Management (ECM) repository. In many cases, it also exports some of the business data to line of business systems such databases or applications. It can output data and contents in a format that is compatible to the input stage of another system, such as IBM Content Manager On Demand.
Taskmaster includes libraries to export to IBM and non-IBM ECM repositories, relational databases, XML, and flat files.
Exporting to FileNet Content Manager
In a Production Imaging Edition implementation, you use the FileNet P8 action library to commit documents to FileNet Content Manager. It provides the following capabilities:
•Establish a connection to a FileNet Content Manager system.
•Attach to a given object store and FileNet P8 document class.
•Define a root folder and create a subfolder to store your documents.
•Map the Taskmaster field values and variables of the runtime document hierarchy to the properties defined in the FileNet Content Manager document class.
•Upload documents to the destination folder.
To bind preparation actions, such as connect, login, and create directory, to the open event at the batch level, bind the populating of document properties. Then upload to the open event at the document level, which automatically iterates for each document of the batch. Figure 3-25 shows the actions to export documents to FileNet Content Manager.
Figure 3-25 Actions to export to FileNet Content Manager in Datacap Studio
Exporting to a database
Similarly, to export to a database that is accessible through ODBC or OLE DB (Figure 3-26), the ExportDB library provides the following actions:
•Establish and close a connection to the database.
•Open the target database table.
•Assemble each database record in memory, and populate it with data from Taskmaster field values and variables of the runtime document hierarchy.
•Commit the database record.
You bind the actions to open the database to the open the events at the batch level. You also bind the actions to close the database to the close events at the batch level. In addition, you bind the actions to create the data records to the open event at the document level.
Figure 3-26 Actions to export to a database in Datacap Studio
Exporting to a flat file
By using the Export library, you can output data to the file system to be picked up for import by another system. The library provides the following actions:
•Set the path, file name, and extension of the export file.
•Format text (new line, blank lines, fields, filler characters, value and OMR separators, value justification, value length, OMR separator, and so on)
•Output text, date, time, field values, and variables of the runtime document hierarchy, and filter on field status
•Save the export file
The information that needs to be exported and how to code the export file depend on the target system. For example, this method can be used to feed the Content Engine Bulk Import Tool as an offline alternative to a direct connection to FileNet Content Manager. Alternatively, it can be used to import documents in IBM Content Manager On Demand by using its ARSLOAD utility.
You bind the actions to create the export file and write out the data needed once to the open event at the batch level. Then you bind the actions to output the information required for each document to the open event at the document level. Finally, you bind the action to save the file to the close event of the batch level.
3.7 Principles and tools of the Taskmaster configuration
This section reviews the tools that are available to configure, deploy, and monitor a Taskmaster application, and to report on its activities.
At a high level, the principles for setting up a Taskmaster application are easily understood if you visualize a document and the types of data that you are trying to extract from it. A document is made up of pages that are typically identifiable by certain characteristics. Such characteristics include a specific structure (cover and trailing pages), the layout of each page, and the location in the page of the specific pieces of information that you are seeking to extract.
Configuring an application consists of providing Taskmaster with a combination of visual clues and processing rules from its catalog of actions. They drive how to automatically recognize and separate the documents; find, capture, and process the data in them; and transfer both images and data to the back-end systems.
Figure 3-27 shows a Taskmaster application and various areas of configuration.
Figure 3-27 Configuring a Taskmaster application
You use Datacap Studio to configure the following information:
•The document hierarchy for your document types
•The fingerprints based on your sample document types
•The zones and data fields to extract
•The rules and actions to execute on the objects of the document hierarchy
•The tasks profiles to expose as actionable functionality in the workflow
For user-attended tasks, you use Batch Pilot or Dot Edit to define and bind the user interface specific to the current task. For more information, see “Batch Pilot for user-attended tasks” on page 62.
After the task profiles and core functionality of the application are designed and tested in Datacap Studio, you expose them to users, workstations, and Rulerunner by using Taskmaster Client. For more information, see “Taskmaster Client (thick client)” on page 58.
If you implement a Flexible Capture application, you must use Flex Manager to configure the data structures and document classes that you expect in your application.
Use the Taskmaster Application Manager to deploy your application on multiple machines and manage the registry of the Taskmaster system components. Use the RV2 Report Viewer to configure and access Taskmaster real-time activity reports over the web. Use NENU to set up and manage system health monitoring and notification, in addition to recurring house cleaning tasks.
3.7.1 Datacap Studio
Datacap Studio is the primary tool to configure and test Taskmaster applications. It offers an intuitive visual environment that makes it easy to assemble the various components of an application. It is organized around three work areas:
•Rule management for defining the document hierarchy to the field level and for tying processing operations to its components
•Zone management for creating classes of fingerprints from document samples and for defining graphical extraction zones tied to the fields of the document hierarchy
•Testing for running batches, checking the state of the objects in them, and debugging your application
Everything in Taskmaster is defined within the scope of an application, including the document hierarchy, zones, fields, fingerprints, rule sets, and tasks. The workflow and allocation of tasks to users and workstations and the user interfaces that are executed by user-attended tasks are also defined.
An application focuses on a specific set of document classes. The more document classes there are to process, the more complex the application is because more functionality must be built into the application to discriminate and address the various cases.
The document hierarchy describes the structure of the documents that are processed in the application, including batch, document, page, and field. This information is the basis for most of the other setup activities, because it describes the objects that are the targets of subsequent processing.
To create the document hierarchy, you import in Datacap Studio template pages that are representative of the class of document. For each page, Taskmaster creates a specific identity, called a fingerprint, based on its visual characteristics (Figure 3-28). Then you further define the zones where you want Taskmaster to extract information from. Taskmaster uses this information to automatically determine the type of document and identify the fields where to extract data.
Figure 3-28 Defining fingerprints, zones, and data fields using Datacap Studio
Rulerunner runs rules and actions to perform the actual extraction work, document identification, data manipulation, and all the processing that happens behind the scene. You need to specify the actions that must be performed on which object of the document hierarchy for the particular task being executed. In concrete terms, you must assemble elemental actions into functions and rules and bind them to the appropriate elements of the hierarchy.
For example, at the batch level, you need to configure rules that apply to the entirety of a batch when it is processed, such as when importing, scanning, or exporting. At the document level, you set the index properties for the whole document from the extracted data when exporting to a repository. At the page level, you set OCR or barcode options when performing recognition and data extraction. At the field level, you typically set field validation rules.
Rulerunner is readily available with hundreds of functions and actions that can be assembled in rule sets as needed. To expose and invoke the Taskmaster functionality in the workflow by operators or Rulerunner, you assemble the appropriate combinations of these rule sets as task profiles that can be used by the Taskmaster clients. Examples of the tasks include Scan, Profiler, or Verify. Figure 3-29 shows actions and rules mapping to the document hierarchy.
Figure 3-29 Datacap Studio - actions and rules mapped to document hierarchy
After you complete the configuration of your application, you can use Datacap Studio to test it. For example, at run time, you can test the values that are being extracted from the zones and check that the actions produce the expected results.
To help you start, Datacap Studio also includes an Application Wizard that generates the basic infrastructure of a new application. It takes you through the following steps:
•Creating a document hierarchy
•Creating a fingerprint class and template document
•Adding a few sample images
•Creating the application directory structure and configuration files for that application to run, including a standard workflow with thick and web client jobs.
With Application Wizard, you can also derive a new application from an existing one. This way, you can take any of the sample or foundation Taskmaster applications that most closely resemble your own use case. Then you can modify it to match your needs by using Datacap Studio, giving you a substantial head start.
3.7.2 Flex Capture and Flex Manager
The Flex Capture sample application is intended for cases where you can describe the types of documents that exhibit predictable types of data, if not a predictable layout. Therefore, by using its Click'n'Key and Intellocate functionality, Taskmaster quickly learns how to automatically recognize the documents from the input of an operator.
To set up the application, you define the data types that you expect in the documents, such as a purchase order number, social security number, or part number. You define the types of data that have a unique character pattern and that can be automatically located and used to help identify the type of document.
In most cases, you can expect to have much variability within a given class of document. For example, purchase orders are likely to all exhibit something called a “Purchase Order” with a specific numeric pattern. However, they might be found in different locations or in combination with a varying number of other types of data. As humans, we can easily recognize classes of document by looking at them and identifying the data types even though they are not always the same. The idea implemented in the Flex Capture application is to get Taskmaster to learn from a human operator by recording the moves of the operator.
When a document cannot be automatically recognized, it is flagged as a problem document and is routed to an operator for manual processing. The operator then selects the class of document and directs Taskmaster to automatically locate the expected data types for that particular class of document. However, it is likely that Taskmaster will be unable to locate some data because the document deviates from the known fingerprints. In such case, the operator needs to locate and select the missing data in the image for actions to be recorded by Taskmaster. Then, when the batch is finally processed in the Export task (after having been verified), Taskmaster automatically adds the image and recognition information to the fingerprint collection. It also adds the location of the zones that have been selected to the document hierarchy. When the same type of document is processed again, Taskmaster automatically recognizes the document type.
Flex Manager is a utility that comes with the Flex Capture sample application. As shown in Figure 3-30, it helps you to define individual data types with the character filter to exclude unwanted characters and a picture string to specify the data format. It also helps you define whether the data is required and to define a regular expression to define the range of valid values.
Figure 3-30 Defining the data types by using Flex Manager
After you define the data types that you need, you can create the document class that matches your type of document and assign it the appropriate combination of data types as shown in Figure 3-31.
Figure 3-31 Defining document classes by using Flex Manager
3.7.3 Taskmaster Application Manager
Application Manager manages environment-specific information about the applications of a Taskmaster system in a central registry. This way, the Taskmaster system components that are running on different machines can be made aware of each other and get access to the information they need to operate. Such information includes pointers to the Taskmaster Server, databases, fingerprint library, and file server.
Application Manager maintains a separate registry for each application, which is cross-referenced by the central registry. To deploy an application on multiple machines, or from a development to a production system, you move the resources of the application (configuration files, working directories, databases, and so on) to their target machine. Then you update the registry of the application with the new locations and the cross-references between the registry of the application in the central registry.
Application Manager (as shown in Figure 3-32) is typically installed in restricted (read-only) mode on all workstations, except the machine running Datacap Studio to prevent users from modifying their deployment settings. In restricted mode, only the reference to the central registry can be modified.
Figure 3-32 Taskmaster Application Manager
In Application Manager, you can define which tasks you want Rulerunner to run automatically in the background, for each application. You can also store application-specific custom values, such as a connection string to a lookup database or even credentials. These values can be retrieved and passed at run time to Rulerunner actions, avoiding hardcoding this information in the rules and making them more portable and secure.
3.7.4 RV2 report viewer
RV2 is a web-based tool to display real-time activity reports on the Taskmaster system. It is delivered with a set of preconfigured reports to monitor batch status, station activity, problem batches, batch aging, batch productivity, and so on, across multiple applications. Custom reports can be developed by using Microsoft Visual Studio and Windows Forms.
You can set up RV2 to report across multiple applications, display individual reports or a dashboard of reports, and filter on specific column values. Report contents can also be exported to PDF or Microsoft Excel, or they can be printed out. Figure 3-33 shows the RV2 report viewer.
Figure 3-33 RV2 report viewer - pending tasks at various steps in the process
3.7.5 NENU
The NENU component automates recurring system health and house cleaning tasks, such as batch monitoring, status notification, and automatic deletion of completed batches. Tasks are scheduled by using the Microsoft Windows Scheduler.
NENU is a versatile tool that can execute any rule sets and actions defined for it in Datacap Studio by associating its rule set and task profile to the applications that you want to monitor. Typically you use NENU to perform selections in the Engine database of the application and execute actions on the selected batches. For example, you might run the following actions:
•Monitor batches, notify statuses, and automatically delete completed batches.
•Identify batches that meet certain criteria, such as batches that stopped.
•Change the status of batches and their order in the queue.
•Delete batches or move them to another location.
•Capture data snapshots to a database to be reported by using RV2.
•Send email notifications, such as of error conditions or a batch stopping.
You can run NENU in three ways:
•Manually using the NENU Manager
•Automatically using the Windows Task Scheduler, either at scheduled times or when triggered by a system event
•Automatically as a task of the workflow of an application
Figure 3-34 shows the NENU Manager window.
Figure 3-34 NENU Manager window
3.8 FileNet Content Manager for production imaging
IBM FileNet Content Manager provides the repository functionality to store, secure, organize, index, and retrieve images and data that are extracted by Taskmaster Datacap. It also provides a document-centric workflow infrastructure to automate the routing and processing of these documents.
FileNet Content Manager has a range of advanced capabilities to manage electronic documents and their life cycle. It can handle large volumes, making it a robust platform for document imaging.
As indicated in Chapter 1, “Production imaging overview” on page 3, FileNet Content Manager typically comes into play in the post-committal stage of the document imaging process. You use it when the Taskmaster process completes by storing the captured documents and their metadata in the repository for processing by business users.
At a minimum, you typically use the following FileNet Content Manager functionality:
•One or several libraries, or Object Stores, to host and organize logically the document collections and to provide the document management logic for users to query and retrieve documents
•File Storage areas to provide storage for the images
•Document classes that match the business artifacts that are processed in Taskmaster (such as claims and invoices)
Document classes define the metadata, or properties, that are needed for retrievals, but also for processing the documents in the business process. Document classes in that context are also seen as a vehicle for transferring data from Taskmaster to the workflow itself.
•Workflows that define the work items parceled out to business users and the processing routes to be followed
Workflow definitions include the data fields, steps, routing conditions, document attachments, and so on. Work items are distributed to the inboxes of users in FileNet Workplace XT. In a single interface, they bring together the data and documents that users need to perform their tasks. All aspects of a workflow definition are specified in a workflow map. A workflow map is an XML document that is stored in the repository and used by the Process Engine as a template to instantiate a workflow at run time.
•Workflow subscriptions that define the relationship between a document class and its metadata, a workflow, and a type of event
When an event (such as creation) occurs on the document, a workflow instance is launched with the document automatically added as an attachment. Also the work item fields are populated with the data transferred from the mapped document properties.
For more information about the capabilities of FileNet Content Manager and planning for its implementation, see the following IBM Redbooks publications:
•IBM FileNet Content Manager Implementation Best Practices and Recommendations, SG24-7547
•IBM FileNet P8 Platform and Architecture, SG24-7667
3.8.1 Workflow management tools
In the course of implementing a Production Imaging Edition project, you must use the following workflow management tools, which are included in FileNet Content Manager:
•Process Designer to design the workflow maps and optionally Microsoft Visio Connector to conveniently edit workflows in Microsoft Visio
•Process Administrator and Tracker to search and provide the status of executing workflow instances
•Case Analyzer to gather workflow statistics
•Process Simulator to project execution of workflow in production
The following section provides high-level information about what these tools are used for and how they participate in the configuration of a Production Imaging Edition implementation. For more information, see Introducing IBM FileNet Business Process Manager, SG24-7509.
Process Designer
Process Designer is used to design the workflows that drive the processing of Production Imaging Edition documents after they are committed to Content Manager. In addition to delivering Production Imaging Edition documents to business users as attachments, the workflow work items carry data that is extracted and verified by Taskmaster. The work items also carry data that constitutes the intelligence needed for applying efficient routing logic and for users to make informed decisions.
The Process Designer defines the activities and resources that are required to accomplish a particular business process. They are represented as a graph, with a series of process activities, or steps. These activities are connected by routes, or vectors, that define the sequence in which the steps are run and the transition conditions that must be satisfied for routing. Steps and routes are organized into reusable maps. Figure 3-35 shows a claim process map in the Process Designer.
Figure 3-35 Claim process map in Process Designer
Process Designer defines the following items among others:
•The instructions to complete the task at each step
•The sequence of steps and step processors (interfaces)
•The data fields and attachments of the work item and how they are exposed and used by step processors and users at each step
•The routing and transition logic that is used on arbitrary responses or the data fields to advance the workflow from step to step along the business process
•The users and roles, system process, or shared work queue that is assigned to each step
•Timers and the deadlines for completing tasks
Process Designer also provides the capability to validate and run a workflow in real time. It and saves the process definition in the repository in the form of an XML Process Definition Language (XPDL) file. It can also import existing processes from IBM WebSphere® Business Modeler or Microsoft Visio.
Microsoft Visio Connector
With the Microsoft Visio Connector, you can use Microsoft Visio to create diagrams of business processes. Then you can import them into Process Designer as a starting point for creating a workflow.
For business analysts who are familiar with Business Process Modeling Notations (BPMN), a Visio stencil of BPMN shapes is provided with each shape. The stencil is mapped to a corresponding Process Designer object, such as a step, route, text annotation, data object, Workflow Group, or submap. Shapes in standard stencils are also mapped to Process Designer objects. You can customize the mapping for shapes on existing Visio diagrams.
On the left side of Figure 3-36, you see the standard shapes that can be used to create a standard Visio diagram and how it looks in Process Designer after it is imported. On the right side, you see the BPMN notations stencil and a sample Visio workflow with submaps and how they look in Process Designer.
Figure 3-36 Process maps and Microsoft Visio connector
Process Administrator and Process Tracker
Process Administrator (Figure 3-37) is used to manage executing workflows, search and view specific instances, and edit their data and properties while they are running. Process Administrator is a powerful tool that workflow administrators use to monitor work items and resolve processing issues.
Figure 3-37 Process Administrator
After you select a specific workflow instance, you can view its status in the Process Tacker. Process Tracker provides the status of a currently running workflow in a graphical view by using the workflow map of the process definition created in the Process Designer (Figure 3-38).
Figure 3-38 Process Tracker displaying the runtime status of a workflow instance
From the graphical view, you can tell which steps have been completed in the workflow, when they were completed, and which steps are currently active.
Case Analyzer
Case Analyzer is used to discover business processing trends (as opposed to looking at specific instances to establish a baseline for measuring productivity) and to identify bottlenecks. It monitors and analyzes large numbers of events generated by the Process Engine, which it aggregates for analysis in an OLAP cube stored in a Microsoft SQL server database at regular intervals.
For reporting, Case Analyzer uses Microsoft Excel by default, which has a rich set of chart and reporting functions for viewing and analyzing the data stored inside the OLAP database.
The Case Analyzer comes with the following set of standard reports:
•Productivity, which measures wait, processing, and completion times of the work items in the various steps and queues (Figure 3-39)
•Queue Load, which measures the number of work items added, completed, and currently left in a particular queue or step
•Workload, which measures the number of workflow runtime instances created, completed, and currently executing, and which measures the average processing time of the various workflow instances
•Work in progress, which provides a real-time view of the currently active work items by measuring their count in various steps and queues and by measuring the time that the work items have spent at the current step
•Workflows in progress, which provides a real-time view of the currently active workflow instances by measuring their count and duration
Figure 3-39 Sample Case Analyzer report
You can also create your own custom reports by using IBM Cognos® Analysis Studio or IBM Cognos Reports if you require more sophisticated reporting capabilities.
Process Simulator
You use Process Simulator to validate a process before placing it into production. Process Simulator can be used in two scenarios. The first scenario consists of validating a new process, where the user provides the information for the simulation based on the experience and knowledge of the user about the domain. The second scenario consists in validating enhancements to an existing process, where historical data from Case Analyzer is used to feed the simulation.
In either case, the user can simulate what-if scenarios and then analyze the results of the simulation to validate the process. That way, a process analyst can test different scenarios to improve the business process before deploying it in production.
You access Process Simulator from the Process Simulation Designer and the Process Simulation Console. The Designer is used to create simulation scenarios that can be saved in the Content Engine repository. Process Simulator supports versioning for the saved scenarios.
The Process Simulator Console is used to manage the scenarios and to execute simulations that have been created. The results of a simulation execution are stored in the Case Analyzer database. The simulation object itself, which contains animation information, is stored in the Content Engine for future animations. Animations are run in the Animator as shown in Figure 3-40 on page 113.
Figure 3-40 Process Animator running a simulation
3.9 Advanced production imaging viewing
The IBM Production Imaging Edition Viewer is based on Daeja ViewONE Pro. It expands on the functionality delivered in the standard viewer of FileNet Content Manager with the following features:
•PDF viewing and annotating directly in the viewer
•Universal viewing and annotating of electronic documents
•Document streaming to improve response time when displaying large documents
•Permanent redaction to burn redactions and annotation to TIFF and PDF
3.9.1 PDF viewing and annotating
With the PDF module of the Production Imaging Edition Viewer, you can view PDF documents without having Adobe Acrobat Reader installed.
You can view Acrobat annotations that have been saved to the PDF document, but you cannot edit them. However, you can add Daeja annotations to the PDF documents that are stored outside the document, similar to other annotations on images in FileNet Content Manager (Figure 3-41).
Figure 3-41 Adding an annotation in a PDF document
You can also search the contents of PDF documents that have text in them by using a search box. Daeja finds all the occurrences of the word you are looking for and highlights them in the page that you display.
The streaming module works with searches. This way, when you search a large PDF document, the search is deferred to the streaming server with only the results. Also the page numbers embedded within them returned to the client, so that the viewer knows which page to jump to. When selecting a given page, the viewer working with the streamer component only retrieves that page. It does not download the entire document on the client, which happens otherwise with a standard PDF document download.
Redaction is also possible on PDF documents. Permanent redaction creates a PDF document with the redacted area secured in such a way that the text is removed and is not searchable in the output PDF.
3.9.2 Universal viewing and annotating
The Universal Viewing module provides the Production Imaging Edition Viewer with the capability to view and annotate Office documents and many other electronic document formats.
Similar to PDF, you do not need to have and launch the native applications every time you want to display a document. You can browse through TIFF, PDF, and Microsoft Office documents in work items without switching applications. You can annotate them in the same way as you do with TIFF documents.
No installation is required on the client for the Universal Viewing module to work. It is downloaded transparently to the local cache folder the first time you view a supported electronic document.
3.9.3 Document streaming
The streaming module is used to reduce network traffic and improve the performance perceived by the users. The largest impact of this feature is on TIFF and PDF documents that have a higher number of pages. It splits up the document into individual pages and sends each page to the client on demand.
The streaming module has a servlet in FileNet Workplace XT (see the architectural details in Chapter 2, “System architecture” on page 35) that caches the full document, but only delivers the pages requested by the Production Imaging Edition Viewer. When used with PDF, it also compresses the PDF pages that are sent to the Production Imaging Edition Viewer applet.
Also, as indicated previously, streaming also works with searches so that the search is run by the streaming servlet. In turn, the servlet sends, to the Production Imaging Edition Viewer applet, the results with the page index for the viewer to know which page to get from the streaming cache.
3.9.4 Permanent redaction
The Production Imaging Edition Viewer also provides a permanent redaction capability. The primary usage is to redact the areas of a document that you do not want to make public, such as a social security number. The Viewer permanently “burns” the redactions in the document so that the information cannot be recovered and you can safely distribute a copy of the document.
Redaction is supported on any format that can be viewed. For TIFF and electronic documents, such as Microsoft Office documents, the redacted document is output to TIFF (see the example in Figure 3-42). For PDF, the redacted document is output to PDF.
Figure 3-42 Redacting a TIFF document in the Production Imaging Edition Viewer
The source PDF document might include text, such as with electronic documents converted to PDF or image documents that have been recognized by using OCR results saved to PDF. In this case, the text of the redacted areas is securely removed by the redaction and cannot be searched or recovered. However, the rest of the PDF content is not affected and can be viewed, and the text is used in searches.
When used with TIFF and electronic documents, the burning part of the redaction process works with most annotations, so that you can create a permanent annotated copy of a document.
With PDF, you can only use rectangular fill areas to redact the document, and the other annotations are not available for burning.
The redaction process is initiated from the Redact to File menu on the document selected in FileNet Workplace XT and sends the redacted output file to the client to be saved locally.
Similar to the Streamer component, the Permanent Redaction component is implemented as a servlet. It is invoked from the Annotation module in the Production Imaging Edition Viewer applet to do the burning to TIFF or PDF and send the output file to the client. (For architectural details, see Chapter 2, “System architecture” on page 35.)
 
Production Imaging Edition Viewer redaction versus Datacap Taskmaster redaction: Production Imaging Edition Viewer redaction is different from the redaction capability offered in Datacap Taskmaster. Taskmaster redaction is typically applied at the precommittal stage where the original, unredacted document, might or might not be stored in the repository. The Production Imaging Edition Viewer redaction is applied manually on documents at the postcommittal stage. The resulting redacted content is saved on the client without affecting the source document in the repository.
3.10 Bulk Import Tool
The Bulk Import Tool is a new utility that ships with FileNet Content Manager 5.1 and later. It provides the capability to ingest large volumes of documents quickly in one or several Content Engine Object Stores.
 
Licensing: Customers can use the Bulk Import Tool as part of the license for Production Imaging Edition.
For example, you can use it in a production imaging scenario for the following purposes:
•Decoupling scanning operations from the back-end ECM system, as is the case when subcontracting digitization to a service bureau
•Migrating large volumes of documents from an existing system
•Ingesting automatically machine-generated documents, such as utility statements
The Bulk Import Tool monitors file system directories for batches of documents to import. It looks specifically for the batch description files (<batchnumber>.eob) that it uses to create corresponding batch import entries in the Content Engine import queue. Then it triggers the import process. Figure 3-43 shows the Bulk Import Tool processing model.
Figure 3-43 Bulk Import Tool processing model
The batch import process reads the queue entry information of Content Engine that points to the location of the metadata description file (transact.dat). This file accompanies each batch and contains the index information of each document with the references to the content files (for example, TIFF files). It then creates and indexes all the documents in the batch and commits it to the Content Engine. For each batch, the Bulk Import Tool generates a journal file that logs the results.
You can configure the Bulk Import Tool to run multiple instances, each configured for a specific Object Store, in addition to batch size, sleep time between runs, and number of background threads to Content Engine. This flexibility provides increased throughput and helps you to tune the consumption of system resources.
The Bulk Import Tool has been modeled after the High Performance Image Import (HPII) utility, which has been used by FileNet Image Services customers for years. It offers near compatibility with earlier versions of HPII so that customers who currently use HPII for Image Services can use the Bulk Import Tool with minimal changes to their current environment.
3.11 Conclusion
This chapter focused on the key functions, component and tools of Taskmaster, how the process works, and how to build an application. It also highlighted FileNet Content Manager and the advanced viewer.
With this knowledge, in Part 2, “Solution implementation” on page 121, we guide you through an actual implementation of a Product Imaging Edition solution with a use-case solution sample. We begin with the solution design and take you through the step-by-step solution implementation.
 
..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset