The objectives of this chapter are to:
Introduce the batch processing framework in Microsoft Dynamics AX 2009.
Describe typical scenarios for using the Batch framework.
Provide instructions for how to create batch tasks and batch jobs.
Explain how to use Dynamics AX tools to manage and control the batch server exection process.
Provide an overview of the X++ Batch API and examples of its usage.
Microsoft extensively updated the framework used to process batch jobs—the Batch framework—for the Dynamics AX 2009 release. Among other enhancements, the Dynamics AX Batch framework is now server based; it gives system administrators and developers increased control over batch jobs and enables greater performance and reliability when those jobs are processed.
To support the new Batch framework, Dynamics AX has some new tools. The Batch Job designer form gives system administrators increased flexibility in the design, setup, and execution of complex batch jobs as well as the ability to add many batch tasks to a single batch job and to define the dependencies among those tasks. An enhanced Batch API gives X++ application developers more control over complex batch jobs along with the ability to process batch jobs directly from business logic.
This chapter introduces the Dynamics AX 2009 Batch framework, first explaining the main concepts behind it and then showing how businesses can use it to support key business scenarios. Next, the chapter walks through the process of creating and executing a batch job in the context of the Batch framework tools, and describes functionality that helps you manage batch jobs after they are scheduled. The process for configuring a batch server is also briefly described, and the chapter concludes with some tips for debugging a batch task.
Note
The intention of this chapter is to give you a solid overview of the concepts, tools, and processes in the new Batch framework. For in-depth procedures and more information, refer to the Microsoft Dynamics AX Server and Database Administration Guide available from http://www.microsoft.com/downloads/en/default.aspx. You can also find a wealth of batch-specific information in the Microsoft Dynamics 2009 software development kit (SDK).
In batch processing, you create batch jobs to organize "batchable" tasks for processing, schedule and define the conditions under which the batch tasks are executed, add the tasks to a queue, and set them to run automatically on a batch server. After the execution is completed, the batch server logs the errors and sends alerts. For example, a batch job might involve printing reports, closing inventory, performing maintenance, or sending electronic documents using the Application Integration Framework (AIF)—at a specified time on a specified computer. When you use a batch job to process these types of resource-intensive tasks, you avoid slowing down the user’s computer or the server during working hours.
In the following sections, we explain how each of these batch processing concepts are put into action in Dynamics AX.
Table 16-1 describes how standard batch processing concepts are represented in Dynamics AX. We provide more information about each of these concepts later in this chapter.
Table 16-1. Batch Processing Concepts in Dynamics 2009
Batch Concept | In Dynamics AX 2009 |
|---|---|
Batch framework | An asynchronous server-based batch execution environment. The environment is capable of executing multiple batch tasks within a batch job, in parallel, across multiple instances of the Application Object Server (AOS). The Batch framework gives developers and administrators the flexibility to control the schedule and the order in which tasks are executed. For example, an administrator can set up a job to process invoices automatically at the end of every month on a specific AOS, and set up a different sequence of tasks depending on whether an earlier task succeeds or fails. The Batch framework also provides transaction control, logging, and alerting mechanisms. |
Batch task | A batch-enabled class. A class must be enabled for batch processing before it can run as a batch task, and any class that is batch-enabled can execute as a batch task. The section "Batch-Enabling a Class" later in this chapter provides more information on this topic. The Dynamics AX classes that are used for batch tasks are designated to run on either the client or the server. Tasks that run on the server can run automatically as part of a batch job, regardless of whether a client is open. However, tasks that run on the client must be run manually by using the Batch processing form. |
Batch job | A complete process that achieves a goal, such as printing a report or performing the inventory closing process. A batch job is made up of one or more tasks. |
Batch group | An attribute of a batch task that allows the administrator to determine which AOS runs the job. Batch groups are used to direct batch tasks to specific servers. If a new task is created, by default it is assigned to an "empty" batch group, which is a valid batch group. |
Batch server | An instance of the AOS that processes batch jobs. Read more about the AOS in Chapter 1, and about configuring an AOS to be a batch server later in this chapter. |
As a system administrator or developer, you can use the new server-based Batch framework to create larger, more complex batch jobs than before. You can also monitor the status of batch jobs and control which AOS runs the batch job and at what time. The following list includes specific tasks you can do with the new Batch framework capabilities in Dynamics AX 2009:
Run server-based batch jobs using the security credentials of the user who created the job. Running the batch job under the credentials of the user who created the job provides the benefits of security isolation.
Create dependencies between batch tasks, and control the order in which the tasks are executed. The tasks in a batch job can run sequentially or simultaneously. You can set up a different sequence of tasks, depending on whether an earlier task succeeds or fails.
Set the time at which the batch job is executed, and specify which AOS executes the batch job. The servers in your environment might have different software installed or might be available at different times of the day. You can select the most appropriate server on which to run a batch job using the Batch Group attribute.
Create recurrence patterns for batch jobs. Some batch jobs need to be run on a recurring basis. For example, your company might need to process invoices automatically at the end of every month.
Monitor the batch job. You can ask for an alert to be sent when a batch job succeeds or fails, or just when it finishes.
Check the status, cancel, or review the history of the batch job. After a batch job has been processed, you can view its history, including any messages encountered while running the job.
We cover these capabilities in more detail throughout the rest of this chapter.
Along with more complex and larger batch jobs comes the need for performance enhancements to the Batch framework. The Dynamics AX 2009 Batch framework provides parallel execution to improve performance for long-running processes.
Microsoft developers use the Batch framework as a foundation for many performance-critical processes, such as maximizing hardware scalability during data upgrade and maximizing posting throughput during journal posting. The white paper "Journal Batch Posting," available at http://www.microsoft.com/downloads/en/default.aspx, describes many of the ways in which Microsoft uses the Batch framework.
Because it is a server-based component, the Batch framework enables you to design multithreading server processes in a controlled manner. By configuring the number of parallel execution threads, defining the set and order of tasks for processing, and setting the execution schedule, you can achieve greater scalability across your hardware. In addition, your batch tasks are running securely on a server, so you can process them without having a client open.
Businesses typically want to create batch jobs to address the following kinds of needs:
To enable scheduling flexibility. The Batch framework can perform periodic data clean-up jobs or run invoice processing on a regular schedule. To run invoice processing at the end of every month, for example, you could set up a recurring batch job that runs at midnight on the last working day of each month and selects the invoice processing job from the list of available jobs. You could then save the schedule (along with its batch job), and the system automatically picks up the job at the specified time and executes it without having to open the client.
To control task execution order. With the Batch framework, you can develop a workflow or perform a complex data upgrade in an order you choose. When creating a new batch job and adding tasks to the job, you can set up dependencies between the jobs and create a dependency tree that ensures that certain tasks run in sequence while others run in parallel.
To enable different processing if a particular task succeeds or fails. Task processing based on decision trees can help you implement a reliable way of processing data. The Batch framework allows developers or system administrators to set up dependencies between tasks such that different tasks are executed depending on whether a particular task succeeds or fails. (Figure 16-4, later in the chapter, shows an example of a dependency tree.) The system administrator or developer can also set up alerts to be notified if a job fails for any reason.
To improve performance using multithreading. Multithreading ensures that your processor’s capabilities are fully utilized, which is particularly important for long-running processes like inventory closing. You can gain additional performance enhancements by breaking a process into separate subjobs and executing them against different AOS instances, increasing throughput and reducing overall execution time.
To gain advanced logging capabilities. The Batch framework lets you see errors or exceptions thrown during the last run of the batch and to see how long a process takes to execute. Advanced logging capabilities are also useful for performance benchmarking and security auditing.
As mentioned earlier in the chapter, each batch task is a batch-enabled class, and a class must be enabled for batch processing before it can be scheduled for execution as a batch task. The classes used for batch tasks are designated to run on either the client or the server. Tasks that run on the server can run automatically as part of a batch job, regardless of whether a client is open. Tasks that run on the client, however, must be run manually via the Batch processing form, which is in the Dynamics AX basic module, at PeriodicBatchProcessing.
Many classes included with Dynamics AX 2009 are already enabled for batch processing. You can also design a class that you create for execution within the Batch framework, as shown in the following example.
To allow your class to run as a batch task, you have to extend it from the RunBaseBatch class.
The RunBaseBatch class itself is an extension of the RunBase framework, so your batch class must adhere to the patterns and guidelines of the RunBase-extended classes. (For more information, refer to the "Wizard Framework Extension" section in Chapter 6.)
To convert a class so that it’s batchable, you need to use the pack-unpack pattern and implement the methods pack and unpack to allow a class to be serialized. When a batch task gets created, its member variables are saved in a container using the pack method and stored in the Batch table. Later, when the batch server picks up the task for execution, it restores class member variables from the container using the unpack method, so it’s important to provide a correct list of variables necessary for class execution. If any member variable isn’t packable, the class can’t be packed and reinstantiated to the same state.
Following is an example of the pack and unpack methods.
The macros #CurrentList and #CurrentVersion referenced in the preceding code must be defined in the class declaration. #CurrentList is a macro holding a list of the class member variables to pack, as shown here.
You must implement the core logic of your batch class in the run method. If your batch class is designed to be executed on the server, there are some limitations on the operations you can use in the run method. For example, you can’t call any client logic or dialogs. However, you can still use Infolog classes—all Infolog and exception error messages are captured during batch class execution and stored in the Batch table.
You can also implement some RunBaseBatch-specific methods in your class to control its behavior as a batch task:
runsImpersonated. Determines whether the batch task is run on the server or on a client. The base method always returns true, which means that the batch must run under the authority of the person who scheduled the batch and that no client session is involved.
canGoBatchJournal. Determines whether the batch task class appears in the list of available classes when you create a new batch task using the Batch task form. See the "Add a Task to a Batch Job" section later in this chapter for an example using canGoBatchJournal.
You can create a batch job in three different ways in Dynamics AX: from the dialog box of a batch-enabled class, using the Batch Job designer form, or using the Batch Developer API.
The technique you use depends on the size and complexity of your batch job. A simple batch job consisting of a single task with no dependencies on other tasks would typically be created from the dialog box of a batch-enabled class; a more complex batch job composed of several batch tasks that might have dependencies is created using the Batch Job form; and highly complex and very large batch jobs, or those that need to be integrated with other business logic, are created from the Batch Developer API. In the following subsections, we provide an example in each category.
The simplest way to execute a class as a batch job is to select the Batch Processing check box on the Batch tab of your class’s dialog box, as shown for the Calendar Cleanup class dialog box in Figure 16-1. If you enable that option and click OK, the class is saved as a new batch job with one task. The batch job doesn’t run immediately; instead, it runs at the date and time you set, and with the frequency you indicate, in the Recurrence form.
Tip
You can remove recurrence from a batch job by clicking Functions and then Remove Recurrence on the Batch Job form.
Two other controls in the dialog box are of particular interest: the Batch Group attribute and the Alerts button.
Batch Group. This batch-task attribute allows an administrator to determine which AOS runs the task, how many tasks an AOS can run in parallel, and on which servers a task can be executed.
Alerts. This functionality sends an alert when a batch job ends successfully, fails, or is canceled. Alerts are particularly helpful for monitoring jobs that run on the server outside of business hours. You can set up alerts for the job by clicking the Alerts button, choosing the conditions for the alert, and saving your choices. The system stores the alert rule and triggers it when the conditions are met.
You use the Batch Job form for jobs that include many tasks that might have a complex set of dependencies. You can use this form to create a batch job, add tasks to it, and set up dependencies between tasks. The Batch Job form is accessed through the Dynamics AX basic module: InquiriesBatch Job.
Creating a batch job from the Batch Job form is straightforward. Press Ctrl+N to create a new batch job, and then enter the details for the job: a description and the date and time at which you want the job to start.
You can also set up Recurrence for the batch job by clicking the Recurrence button and entering a range and pattern for the recurrence. Make sure to save the batch job by pressing Ctrl+S.
After you create the batch job, you add tasks to it and any dependencies the tasks have.
Figure 16-2 shows the Batch Job form.
After you create a batch job, you can start adding tasks to it and creating dependencies between them by using the Batch Tasks form. The Batch Tasks form opens when you click the View Tasks button on the Batch Job form.
Note
You can open the Batch Tasks form from several places. For example, you can open it by clicking Batch Jobs in the main menu or by choosing Batch Job List—User. Both menu items open the same form, but information presented to the user in the Batch Tasks form differs depending on which menu item you use to open it.
Depending on how you open the Batch Tasks form and your level of access, you can view either the batch jobs that you have created or all batch jobs that have been scheduled in the system.
From the Batch Tasks form, you can also change the status of batch tasks or delete tasks that are no longer needed. Press Ctrl+N to create a new task, and then specify the data associated with that task: enter a description for the task in the Task Description field, select the company in which the task runs in the Company Accounts field, and in the Class Name field, select the process you want the task to run—classes appear in the list (a look-up list that appears, containing all available classes) only if the property CanGoBatchJournal is enabled.
If necessary, select a batch group for the task, and then make sure you save the task by pressing Ctrl+S.
Note
Some classes are designated to run on the client. You can add client tasks to a batch job. However, you must run client tasks manually by using the Set Up Batch Processing form, available in the Dynamics AX basic module: PeriodicBatchProcessing. In addition, client tasks must belong to a batch group.
Figure 16-3 shows the Batch Tasks designer form.
As we’ve said, each batch task represents a batch-enabled class. Sometimes you need to set up parameters for that class. For example, you might need to specify posting parameters for invoice posting. To do that, click the Parameters button in the Batch Tasks form. A dialog box specific to the selected class is displayed. After you specify the necessary parameters and click OK, the class parameters are packed and saved in the Batch table and are restored during class execution.
After you’ve created the batch job and added tasks to it, you can use the Batch Tasks designer form to define dependencies between the tasks. If no dependencies or conditions are defined within a job, the batch server schedules all tasks in parallel, up to a maximum you configure with the parameter Maximum Batch Threads in the Server Configuration form.
If you need to use advanced sequencing for batch tasks to accommodate your business process flow requirements, you can use the Batch Task designer or Batch Developer API, both of which are included in the Batch framework. You can use these tools to construct complex dependency trees, which allow you to schedule batch jobs tasks in parallel, add multiple dependencies between jobs tasks, choose different execution paths based on the results of the previous job task, and so on.
To control execution order, you can make a task dependent on another task in the same job, and set up conditions for the execution of both tasks.
For example, consider the job JOB1, with seven tasks: TASK1, TASK2, TASK3, TASK4, TASK5, TASK6, and TASK7:
TASK1 is the first task.
TASK2 runs ON COMPLETION ("Ended or Error") of TASK1 (regardless of the success or failure of TASK1).
TASK3 runs ON SUCCESS ("Ended") of TASK2.
TASK4 runs ON SUCCESS ("Ended") of TASK2.
TASK5 runs ON FAILURE ("Error") of TASK2.
TASK6 runs ON FAILURE ("Error") of TASK3.
TASK7 runs ON SUCCESS ("Ended") of both TASK3 and TASK4.
Figure 16-4 shows the dependency tree for JOB1.
To define these task dependencies and to tell the system how to handle them, select a child task (for example, TASK2 from the preceding list), click the Has Conditions grid in the Batch Tasks form, and follow these steps:
Press Ctrl+N to create a new condition.
Select the task ID of the parent task, such as TASK1.
Select the expected status that the parent task must reach before the dependent task can run. For example, TASK2 starts when TASK1 becomes "Ended or Error."
Press Ctrl+S to save the condition.
If you’ve entered more than one condition and if all conditions must be met before the dependent task can run, select a condition type of All.
Select a condition type of Any if the dependent task can run after any of the conditions are met.
You can also choose how the system handles task failures in the Batch Tasks designer form. To ignore the failure of a specific task, select Ignore Task Failure for that task. If this option is selected, the failure of the task doesn’t cause the job to fail. You can also use the Maximum Retries field to specify the number of times a task should be retried before it is considered to have failed.
For advanced scenarios requiring complex or large batch jobs, such as inventory closing or data upgrades, the batch server framework provides an X++ API that enables developers to create or modify batch jobs, tasks, and their dependencies as needed as well as to dynamically create run-time batch tasks on the fly. This flexible API helps automate task creation and integrate batch processing into other business processes. It can also be useful when your batch job or task requires some additional logic.
First, you have to create a Batch Job instance class named BatchHeader.
Note
When you construct an instance of the BatchHeader class, no batch jobs are physically created in the system yet. All operations are performed in memory until you call the batchHeader.save function, which saves your changes in the database.
You can also construct a BatchHeader object for an existing BatchJob by providing an optional batchJobId parameter to the construct method, as shown here.
The BatchHeader class allows you to access and modify most BatchJob parameters using parm methods. For example, you can set up recurrence and alerts for your batch job as shown in the following example.
After you create a batch job using the Batch API, you can add tasks to it by calling the addTask method. The first parameter for this method is an instance of a batch-enabled class that will be scheduled for execution as a batch task.
Another way to create a task is to use the method addRuntimeTask, which creates a dynamic batch task. It exists only for the current run, and it gets copied into the history tables and deleted at the end of the run. It copies settings such as batch group and child dependencies from the inheritFromTaskId task.
After you add tasks, you can use the Batch API to specify any dependencies between them. The BatchHeader class provides the method addDependency, which you can use to define a dependency between the tasks batchTaskToRun and dependsOnBatchTask.
The optional parameter batchStatus allows you to specify the type of the dependency. By default a dependency of type BatchDependencyStatus::Finished is created, which means that the task starts execution only if the task which it depends on finishes successfully. Other allowed options are BatchDependencyStatus::Error (the task will start execution if the preceding task finished with an error) and BatchDependencyStatus::FinishedOrError (the preceding task finished processing with any status result).
After you have finished defining the batch job, it’s important to call the batchHeader.save method. Under the hood, the save method inserts records into the BatchJob, Batch, and BatchConstraints tables where the batch server can automatically pick them up for execution. However, you must run client tasks manually by using the Set Up Batch Processing form (from the Dynamics AX basic module, click PeriodicBatchProcessing).
The following example shows how you can create a batch job and two batch tasks using the Batch API.
For more examples on programmatic batch job creation, you can refer to the Microsoft Dynamics AX 2009 SDK section "Walkthrough: Extending RunBaseBatch Class to Create and Run a Batch" at http://msdn.microsoft.com/en-us/library/cc636647.aspx.
Before a batch job can be executed on an AOS, you must configure the AOS as a batch server and set up the batch groups that tell the system which AOS needs to execute the job. In addition to these initial configuration duties, you’ll likely need to manage the batch tasks and jobs: checking status, reviewing history, and sometimes canceling a batch job. You’ll probably need to debug a batch task at some point as well. In the following section, we describe how to configure an AOS as a batch server, set up batch groups, manage batch jobs, and debug a batch task.
You can configure an AOS to be a batch server, including specifying when the batch server is available for processing and how many tasks it can run, using the Server Configuration form. The Server Configuration form is in AdministrationSetupServer Configuration. Note that the first AOS is automatically designated as a batch server. Figure 16-5 shows the Server Configuration form.
Tip
Use multiple batch servers to increase throughput and reduce the amount of time it takes to run batches.
On the Overview tab, select a server.
Select Is Batch Server to enable batch processing on the server, as shown in Figure 16-5.
On the Batch Server Schedule tab, enter the maximum number of batch tasks that can be run on the AOS instance at one time. The server continues to pick up tasks from the queue until it reaches its maximum.
Enter a starting time in the Start Time field and an ending time in the End Time field to schedule the batch processing. Press Ctrl+N to enter an additional time period.
Tip
It’s a good idea to exclude a server from batch processing when it is busy with regular transaction processing. You can set server schedules so that each AOS is available for user traffic during the day and for batch traffic overnight. Keep in mind that if the server is running a task when its batch processing availability ends, the task continues running to completion. However, the server doesn’t pick up any more tasks from the queue.
On the Batch Server Groups tab, use the arrow buttons to specify the batch groups that can run on the selected server.
A batch group is an attribute of a batch task that allows a user (typically a system administrator) to determine which AOS runs the batch job. In this section, we show you how to create a batch group so that it can be selected from the Batch Server Groups tab in the Server Configuration form. You create batch groups on the Batch Group form, which is in AdministrationSetupBatch Group.
Note
For client tasks to be run manually from the Set Up Batch Processing form, they must be assigned to batch groups.
To create a batch group, press Ctrl+N in the Batch Group form, and then type a name and description for the batch group. The Batch Group form is shown in Figure 16-6.
After you create a batch group, you have to set it up to run server batch tasks on a specific server. Select the appropriate batch group in the Batch Groups form, and then follow these steps:
Click the Batch Servers tab. The Selected Servers list displays the AOS instances that the selected group is specified to run on. The Remaining Servers list displays the remaining AOS instances that are available as batch servers.
Click the arrow buttons to select additional servers or remove servers from the Selected Servers list.
Once you’ve created and scheduled a batch job, you might want to check its status, review its history, or even cancel it. To verify the status of batch jobs or to cancel them, use the Batch Job form, which is at BasicInquiriesBatch Job.
If jobs are in progress, you can view their percentage complete. You can also see whether any jobs are scheduled to start soon.
You can view the status of all batch jobs in the system or only the jobs that you ordered. You can change the status of batch jobs by selecting the batch job from the list and then following these steps:
When an AOS restarts and is configured to run batch jobs, it immediately attempts to recover from previous runs if it finds batch tasks in the Executing state. These tasks go back to the status Ready, and their Actual Retries field, in the Update tab in the Batch Tasks form, increases by one.
You use Maximum Retries to configure how many times an AOS can rerun a batch task that has failed because of infrastructure problems, such as a power outage or a server crash. By default, this value is set to 1; you can change it in the General tab of the Batch Tasks form. When the Actual Retries field in the Update tab exceeds the maximum number of retries, the AOS fails the batch task instead of attempting to run it again. If this happens, the recurrence of the batch job is not honored, and the status of the batch job is set to either Success or Error.
You can cancel a batch job by changing its status to Canceling in the Batch Job form. If the job is currently executing, the tasks in the Waiting or Ready state are changed to Not Run, and currently executing tasks are interrupted and their status changed to Canceled.
After a batch job has finished running, you can view its history, including any messages encountered while running the batch job, in the Batch Job History form, which is in the Dynamics AX basic module: InquiriesBatch Job History.
You can also view the logs for each batch job in the Batch Job History form.
To view log information for the whole batch, select a batch job and then click Log.
To view log information for individual tasks, select a batch job and then click View Tasks. In the Batch History List form, select a task and click Log.
Because server batch tasks run in noninteractive mode, to debug a server batch task, you have to perform additional steps to configure your Dynamics AX server and client as well as set up the usual debug breakpoint steps.
Note
Running in noninteractive mode means that a user can’t interact directly with a batch task while it’s running. So you can’t, for example, present a dialog box to gather user input once a batch task has started executing on a server.
You first configure the AOS for batch debugging, and then you configure the Dynamics AX client for batch debugging. For both tasks, you use the Server Configuration Utility, which you access through All ProgramsAdministrative ToolsMicrosoft Dynamics AX Server Configuration. For configuration specifics, consult the Microsoft Dynamics AX 2009 SDK, available on MSDN.
You can use the Dynamics AX Server Configuration Utility to configure the AOS for batch debugging. The utility is available on the machine where you installed your AOS, from All ProgramsAdministrative ToolsMicrosoft Dynamics AX 2009 Server Configuration.
In the Server Configuration Utility, select the check boxes for the following:
Enable Breakpoints To Debug X++ Code Running On This Server
Enable Global Breakpoints To Debug X++ Code Running In Batch Jobs
Click OK to close the utility, and then restart the AOS.
Figure 16-7 shows the Dynamics AX 2009 Server Configuration Utility.
To configure the client for batch debugging, you use the Microsoft Dynamics AX 2009 Configuration Utility. (This utility is different than the Server Configuration Utility.) Figure 16-8 shows the Microsoft Dynamics AX Configuration Utility.
The Configuration Utility is available from All ProgramsAdministrative ToolsMicrosoft Dynamics AX 2009Configuration. (The line might not specify server versus client.)
In Windows Explorer, locate the ClientBinDirectory of your Dynamics AX installation. Double-click the AxDebug.exe program to run it. Now you’re ready to debug batch jobs running on the server.
Note
For more information about debugging server-side batch jobs, refer to the Microsoft Dynamics AX 2009 SDK section "Walkthrough: Debugging a Batch that Runs on the AOS" at http://msdn.microsoft.com/en-us/library/cc588679.aspx.