Data Processing Methods

Basically, the functionality of every computer program comes down to processing data. Whether it is a simple calculator tool or a full-blown spreadsheet with Internet stock market interactivity, data is being provided and the program processes it and returns results. When considering the performance of a program, the data processing methods are thus of utmost importance. One way to introduce enhancements in this area is to optimize the processing of the data. However, the best optimization of processing data is not having to process data at all. The following sections discuss the moments when data processing occurs. Often, a choice can be made, hopefully in such a way that data is processed during idle program times.

Processing Immediately

The most intuitive form of processing occurs when data is processed as it becomes available. You do not expect your calculator tool to accept your calculation and send you an email with the answer the next morning. Unless the processing of data is expected to take a long time, most folks basically expect processing to occur immediately.

Some situations where it's a good idea to use immediate processing include

• The data rate is low enough for the program to handle it immediately. To put it another way, there is enough idle time for the program to do immediate processing (as with for instance software embedded in a train-ticket machine).

• The data rate is high, but the process the data has to undergo is simple enough to be executed immediately without causing unacceptable slowdown of the whole program, such as decompression software—MP3 players, for example. (The process does not necessarily have to be simple in logical terms; the hardware just has to be able to execute it in a reasonable amount of time.)

• The data rate is high, and the process is complex, but there is simply no other choice than to process the data immediately. Consecutive steps need the processed data to continue. This means placing data in waiting queues and simply processing until the job is done (such as is the case when copying a large number of files with a file manager).

Processing on Request

Processing on request is basically the opposite of processing immediately. Instead of processing the data as it comes in (input), processing occurs when the data is needed (output). This means no processing occurs other than that which is absolutely necessary. It also means that processing demands can become high when the demand for data becomes high. Still, it can sometimes be useful to postpone processing. The following advantages can be identified:

• No time is lost on processing data during input; maximum performance is thus available for the input function. This is important for programs which need to respond fast to capture available data, such as sampler software for audio (audio sampling), video, or medical data (CAT scan).

• Data that is never retrieved will not be processed. No time and memory is lost on results that are never used. Consider the downloading of compressed files. It is possible to decompress during downloading, but some files may, in fact, not be needed at all.

• When the processed data occupies more space (because of derived data), storing data before processing will use up less memory or storage. Think of tools that perform some kind of decompression (such as text file versus desktop publishing formatting, MP3 versus WAV, and so on).

Depending on the specific situation, the following disadvantages can, however, come into play:

• Timing problems may occur if the demand for processed data (output) suddenly becomes high. Data may not be instantaneously available because of the high workload. An administration program that needs to churn out paychecks at the end of the month will be very busy if all the salaries still need to be calculated.

• When the input size of the data is larger than that of the processed data, space (memory or storage) is, in effect, lost (for example, keeping audio in WAV files instead of in MP3 and so on).

• A mechanism may be needed to locate and remove irrelevant input data or allow new input to overwrite certain existing data (for example, purging a database of Christmas card recipients from the names of people who declined to send you a card for the last three years).

To make a viable decision, look closely at the characteristics of the data and its usage. In essence, processing time and speed are set against available resources.

Periodic Processing

Periodic processing is likely to be used when the results of processing are not needed immediately and the program or system has an uneven workload. Perhaps there are certain times when the data rate is low (lunch, evening, early morning), the processing is cheap (power and phone costs), or the processing is less intrusive (network communications and so on). Benefits of periodic processing are

• No extra processing during input.

• Processed data is generally available (though not immediately after being input).

• The processed input data and irrelevant data can easily be discarded.

But there can also be a downside:

• When the system needs to be active longer—for example, around the clock to catch up on processing data during the night—a high demand is placed on the availability of the system.

• Maintenance (including preventive maintenance) can be difficult to schedule when the system is active longer.

• The input data cannot be demanded instantly and limits the availability of processed data.

Batch Processing

As with periodic processing, batch processing takes advantage of the uneven workload of a system. This time, however, the processing is not necessarily triggered at a prespecified time—more likely, an event triggers the processing. This characteristic makes batch processing similar to processing on request, with the difference that whole batches or sets of data are processed.

Batch processing shares the pros of periodic processing and has the added advantage to be triggered outside the specified period(s).