You need a text file containing sentences, for example:

This is a sample text.
Another text with special characters, , , END OF FILE
hello,man
I wish you a happy new year:2011
The,last.but,not;the.least

You can download the sample file from the book's website.

Carry out the following steps:

The first step in the recipe has the task of reading the text file. You have used ##### as separator characters because that string is not present in the file. This assures you that the field line will contain the entire line of your file.

Now, you need to develop some custom code that counts the words in the line field. This task can't be accomplished using standard Kettle steps, so you have programmed the necessary functionality in Java code inside a UDJC step.

Let's explore the dialog for the UDJC step, which is shown in the following screenshot:

Most of the window is occupied by the editing area. Here you write the Java code using the standard syntax of that language. On the left, there is a panel with many code fragments ready to use (Code Snippets), and a section with sets and gets for the input and output fields. To add a code fragment to your script either double-click on it, drag it to the location in your script where you wish to use it, or just type it in the editing area.

The Input and Outputs fields appear automatically in the tree when the Java code compiles correctly; you can use the Test class button to verify that the code is properly written. If an error occurs, then you will see an error window otherwise, you will be able to preview the results of the transformation step.

The Fields tab on the bottom is to declare the new fields added by the step. In this case, you declared the field qty of Integer type. This field will be the output variable that will return the word count.

Under the Target steps tab, you can declare the steps where the rows will be redirected. In the recipe, you pointed to the two target Dummy steps; one for the short sentences, and the other for the long ones.

Let's see the Java code in detail:

At the beginning, there is a section with the variable declarations:

private RowSet shortSentences = null;
private RowSet longSentences = null;

These variables will represent the two possible target steps. They are declared at the beginning because this way they keep their values between the row processing.

Then, you have the main function:

public boolean processRow(StepMetaInterface smi, StepDataInterface sdi) throws KettleException
{
Object[] r = getRow();
if (r == null) {
setOutputDone();
return false;
}

The processRow() function process a new row.

The getRow() function gets the next row from the input steps. It returns an object array with the incoming row. A null value means that there are no more rows for processing.

The following code only executes for the first row:

if (first) {
first = false;
shortSentences = findTargetRowSet("shortOnes");
longSentences = findTargetRowSet("longOnes");
}

You can use the flag first to prepare a proper environment before processing the rows. In this case, you set the target steps into two variables for further use.

The next code use the get() method to set the internal variable linein with the value of the line field.

r = createOutputRow(r, data.outputRowMeta.size());
String linein;
linein = get(Fields.In, "line").getString(r);

Here is the main cycle:

long len = linein.length();
long qty = 0;
boolean currentSpecialChar = false;
for (int i=0;i'len;i++) {
char ch = linein.charAt(i);
switch(ch) {
case ',':
case '.':
case ' ':
case ';':
case ':':
if (!currentSpecialChar) qty++;
currentSpecialChar = true;
break;
default:
currentSpecialChar = false;
break;
}
}

It parses the entire sentence looking for characters used as separators. If one of these separators is found, then it will increment the qty variable by one and set the flag currentSpecialChar to true, in order to not increment the value if the next character is also a separator.

The next line is to count the last word of the sentence only if it wasn't counted in the main cycle:

if (!currentSpecialChar) qty++;

Here we set the new field named qty with the value of the internal variable qty, which has the word count:

get(Fields.Out, "qty").setValue(r, qty);

Finally, if the word count is lower than 7 (arbitrary value), then the row will be passed to the short sentences step; otherwise the target will be the long sentences step.

if (qty ' 7) {
putRowTo(data.outputRowMeta, r, shortSentences);
}
else {
putRowTo(data.outputRowMeta, r, longSentences);
}
return true;
}

To learn about the syntax of the Java language, visit the following URL:

http://download.oracle.com/javase/tutorial/

As mentioned earlier, you can access the Kettle API from inside the UDJC code. To learn the details of the API, you should check the source. For instructions on getting the code, follow this link: http://community.pentaho.com/getthecode/.

Let's see some more information to take advantage of this very useful step:

long qty = getParameter("qtyParameter");

RowSet data = findInfoRowSet("charsList");
Object[] dataRow = null;
while((dataRow = getRowFrom(data)) != null){
//Do something
}

if (qty ' 10) logBasic("Long sentence found!");