C H A P T E R  3

Processing

by Enrique Ramos

This book is about making projects with Kinect and Arduino, and one of the most direct ways of interfacing them is through Processing, a Java-based, open source programming language and IDE. You should be familiar with Arduino by now, so you will be glad to know that the Processing IDE much resembles Arduino’s. In fact, the Arduino IDE was based on the Processing IDE (see Figure 3-1).

Figure 3-1. Processing and Arduino IDEs side by side

One of the best features of Processing is that, being built on Java, there is a vast community of people creating Processing libraries that you can use for your projects without having to worry (too much) about the scary, dark places hidden behind every class and method. Kinect is no exception to this, and thus Processing is able to talk to Kinect devices through several libraries available from its web site. Processing is also capable of talking to Arduino using serial communication.

This chapter will get you acquainted with the basics of the Processing language and Simple-OpenNI, one of Processing’s Kinect libraries. Throughout the rest of the book, you will be creating Kinect and Arduino projects that talk through Processing and Simple-OpenNI, so the understanding of these techniques is essential to the development of any of the upcoming projects.

Processing Language

Casey Reas and Ben Fry, both formerly of the Aesthetics and Computation Group at the MIT Media Lab, created Processing in 2001 while studying under John Maeda. One of the stated aims of Processing is to act as a tool to get non-programmers started with programming. On the Processing web site (http://www.processing.org), you can read this introduction:

Processing is an open source programming language and environment for people who want to create images, animations, and interactions. Initially developed to serve as a software sketchbook and to teach fundamentals of computer programming within a visual context, Processing also has evolved into a tool for generating finished professional work. Today, there are tens of thousands of students, artists, designers, researchers, and hobbyists who use Processing for learning, prototyping, and production.

Processing is based in Java, one of the most widespread programming languages available today. Java is an object-oriented, multi-platform programming language. The code you write in Java is compiled into something called bytecode that is then executed by the Java Virtual Machine living in your computer. This allows the programmer (you) to write software without having to worry about the operating system (OS) it will be run on. As you can guess, this is a huge advantage when you are trying to write software to be used in different machines.

Processing programs are called sketches because Processing was first designed as a tool for designers to quickly code ideas that would later be developed in other programming languages. As the project developed, though, Processing expanded its capabilities, and it is now used for many complex projects as well as a sketching tool.

We won’t have space in this book to discuss the ins and outs of the Processing language, but if you are familiar with Java, C++, or C#, you will have no problem following the code in this book. For a thorough introduction to Processing, I strongly recommend Ira Greenberg’s monumental Processing: Creative Coding and Computational Art.

Installing Processing

Installing Processing is definitely not much of a hassle. Go to http://www.processing.org and download the appropriate version for your OS of choice. If you are a Windows user and you aren’t sure if you should download the Windows or Windows without Java version, go for the Windows version. The other version will require you to install the Java Development Kit (JDK) separately. Once you have downloaded the .zip file, uncompress it; you should find a runnable file. If you run it, you should get a window like the one depicted in Figure 3-2.

Processing will work on any OS from wherever you store it. If you are a Mac OS X user, you might want to put it in your Applications folder. On Windows, Program Files would seem like a good place for a new program to live. On Linux, users tend to store their programs in /usr/bin or /usr/local, but you can create your own folder.

Figure 3-2. Processing IDE

Processing IDE

As mentioned, the Processing IDE is very similar to the Arduino IDE described in Chapter 2. There is a console at the bottom of the Processing IDE; a message area that will warn you about errors in the code and at runtime; a text editor where you will write your code; plus a tab manager and a toolbar at the top of the IDE.

The Processing toolbar has a distribution of buttons similar to Arduino, yet with some differences (see Figure 3-2). The New, Open, and Save buttons play the same roles as in Arduino. But instead of having Verify and Upload buttons at the left, there are Run and Stop buttons. You might have guessed what they are for, but just in case, the Run button will run your application and the Stop button will halt it. There is another button called Export PApplet; pressing it will convert the Processing code into Java code and then compile it as a Java Applet, which can be embedded in a web browser. This is the way you’ll find Processing Applets, or PApplets, displayed on the Internet.

The Standard button on the right side of the toolbar is used to change the mode from Standard to Android (this is for Android Application development, and we are not going to get anywhere near Android Apps in this book, so you can forget about it.)

A First Processing Sketch

Processing was designed for simplicity, and it allows you to create runnable programs with just a few lines of code. Your first Processing sketch will get you acquainted with the basic structure on a Processing program. But first, let’s get into some general programming concepts.

Processing Variables

Variables are symbolic names used to store information in a program. There are eight primitive data types in Java, and they are all supported in Processing: byte, short, int, long, float, double, boolean, and char. A detailed explanation of each one of these data types would be beyond the scope of this book, but we will state some general rules.

Integers (int) will be used to store positive and negative whole numbers (this means numbers without a decimal point, like 42, 0, and -56. An integer variable can store values ranging from -2,147,483,648 to 2,147,483,647 (inclusive). If you need to store larger numbers, you need to define your variable as a long.

Whenever you need more precision, you will be using floating-point numbers, or floats, which are numbers with a decimal place (23.12, 0.567, -234.63). The data type double works the same way as float, but it is more precise. In general, Processing encourages the use of floats over doubles because of the savings in memory and computational time.

The data type boolean stores two values: true and false. It is common to use booleans in control statements to determine the flow of the program.

The character data type, or char, stores typographic symbols (a, U, $). You will use this data type when you want to display or work with text.

Even though String is not a proper Java primitive, we are going to discuss it here, as you will be using it to store sequences of characters. String is actually a class (more about this in Chapter 4), with methods for examining individual characters, comparing and searching strings, and a series of other tools that you will use to work with text and symbols in your programs.

Variable Scope

Whenever you declare a variable, you are implicitly setting a scope, or realm of validity for the variable. If you declare the variable within a function, you will be able to access it only from within that specific function. If you define the variable outside of any function, the variable will be set as a global variable and every function in the program will be allowed to “see” and interact with the new variable.

Structure of a Processing Sketch

Open your Processing IDE and declare a timer integer variable outside of any function. As mentioned, variables defined like this are treated as global variables in Processing, so you can call them from any method and subclass.

setup() Function

Next, include a setup() function. This function is called once in the lifetime of a program (unless you call it explicitly from another function). For this project, you are only going to specify the size of your sketch within this function.

You can include comments in your code as a means to remind yourself—or inform anybody else reading your code—about specific details of the code. There are two main types of comments: single line comments, which start with two forward slash characters, and multi-line comments, used for large descriptions of the code. You can include some comments in your setup() function in the following way:

There is a third kind of comment, the doc comment (/** comment */), which can be used to automatically generate Javadocs for your code. You won’t be using this sort of comment in this book, but they can be extremely useful when writing programs intended to be used by other people.

draw() Function

Next, you need to include a draw() function that will run as a loop until the program is terminated by the user. You are going to draw a circle on screen that will move to the right of the screen as time passes. Remember to include a background() function to clear the frame with a color at every iteration. If you comment it out (temporarily remove it by adding // to the beginning of the line), you will see that the circles are drawn on top of the previous ones.

Now you are ready to run your sketch. Press the Run button at the top-left of your Processing IDE (marked with a red arrow in Figure 3-3) and you will see a new window open with a white circle running to the right side.

Well, that was your first Processing sketch! I take it you are much impressed. You’re going to rush into a much deeper discussion on vector classes and 3D processing in the next few pages, so if you are completely new to Processing, you should have one of the introductory books to Processing at hand.

Figure 3-3. Your first Processing sketch

Processing Libraries

Processing libraries are lines of Java code that have been packaged into a .jar file and placed in the libraries folder in your system so you can call its functions from any Processing sketch. There are two sorts of libraries: core libraries, like OpenGL and Serial, which are included with Processing when you download it, and contributed libraries, which are created and maintained by members of the Processing community. Contributed libraries need to be downloaded and stored in the libraries folder in your system. If you are a Mac OS X user and you have saved the Processing application in the Applications folder, the path will be the following:

To open your Processing application on Mac OS X (it is essentially a folder, so to find the libraries folder), right-click the icon and select “Show Package Contents” from the pop-up menu.

If you are working on a Windows machine and you have put the Processing folder in the Program Files folder, this will be your path:

If you are a Linux user, wherever you save your Processing folder, the path to the libraries folder will be the following:

At the time of writing, there are three Kinect libraries available for Processing (shown in Table 3-1). Two of them are OS-specific, and one works on Mac OS X, Linux, and Windows. They also differ in the functions they implement.

The previous chapter covered how Kinect was developed and why there are several ways to access the Kinect data stream depending on the drivers and middleware you choose. If you look at the list of Kinect libraries for Processing in Table 3-1, you will realize that the two of them are based on OpenKinect and available only for one operating system. In this book, you are going to focus on Max Rheiner’s Simple-OpenNI because it is multi-platform and built upon Primesense’s OpenNI/NITE middleware, which means you can perform skeletonizing and hand recognition off the shelf. If you are doing projects where you only need the point cloud, you can use one of the other two libraries, which, as an advantage, provide access to the motor built inside your Kinect device.

Simple-OpenNI

Simple-OpenNI is a NITE and OpenNI wrapper for Processing implemented by Max Rheiner, a media artist and software developer currently lecturing at Zurich University.

What does this mean? Well, NITE and OpenNI are implemented in C++, and Rheiner has wrapped all the functions and methods in such a way that they are accessible from Java; then he wrote a library to make the Java code available from Processing. This allows us, Processing lovers, to have all the functionality of OpenNI and NITE available from our favorite programming language.

Installing Simple-OpenNI

You learned how to install a Processing library in the previous section, but there are some specific prerequisites for this library to work on your computer. After you have installed the prerequisites described next, you can go to the download web site at http://code.google.com/p/simple-openni/downloads/list to download the library and install it in your Processing libraries folder.

Installing Simple-OpenNI requires the previous installation of OpenNI and NITE on your machine. There is a very detailed description on how to do this on the Simple-OpenNI web site at http://code.google.com/p/simple-openni/, which we have reproduced here for you to follow.

Installation on Windows

Follow these steps if you are running Simple-OpenNI on a Windows machine:

• Download and install OpenNI.

http://www.openni.org/downloadfiles/opennimodules/openni-binaries/20-latest-unstable

• Download and install NITE.

http://www.openni.org/downloadfiles/opennimodules/openni-compliant-middleware-binaries/33-latest-unstable

The installer will ask you for the key; use this one:
0KOIk2JeIBYClPWVnMoRKn5cdY4=

Install the Primesense driver (Asus Xtion Pro, etc.).

• Download and install Primesensor.

http://www.openni.org/downloadfiles/opennimodules/openni-compliant-hardware-binaries/31-latest-unstable

Install the Kinect driver.

• Download SensorKinect, unzip it, and open the file at /avin2/SensorKit/SensorKinect-Win-OpenSource32-?.?.?.msi

https://github.com/avin2/SensorKinect

If everything worked out, you should see the camera in your Device Manager (under Primesensor: Kinect Camera, Kinect Motor). You can now download and install the Simple-OpenNI Processing library.

Installation in Mac OS X

If you are working on Mac OSX, this is your lucky day! Max Rheiner made an installer that goes through the whole installation process automatically. This should at least work with Mac OS X 10.6.

Install OpenNI the Short Way (OpenNI v1.1.0.41 and NITE v1.3.1.5)

• Download the Installer.

http://code.google.com/p/simple-openni/downloads/detail?name=OpenNI_NITE_Installer-OSX-0.20.zip&;can=2&q=#makechanges

• Unzip the file and open a terminal window.
• Go to the unzipped folder, OpenNI_NITE_Installer-OSX.
  > cd ./OpenNI_NITE_Installer-OSX
• Start the installer.
  > sudo ./install.sh

If this installation didn’t work for you, try the following, longer method.

Install OpenNI the Official Way

Those programs have to be on your system in order to install OpenNI:

• Xcode

http://developer.apple.com/xcode/

• MacPorts

http://distfiles.macports.org/MacPorts/MacPorts-1.9.2-10.6-SnowLeopard.dmg

• Java JDK (This will be already installed after you installed Processing)

When you have finished installing these programs, you can install OpenNI. Open the terminal and type in these commands:

You can now download and install the Simple-OpenNI Processing library.

Installation on Linux

If you are a Linux user, these are the steps to install Simple-OpenNI. First, download the install files (stable or unstable, 32-bit or 64-bit):

• OpenNI

http://openni.org/downloadfiles/2-openni-binaries

• Nite

http://openni.org/downloadfiles/12-openni-compliant-middleware-binaries

• Kinect drivers

https://github.com/avin2/SensorKinect

• Primesense drivers (Asus Xtion Pro, etc.)

http://openni.org/downloadfiles/30-openni-compliant-hardware-binaries

• Uncompress all files to a folder.
• Install OpenNI.
  > cd ./yourFolder/OpenNI-Bin-Linux64-v1.1.0.41
> sudo install
• Install NITE (the installer asks you for the key; enter the following: 0KOIk2JeIBYClPWVnMoRKn5cdY4=)
  > cd ./yourFolder/Nite-1.3.1.5
> sudo install
• Install the Kinect driver.

  Decompress ./yourFolder/avin2-SensorKinect-28738dc/SensorKinect-Bin-Linux64-v5.0.1.32.tar.bz2

  > cd ./yourFolder/SensorKinect-Bin-Linux64-v5.0.1.32
> sudo install
• Install the Primesense driver (only if you use an Asus Xtion or the Primesense Developer Kit).
  > cd ./yourFolder/Sensor-Bin-Linux64-v5.0.1.32
> sudo install

You can now download and install the Simple-OpenNI Processing library.

Accessing the Depth Map and RGB Image

Now that you have installed the Simple-OpenNI Processing library, let’s try a very simple example that will access the data streams from Kinect from Processing.

First, import the Simple-OpenNI library, and declare the variable kinect that will contain the Simple-OpenNI object, like so:

Within setup(), initialize the kinect object, passing this (your PApplet) as an argument. As you saw in Chapter 2, the Kinect has a standard RGB camera and an infrared camera on board, used in combination with an infrared projector to generate the depth image for 3D scanning. Enable the depth map and RGB images in your kinect object and the mirroring capability that will mirror the Kinect data so it’s easier for you to relate to your image on screen. (Try the same sketch without enabling mirroring, and you will see the difference.) Then, set your sketch size to the total size of the RGB and depth images placed side by side, so you can fit both in your frame.

The first thing you need to do in the draw loop is to update the kinect object so you will get the latest data from the Kinect device. Next, display the depth image and the RGB images on screen, as in Figure 3-4.

Figure 3-4. Depth map and RGB image

Alright, so in 14 lines of code you are accessing the RGB and depth images of the Kinect. If you want to access the infrared image, you can use enableIR instead of enableRGB, and then kinect.irWidth() instead of kinect.rgbWidth(), and kinect.irImage() for kinect.rgbImage().

Both images are 640x480 pixels. Well, that’s not very impressive for the RGB camera, just an average low-quality webcam. The magic comes from the depth map. You are actually getting a 640x480 pixel three-dimensional scan in real time (30fps). The levels of grey you see in Figure 3-4 represent the distances to the Kinect camera. So from this image, you are getting a two-dimensional image in which every pixel has embedded information of its distance to the camera. Could you reconstruct the spatial volume in the image? Yes, you can, and you’re going to do it. But let’s first make clear what we’re talking about when we talk about 3D.

The Third Dimension

This might sound less intriguing than talking about the fourth or the fifth dimension, but it is still a subject that provokes many a headache to anybody getting into the world of computer graphics, so let’s take a close look at what we mean when we talk about 3D.

We all live in a three-dimensional world, though our screens and human interfaces have traditionally been two-dimensional. With the advent of real-time 3D scanners like the Kinect, this limitation is somewhat left behind. We can interact with the computer using the three dimensions of physical space (or at least the three we are aware of, if you ask some physicists!).

Our screens are still 2D, though, so representing the three dimensions of space on the display requires some nasty math dealing with perspective and projections—which luckily we don’t have to work out by ourselves because somebody did for us already. Processing has full 3D support and two different 3D renderers, P3D and OpenGL. You’re going to focus on OpenGL because it is the more powerful of the two.

Processing in 3D

Let’s do a quick example of a Processing 3D sketch. There are two 3D primitives already implemented in Processing, box() and sphere(). You’re going to draw a three-dimensional box on screen.

For this, you need to import the OpenGL Processing core library into your sketch so you can set your renderer to OPENGL in the size() function within setup(). Then, in the draw() loop, set the background to white, move to the center of the screen, and draw a cube of 200 units each side (see Figure 3-5).

Figure 3-5. A cube in Processing

The function translate() that you have used in the previous example makes part of the Processing transform functions. These functions have in common the fact that they affect the current transformation matrix.

Matrix Transforms

The transformation matrix is the frame of reference that Processing uses to print objects on screen. If you don’t alter the transformation matrix, the origin of coordinates (the zero point to which all the coordinates refer) will be placed at the top-left corner of your screen, the x-axis pointing right, and the y-axis pointing down.

If you draw a rectangle on screen, you can specify the x and y coordinates of its center in the parameters because the function is rect(x, y, width, height). But if you want to draw a box, it has no position parameters, being just box(width, height, depth) or even just box(size).

So if you want to draw a box at a different position than the global origin of coordinates, you need to change the frame of reference to the place and orientation that you want the box to be in, so that when you draw the box, it will appear at the desired position. This will be performed with matrix transforms.

Look at the following code and see what it does on screen (shown in Figure 3-6):

Figure 3-6. Controlling a box with the mouse position

If you want to modify the transformation matrix for some operations and then return it back to where it was, you can use the methods pushMatrix() and popMatrix(). The first method will push the current matrix onto something called the matrix stack, from where you can recover it using popMatrix(). You don’t need to worry too much about the matrix stack at the moment; you just need to understand that you can save the current matrix with pushMatrix(), do some transformations and draw some objects, and then restore the previous matrix with popMatrix(), so you can continue drawing objects within the previous frame of reference. This is illustrated by the following code and shown in Figure 3-7:

Figure 3-7. Three boxes drawn on different reference systems

If you play with the previous applet for a while, you’ll realize that the two larger cubes follow the mouse and are rotated 45 degrees, while the intermediate cube remains static in its place. Analyzing the code, you will understand why. First, you pushed the matrix, and then you translated it to the mouse coordinates, rotated it by 45 degrees, and drew the largest box. Then you translated the matrix again on the x-axis, but because the matrix was already translated and rotated, it resulted in a diagonal movement, so the second cube is drawn rotated at the bottom right of the first.

Then, you popped the matrix, so you came back to the original reference system; you then translated the restored matrix by a certain distance in x and y, and drew the third box, which will not move at all with the mouse. The last transform doesn’t accumulate because every time you start a new draw() loop, the matrix gets reset.

For now, you will be drawing three-dimensional geometry on screen and looking at it from one point of view. But the best part of 3D geometry is that it can be seen from any point of view in space. This is what you are going to explore in the next section.

Camera Control Libraries

In order to visualize 3D space, all CAD packages make use of mouse-driven camera control capabilities, so you can orbit around your objects and get a better understanding of the geometry. As you’re dealing with 3D data acquisition with the Kinect, you are going to include a camera control library into your 3D sketches. At the time of writing, there are several camera control libraries on the Processing web site.

• Peasycam by Jonathan Feinberg
• Obsessive Camera Detection (OCD) by Kristian Linn Damkjer
• Proscene by Jean Pierre Charalambos

In this book, you are going to use the KinectOrbit library for your examples. This library is quite simple and somewhat rudimentary, but it is pretty straightforward to use. Also, it incorporates the possibility of saving viewpoints, so you don’t need to be orbiting around every time you open a sketch. Furthermore, it keeps the camera roll to zero to avoid the Kinect point cloud appearing upside-down or strangely rotated on screen. The camera in the library is especially set up to work with the Kinect point cloud, so you won’t need to worry about clipping planes, unit scales, direction of the axes, and other issues that can be off-putting when beginning to work in 3D.

You can download the KinectOrbit library from the book’s web site at http://www.arduinoandkinectprojects.com/kinectorbit. If you have a preference for another camera control library, or want to implement your own, feel free to substitute it for KinectOrbit in every project using it.

KinectOrbit Example

Let’s do a quick test with this library. After installing the KinectOrbit library following the directions in the “Processing Libraries” section, open a new sketch and import the KinectOrbit and OpenGL libraries. Then declare a KinectOrbit object and call it myOrbit.

Within setup(), specify the size of the sketch and the 3D renderer, and initialize the myOrbit object.

Within the draw() method, set a background color, push the myOrbit object, draw a box at the origin of the coordinate system with a side length of 500 units, and pop the orbit. After this, draw a square on screen using the Processing function rect() for comparison.

Figure 3-8. 3D View Controlled by KinectOrbit

Run the sketch and observe that the box appears to be in perspective from a defined viewpoint, while the rectangle has been drawn in screen coordinates (see Figure 3-8). The library is actually designed to work in a very similar way as the pushMatrix() and popMatrix() functions introduced earlier in this chapter. This means that your KinectOrbit object will allow you to control the perspective from which you are seeing everything drawn inside the pushOrbit/popOrbit loop, while anything drawn outside that loop will be drawn from the default coordinate system. This is why the rectangle is drawn on screen as if no orbit library were used.

You can control the point of view by using the mouse. The controls are the following:

• Right button drag: Pan the camera.
• Left Button drag: Orbit around the objects.
• Wheel: Zoom in and out.

If you like a view in particular, you can save it by pressing the P key, and then retrieve it by pressing the O key. When you start the sketch again, it will start in the latest viewpoint that you saved. The viewpoint parameters will be stored in a file named orbitSet_0.csv in your data folder. If you delete this file at some point, it will be created again when you run the sketch and set to the default values.

You can also control the gizmos and visualization style of your orbit object. For more information, refer to the Javadoc of KinectOrbit on the web site. Now let’s do a test with the Kinect in 3D.

Kinect Space

You have already accessed Kinect’s depth map in the “Accessing the Depth Map and RGB Image” section and wondered if you can get that map translated back into the real-scale 3D coordinates of the objects. The following sketch does that for you. It starts pretty much like the depth map example but includes a function that shows the real coordinates of the points. First, import the appropriate libraries, declare Simple-OpenNI and KinectOrbit objects, and initialize them in the setup() function, enabling the depth map only.

In the draw() function, include a pushOrbit/popOrbit loop; inside it, add the drawPointCloud() function and draw the Kinect frustum with the method drawCamFrustum(), already included in SimpleOpenNI.

Now, you need to implement the function drawPointCloud(), which draws each pixel from the depth map in its real-space coordinates. You first store the depth map into an array of integers. You realize that it is not a bi-dimensional array, but a linear array; Processing images are linear arrays of pixels. If you want to access a pixel with coordinates x and y, use the following code:

You are going to represent only a number of pixels in the array because the full 640x480 image gets pretty heavy for your processor when represented as 3D points. The integer steps is going to be used as a step size within the for() loops in order to skip some of the pixels.

Next, initialize a PVector that will contain the x, y, and z coordinates of the point you want to draw on screen and then run through the depth image pixels using two nested for loops.

If you want to see each point with the depth map color, include the following line:

Within the loop, you first find out the index associated with the x and y coordinates in the depth map. Then, if the depthMap point with the current index is not 0, call the function depthMapRealWorld()[index] to return the 3D coordinates of the current point, which you display with Processing’s point() function. (Note that all dimensions returned are in millimeters.)

Figure 3-9. Depth map image and depth map in real-world coordinates

Run the example and spend some time spinning around the model (see Figure 3-9). Have a look at the Kinect frustum in relation to the Orbit gizmo. The horizontal axis is X (red), the vertical axis is Y (green), and the depth axis is Z (blue), as you can see in Figure 3-10. The Kinect considers itself the origin of coordinates (a little pretentious, I would say), so all coordinates are a function of the Kinect’s position in space.

Figure 3-10. Kinect device and frustum plus the Orbit gizmo

Linear and Two-Dimensional Arrays

If you’re now wondering why you have been going from linear arrays to two-dimensional arrays and back, note that you could have done the deal with the following code:

But if you get closer to your point cloud, you can see that you have lost control of your “pixel dropping.” In the previous code, you were drawing one every steps rows of pixels and one every steps columns of pixels, so you were still drawing a consistent grid of points. With the linear array approach, you are drawing one every steps pixels on the linear array representing the image, whichever column and row that pixel is placed in. In this way, you aren’t producing a consistent grid. Working with two-dimensional arrays gives you more control over the way you parse and represent your Kinect point cloud, so you’ll continue to do it this way.

So you now have a 3D point cloud that represents the real world position of every point scanned by the Kinect. You will be doing many things with this information later in the book. For starters, as you also have an RGB camera installed on your Kinect device, could you associate the color image from the RGB camera to the point cloud to get a 3D picture of what the Kinect sees?

Coloring the Point Cloud

Coloring the point cloud should  be as straightforward as setting the color of each point as the color of the equivalent point in the RGB Image. This could be done by enabling the RGB image in the setup() and then changing the stroke color to the color of the RGB image at the same index. You should comment out the following line:

And you should add this line to the code after setting the value of index:

Your points are now colored according to the RGB camera, but to your surprise, the image and the 3D point cloud don’t match! This is easy to explain. The RGB camera and the infrared cameras can’t occupy the same spot in the Kinect, so they are close to each other but slightly separated. The images they are capturing are thus slightly off. In order to make them match, you need to call a Simple-OpenNI method to align the point cloud with the RGB image. The setup() function looks this:

Now you should get a pretty neat point cloud that displays the real colors of every pixel.

Figure 3-11. RGB image on non-calibrated point cloud (left) and calibrated point cloud (right)

If you do the same thing and render the results on screen, you will realize what’s happening. If you don’t add this method, the depth map image is different from your RGB image. It appears slightly bigger and displaced (have a look at Figure 3-11). If you add the method, you can observe in Figure 3-12 that the depth map has been modified as to be more similar to the RGB image.

Figure 3-12. Depth map calibrated to fit RGB image

NITE Functions

Up until now, everything you have been doing could have been done with any of the other two libraries based in OpenKinect. But you have seen a lot of cool applications on the Internet in which the programmers make use of hand and skeleton tracking to allow user interaction. Well, these functions are not included with OpenNI itself; they require an external module called NITE, which is not open source but is still free and publicly available, and which includes all the algorithms that Primesense has developed for user interaction.

Luckily, Max Rheiner has also wrapped nearly all of NITE functions within Simple-OpenNI, so you can access them from Processing. Let’s see how.

Hand Tracking

Because of our skillful use of our upper limbs, hand tracking is a very useful feature for natural interaction. We can very precisely position our hands in space, and we can perform a vast amount of gestures with them. These abilities can be used to drive parameters or trigger different behaviors in your natural interaction applications. OpenNI/NITE includes a framework for hand tracking and hand gesture recognition that you can use off the shelf. This is a great feature that doesn’t exist in the official Microsoft SDK, at least in the beta version.

To take advantage of this, you need to include a series of functions or methods in your main Processing sketch that will be called (the proper term is “invoked,” which sounds a little spooky) from the Simple-OpennNI library when certain events are triggered.

Let’s have a look at these methods and their application. The onCreateHands() method is called when a hand is detected and a hand object is initialized. You get an integer containing the hand ID, a PVector defining its position on recognition, and a float with the timestamp value. You can add more code here that will run when a hand is recognized.

The function onUpdateHands() is called every time Kinect updates the hand data. You get a new position vector and time for the hand. Add here the code you want to run when a hand is updated.

When a hand is destroyed (when it disappears from screen or is not recognizable any more), the function onDestroyHands() will be invoked.

NITE also calls two methods related to gesture recognition. The onRecognizeGesture() function is called when NITE recognizes a gesture. The string strGesture contains the name of the gesture recognized (Wave, Click, or RaiseHand). You also get the start and end positions of the gesture in two three-dimensional PVectors.

Finally, the function onProgressGesture() is triggered while the gesture is in progress but before it has been recognized. It gives you the percentage of completion of the gesture, along with the gesture type and the current position of the hand. This is normally a very short period of time, but it can be useful for some applications.

You will learn how to work practically with these techniques later in this book. In the next chapter, you will be using hand tracking to drive the brightness of a couple of LED lights; in Chapter 5, you will use it to create a gesture-based remote control. You will even control a robot using hand tracking in the last chapter of the book.

Skeleton Tracking

Skeleton tracking is probably the most impressive of NITE’s, and therefore of Kinect’s, capabilities. This framework allows the computer to understand a person’s body position in 3D and to have a quite accurate idea of where the person’s joints stand in space at every point in time. This is quite an overwhelming feature if you think that not so long ago you needed to buy a daunting amount of material including several cameras, special suits, and so on in order to acquire the same data.

The only downside of skeleton tracking (for certain projects) is the need for the user to stand in a certain “start pose” to allow the middleware to detect the user’s limbs and start the tracking (see Figure 3-13). This inconvenience seems to have been fixed in the current version of the official Microsoft SDK and will likely disappear from NITE, but for the time being, you’ll have to make do with it.

Figure 3-13. User in start pose, calibrated, and tracked

Let’s have a look at the callback functions that you need for skeleton tracking. You can add any code you want to all of these functions and it will run when the functions are called by Simple-OpenNI. In each of the functions, add a println() function to print out the user ID and some other messages to the console, so you know what methods are being triggered in runtime.

When a new user is recognized, the function onNewUser() is called. The user hasn’t been “skeletonized” yet, just detected, so you need to start the pose detection routine if you want to track the user’s limbs.

The function onLostUser() is invoked when Simple-OpenNI isn’t able to find the current user. This usually happens if the user goes off-screen for a certain amount of time.

When you recognize the user, you ask Simple-OpenNI to start looking for a starting pose. When this pose is detected, the method onStartPose() is called. At this moment, you can stop the pose detection routine and start the skeleton calibration.

If the user abandons the start pose before the calibration has finished, Simple-OpenNI calls the method onEndPose().

The method onStartCalibration() marks the beginning of the calibration process, which can take some seconds to perform.

Finally, if the user has stood long enough for the calibration to take place, the method onEndCalibration() is called, indicating the end of the calibration process. If the process was indeed successful, Simple-OpenNI starts tracking the skeleton; from that point on, you can abandon your pose and start moving. The newly created skeleton closely follows each one of your movements.

If the calibration process was not successful for some reason, Simple-OpenNI returns to the pose detection mode so you can restart the whole process.

As with hand tracking, you will be learning how to effectively work with skeleton tracking in this book using practical examples. In Chapter 6, you will be controlling a puppet with skeleton tracking; in Chapter 7, you will do three-dimensional skeleton tracking to develop a user control system for a lighting installation.

Summary

This chapter has packed many important concepts that will be further explored in practical projects throughout the rest of the book. You learned how to install Processing on your computer and how to install the Simple-OpenNI library that you will be using to acquire the data from Kinect.

You went through a couple of simple examples to illustrate the structure of Processing sketches and the use of the Simple-OpenNI library. You will be using Processing and Simple-OpenNI to interface with Kinect and Arduino in every single project, so this is a very important tool.

You also learned about the concept of three-dimensional space, which you will be using in some of the more advanced projects in the second half of the book, and you implemented a couple of examples of 3D applications based on the Kinect point clouds.

As you move forward in this book, you may want to re-read some of the sections of this chapter to better understand the basic concepts.