C H A P T E R 4
Arduino and Kinect: “Hello World”
In previous chapters, you learned to work with Arduino, Processing, and Kinect independently. Now you are going to build a simple project to illustrate the power of the combination of the three. This is the Arduino and Kinect equivalent of the typical “Hello World” programming exercise.
Imagine you have several LED lights and you want to turn them on and off—or even dim them at will from your favorite sofa—just by waving a hand at them. You need sensors that can recognize the waving gesture and the brightness control gestures. Here is where the Kinect hand-tracking and gesture recognition capabilities come in handy! You also need to be able to send the order to the lights to turn on or off and to set their brightness to a certain level; this is done through the Arduino board. In the course of this project, you will learn several techniques such as serial communication and pulse width modulation (PWM) that you will be using in many other projects throughout this book.
Figure 4-1. “Hello World” built on a breadboard
Parts List for Hello World
This is a very simple project so you are going to build it on a breadboard (see Figure 4-1). All you need is a couple of LEDs, your Kinect, an Arduino board, and a light-sensing resistor or photocell. Table 4-1 lists the required parts.
Note For every project in this book, you are expected to have a Kinect Device ($140) and an Arduino ($30), so these won't be included in the parts lists in the following chapters.
Before you start building, let's talk about serial communication. Serial is the base of any communication between your Arduino board and your computer, unless you are using a Bluetooth or Ethernet shield. Whenever you are uploading a program to the board, you are doing it through one of your computer's serial ports.
Serial Communication
Serial communication is a somehow “simple” communication protocol based on sequentially sending data bit by bit. All Arduino boards have one or more serial ports that load the sketches into the board and communicate with the computer at runtime. You are going to use serial communication to talk to the Arduino board from Processing and vice versa.
Of course, there is a library that allows you to control an Arduino board from a Processing sketch without having to write any Arduino code. But wait a minute; you're reading this book, so you must like coding! You're going to write your own serial communication protocol! Apart from the sheer pleasure of coding, writing your own protocols allows you to have a greater control over the data flows and the way the devices communicate.
Caution Digital pins 0 (TX) and 1 (RX) on the Arduino are used for serial communication. If you are using serial, you can't use them for digital input or output! One very common mistake is forgetting this and thus connecting a sensor or actuator to one of these pins, which will result in crazy values or behaviors. Watch out!
You're going to write a very simple serial communication routine that will allow you to turn an LED on and off from a Processing sketch. There are other serial communication examples in Processing and Arduino, so you may want to have a look at those, too.
Serial-Controlled LED
First, you need to write an Arduino sketch that will turn an LED on and off depending on the data in the serial buffer. You'll use the built-in LED at pin 13 for this example; it's the very tiny square light with an “L” next to it next to the Arduino logo (see Figure 4-2).
Figure 4-2. Arduino with serial received (RX) LED and built-in LED at pin 13 (L) on
Within setup(), you first initialize the serial communication and set pin 13 as an output pin. (Remember that pins are set to input by default, so you need to set them as output if you are planning to use them as such.)
void setup() {
Serial.begin(9600);
pinMode(13, OUTPUT);
}
In loop(), you check if there is any data in the serial buffer, and if so, read the most recent byte of information and store it in input. If input equals 1, turn the LED on; otherwise, turn it off.
void loop(){
if (Serial.available()>0) {
byte input=Serial.read();
if(input == '1'){
digitalWrite(13, HIGH);
}else{
digitalWrite(13, LOW);
}
}
}
Note The number ‘1’ is within single quote marks both in the Processing sketch and in the Arduino code to indicate that you mean the character 1, represented by the binary value 00110001. If you wrote it without quotes in both places, it would still work because you are sending the decimal value 1, represented by the binary value 00000001. If you mix characters and decimal values, such as sending '1' and then writing if(input == 1) without quote marks, you are comparing two different values (the binary number 00110001 is not the same as 00000001), which would cause your if statement to always be wrong!
Now it's time to write a Processing sketch that will send a serial ‘1’ message if the mouse is on the right side of the Processing frame, and a ‘0’ if it is on the left side. First, import the Serial library into your sketch and create an object of the serial class.
import processing.serial.*;
Serial myPort;
Within setup(), set the size to 255×255, so no value of your mouse coordinate is higher than what can be contained in a byte of information. (You'll see why later on.) Then you get the name of the first serial port on your computer and initialize the Serial object to that port.
void setup()
{
size(255, 255);
String portName = Serial.list()[0];
myPort = new Serial(this, portName, 9600);
}
In the draw() function, create a rectangle for visualization purposes and check which side of the screen your mouse X-coordinate is on. If it is on the left side of the screen, write a ‘1’ character to the serial port, and a ‘0’ if it is on the right side.
void draw() {
noStroke();
rect(0,0,width/2,height);
if (mouseX>width/2) {
myPort.write('1'),
}
else {
myPort.write('0'),
}
}
If everything went as planned, the Arduino LED should now be controlled by the position of your mouse. This is great for controlling one device in binary mode (on/off), but what if you want to set the brightness of the LED to a certain level? You could send one byte to control 256 levels of brightness, thus creating a dimmer.
Arduino has two kinds of pins, analog and digital. You can emulate analog values by using PWM. If you have been wondering what these three letters next to some digital pins meant, now you know!
Pulse Width Modulation
Pulse width modulation is basically turning the power on and off so quickly that, because of our eye's persistence of vision, we perceive the light as being permanently on, but with less intensity. In this way, by changing the percentage of time that the electricity is on and off, we can trick our eyes into perceiving different intensities of continuous light (see Figure 4-3). This is very different from using a digital-to-analog converter (DAC), which instead creates a genuine varying voltage that we can use to get different intensities from an incandescent lamp.
Figure 4-3. Pulse width modulation
PWM-Controlled LED
You're going to use pulse width modulation to create a dimmer for your light depending on the X-position of your mouse in the Processing sketch. Change the pin number to a PWM-enabled pin (we have chosen 11 here), read your input value as an integer, and send the value straight to your pin using analogWrite. Done! You have a serial driven dimmer!
Well, this is not quite true. When working with LEDs, you need to limit the current to a safe value for the LED, so let's add a resistor. (This is why you're building this circuit on a breadboard.) There are many LED resistor calculators on the Internet, but trust me on this one—you need a 220-ohm resistor for this circuit, as indicated in the diagram in Figure 4-4.
Figure 4-4. LED circuit
Note In the previous example you didn't use a resistor because pin 13 has a resistor attached to it, soldered to the board. You should try not to use this pin for input, as the internal resistor will pull the voltage level down, giving you a LOW signal at all times. If you absolutely need to use it as an input, you must use an external pull-down resistor. Pull-up and pull-down resistors are described in Chapter 1.
void setup() {
Serial.begin(9600);
pinMode(11, OUTPUT);
}
void loop(){
if (Serial.available()) { // If data is available to read,
int input=Serial.read();
analogWrite(11, input);
}
}
In the Processing sketch, you only need to replace the draw() function with the following one:
void draw(){
// Create a gradient for visualisation
for(int i = 0; i<width; i++){
stroke(i);
line(i,0,i,height);
}
// Send the value of the mouse's x-position
myPort.write(mouseX);
}
Note that you are now sending and reading the X-coordinate as an integer, so you can use it to set the brightness directly on the write() function. Figure 4-5 shows the Processing sketch and the setup on the breadboard.
Figure 4-5. Processing sketch and LED mounted on breadboard
Writing Your Own Communication Protocol
Everything you did in the previous example works well because you're only controlling one LED and you can live with 256 values. But what if you want to control two or more devices and define a whole range for each? What if you want to pass on float values? Then you need to set up your own communication protocol, which you will do in this example.
You're going to use an event trigger, a signal to tell Arduino that the communication has started and that it can start reading the following bytes and use them to modify its behavior. Let's build on the previous example by adding another LED that will be controlled by the Y-coordinate of the mouse.
Serial-Controlled LEDs
You are now going to try to dim two LEDs by sliding your mouse through a Processing sketch. The X-position of the mouse will drive the brightness of one of the LEDs, while the Y-coordinate of the mouse will drive the other LED.
First, you need to declare the integers you're using in your Arduino sketch and define your pins as output.
int val, xVal, yVal;
void setup() {
Serial.begin(9600);
pinMode(10, OUTPUT);
pinMode(11, OUTPUT);
}
In the loop, you check if enough data has been sent from the computer. You need two different values (one for each LED) and an event trigger character that indicates the start of the communication. Once you have identified the event trigger, you can read the next two values and send them straight to your LEDs.
void loop(){
// check if enough data has been sent from the computer:
if (Serial.available()>2) {
// Read the first value. This indicates the beginning of the communication.
val = Serial.read();
// If the value is the event trigger character 'S'
if(val == 'S'){
// read the most recent byte, which is the x-value
xVal = Serial.read();
// Then read the y-value
yVal = Serial.read();
}
}
// And send those to the LEDS!
analogWrite(10, xVal);
analogWrite(11, yVal);
}
Load the Arduino sketch into your Arduino board. Now you need to implement the Processing sketch that will write the X and Y values to the serial port and talk to the code that is running in the Arduino. This is pretty similar to the previous example; you're only adding the event trigger character and two different values sequentially.
import processing.serial.*;
Serial myPort;
void setup()
{
size(255, 255);
String portName = Serial.list()[0]; // This gets the first port on your computer.
myPort = new Serial(this, portName, 9600);
}
void draw() {
// Create a gradient for visualisation
for (int i = 0; i<width; i++) {
for (int j = 0; j<height; j++) {
color myCol = color(i,j,0);
set(i, j, myCol);
}
}
// Send an event trigger character to indicate the beginning of the communication
myPort.write('S'),
// Send the value of the mouse's x-position
myPort.write(mouseX);
// Send the value of the mouse's y-position
myPort.write(mouseY);
}
That's it! You are now sending two different values from Processing to control two different devices with your Arduino board (see Figure 4-6). Of course, you can extend the same logic to three, four, or any higher number of devices attached to your Arduino board.
Figure 4-6. Two LEDs mounted on the breadboard
Now you know how to send serial data from your Processing sketch to your Arduino board, so what about telling your lights when to turn off without a mouse or a keyboard? It's time to bring in the Kinect!
Kinect-Controlled LEDs
You are going to take the previous example and change the mouse input to Kinect input. The aim is to have the brightness of two LEDs driven by the position of your hand. This is the first project in which you use your Kinect, so let's proceed one step at a time.
The Arduino code for this example is the same as that in the previous example, so if you skipped it, go back and copy the Arduino code and upload it to your board.
First, import the required libraries for the project and declare the variables that will be associated to the Simple—OpenNI and Serial objects.
import SimpleOpenNI.*;
import processing.serial.*;
SimpleOpenNI kinect;
Serial myPort;
Then define two PVectors to contain the position of the hand as defined by the Kinect (handVec), the position of the hand in projective screen coordinates (mapHandVec), and a color for the hand-tracking point.
PVector handVec = new PVector();
PVector mapHandVec = new PVector();
color handPointCol = color(255,0,0);
In setup(), you start by initializing the Simple-OpenNI object, passing the current PApplet (this) as a parameter. Then you enable all the functions you will be using. You need the depth map (to see yourself on screen), the hands, and the gestures. Use the Wave gesture to tell the Kinect when to start tracking your hand. Finally, set the sketch size according to Kinect's depth map size and initialize the Serial object as you did in the previous example.
void setup() {
kinect = new SimpleOpenNI(this);
// enable mirror
kinect.setMirror(true);
// enable depthMap generation, hands and gestures
kinect.enableDepth();
kinect.enableGesture();
kinect.enableHands();
// add focus gesture to initialise tracking
kinect.addGesture("Wave");
size(kinect.depthWidth(),kinect.depthHeight());
String portName = Serial.list()[0]; // This gets the first port on your computer.
myPort = new Serial(this, portName, 9600);
}
Now is the moment to add the Simple-OpenNI callbacks (the functions that the library invokes when it recognizes certain events). The first one is triggered when a gesture is recognized, in this case when you wave at the Kinect. You then remove the gesture from the capabilities, so Kinect doesn't keep looking for gestures. Once you know a gesture has been performed, you tell NITE to start tracking hands at the position where the gesture occurred.
void onRecognizeGesture(String strGesture, PVector idPosition, PVector endPosition)
{
kinect.removeGesture(strGesture);
kinect.startTrackingHands(endPosition);
}
This will trigger the creation of a Hand object. After creating the hand, change the color of your hand tracking point to green to indicate that the gesture was recognized.
void onCreateHands(int handId, PVector pos, float time)
{
handVec = pos;
handPointCol = color(0, 255, 0);
}
Finally, every time the position of the hand is updated, update the hand vector so you can work with it in the draw() function.
void onUpdateHands(int handId, PVector pos, float time)
{
handVec = pos;
}
The first thing to do in the draw() function is to update the Simple-OpenNI object (kinect), which in turn calls the three previous functions. Once updated, the position of handVec is also updated, but handVec contains the real-scale 3D position of the hand, so you need to map its coordinates to your 2D screen coordinates. Luckily, Simple-OpenNI has a specific method for performing this operation (with a rather long name), which you will use to this effect.
void draw() {
kinect.update();
kinect.convertRealWorldToProjective(handVec,mapHandVec);
You now have all the data you need from the Kinect, so it's time to print the depth image to the screen and draw the hand-tracking point from the mapped coordinates.
image(kinect.depthImage(), 0, 0);
strokeWeight(10);
stroke(handPointCol);
point(mapHandVec.x, mapHandVec.y);
And lastly, send the event trigger character 'S' to the serial port, followed by the values of your hand position mapped to the limits of a serial message (0-255).
// Send a marker to indicate the beginning of the communication
myPort.write('S'),
// Send the value of the mouse's x-position
myPort.write(int(255*mapHandVec.x/width));
// Send the value of the mouse's y-position
myPort.write(int(255*mapHandVec.y/height));
}
So, if you did things right, you should now have two LED lights on your breadboard that you can control with the vertical and horizontal movements of your hand. The Processing screen should look something like Figure 4-7.
Figure 4-7. Processing screen with hand tracking
Let's add one more feature to this project; you have been talking to Arduino from Processing, but serial communication works both ways, so you can use serial to send data back from the Arduino sensors into Processing. This generates a communication feedback loop that will be very useful in later projects.
Feedback from Arduino
Your sketch works and you are seeing your depth image on the screen. Let's imagine that you prefer seeing your RGB image on the screen whenever possible, but if the lighting level of the room is too low, you want the image to revert back to the depth image. You can do this by getting feedback from a light-sensing resistor connected to your Arduino board.
You're going to use a very simple and cheap photocell. This cell provides a full range of values between 0 and 1023 (depending on the resistor), so connect it to the analog input pin 0. The circuit for the photocell includes a 10k resistor and looks something like the diagram in Figure 4-8.
Figure 4-8. Photocell circuit 1; voltage output increases with brightness
This kind of circuit is called a resistive divider, which is a special case of voltage divider, a circuit that produces an output voltage (in your case, going to analog pin 0) that is a fraction of the input voltage (5V). The output voltage can be easily found applying Ohm's law.
R1 is the value of the resistor connected to Vin, and R2 is the value of the resistor connected to ground. The fixed resistor should have the same order of magnitude as the variable's resistor range.
In your case, the photocell has a range of 1K (in bright light) to 10K (in the dark). You're using a 10K fixed resistor, which, applying the equation, will give you a voltage output range of 4.54V to 2.5V.
There is a catch to this circuit. If you switch the position of the photocell and the resistor (see Figure 4-9), it will still work but the values will be inverted; this means the numbers will grow as the light decreases. You can use it either way; just be aware of the input you are getting when you read the data in analog pin 0.
Figure 4-9. Photocell circuit 2; voltage output decreases with brightness
The full circuit is getting a little more complicated now, so we are including the full schematics that we produced with Fritzing, as covered in Chapter 1 (see Figures 4-10 and 4-11).
Figure 4-10. The physical project and Fritzing breadboard view
Figure 4-11. Project schematics produced with Fritzing
In the Arduino sketch, you add a variable to store the value of the photocell and the pin number for your analog input. The setup() function will stay unchanged.
int val, xVal, yVal;
int sensorValue;
int sensorPin = 0;
void setup() {
// initialize the serial communication:
Serial.begin(9600);
// initialize the serial communication:
pinMode(10, OUTPUT);
pinMode(11, OUTPUT);
}
Then add two lines at the beginning of the loop() to read the values of the resistor and send a message to the serial port. You can add a little delay later in order to allow the analog-to-digital converter to stabilize after the last reading.
void loop(){
// Check Values from the Photocell
sensorValue = analogRead(sensorPin);
Serial.println(sensorValue);
// wait a bit for the analog-to-digital converter
// to stabilize after the last reading:
delay(10);
// check if data has been sent from the computer:
if (Serial.available()>2) { // If data is available to read,
val = Serial.read();
if(val == 'S'){
// read the most recent byte (which will be from 0 to 255):
xVal = Serial.read();
yVal = Serial.read();
}
}
analogWrite(10, xVal);
analogWrite(11, yVal);
}
In the Processing sketch, add a variable to store the values received.
import SimpleOpenNI.*;
import processing.serial.*;
SimpleOpenNI kinect;
Serial myPort;
PVector handVec = new PVector();
PVector mapHandVec = new PVector();
color handPointCol = color(255, 0, 0);
int photoVal;
Within setup(), enable the RGB image and, after initializing the port, specify that Processing doesn't generate a serialEvent() until a newline character has been received.
void setup() {
kinect = new SimpleOpenNI(this);
// disable mirror
kinect.setMirror(true);
// enable depthMap generation, hands and gestures
kinect.enableDepth();
kinect.enableRGB();
kinect.enableGesture();
kinect.enableHands();
// add focus gesture to initialise tracking
kinect.addGesture("Wave");
size(kinect.depthWidth(), kinect.depthHeight());
String portName = Serial.list()[0]; // This gets the first port on your computer.
myPort = new Serial(this, portName, 9600);
// don't generate a serialEvent() unless you get a newline character:
myPort.bufferUntil('
'),
}
In the draw loop, create a conditional clause that will display RGB or the depth image on screen depending on the value of the photocell that you are acquiring inside the serialEvent() method described next. Then draw the hand point and send its values, as you did previously.
void draw() {
kinect.update();
kinect.convertRealWorldToProjective(handVec,mapHandVec);
// Draw RGB or depthImageMap according to Photocell
if (photoVal<500) {
image(kinect.rgbImage(), 0, 0);
} else {
image(kinect.depthImage(), 0, 0);
}
strokeWeight(10);
stroke(handPointCol);
point(mapHandVec.x, mapHandVec.y);
// Send a event trigger character to indicate the beginning of the communication
myPort.write('S'),
// Send the value of the mouse's x-position
myPort.write(int(255*mapHandVec.x/width));
// Send the value of the mouse's y-position
myPort.write(int(255*mapHandVec.y/height));
}
Next, add a serialEvent() method, which is called by Processing when you receive the newline character (remember you sent a Serial.printline from Arduino, which includes a newline character at the end of the message). Read the message, trim any white spaces, and convert it to an integer.
void serialEvent (Serial myPort) {
// get the ASCII string:
String inString = myPort.readStringUntil('
'),
if (inString != null) {
// trim off any whitespace:
inString = trim(inString);
// convert to an integer
photoVal = int(inString);
}
else {
println("No data to display");
}
}
The NITE callbacks stay the same, as in the previous example. You just add some print line commands to keep track of the process on the console.
void onCreateHands(int handId, PVector pos, float time)
{
println("onCreateHands - handId: " + handId + ", pos: " + pos + ", time:" + time);
handVec = pos;
handPointCol = color(0, 255, 0);
}
void onUpdateHands(int handId, PVector pos, float time)
{
println("onUpdateHandsCb - handId: " + handId + ", pos: " + pos + ", time:" + time);
handVec = pos;
}
void onRecognizeGesture(String strGesture, PVector idPosition, PVector endPosition)
{
println("onRecognizeGesture - strGesture: " + strGesture + ", idPosition: " + idPosition + ", endPosition:" + endPosition);
kinect.removeGesture(strGesture);
kinect.startTrackingHands(endPosition);
}
In this way, whenever the lighting level drops under a certain threshold, your image is automatically changed to the depth image, which is independent of the ambient light.
Summary
This chapter was all about making Kinect and Arduino talk through Processing, and you now have a working example. You are indeed gathering information from the physical world with your Kinect device and sending that information from Processing to Arduino through serial communication. You are also gathering lighting level information through a photocell connected to the Arduino board and modifying the Processing visualization according to it.
During this process, you used Kinect's hand-tracking capabilities for the first time, learned how to use serial communication, and wrote your own protocols. You have also learned how to use photocells, voltage dividers, and Arduino PWM-enabled pins. This is probably enough for an introductory project; the next chapter has even more exciting stuff!