C H A P T E R 5
Kinect Remote Control
Imagine controlling all your home appliances remotely just using body movements and gestures. There are a number of machines and devices that can be controlled electrically in an automated house, such as television sets, DVD players, radios, lamps, and the heating system. In order to avoid the problem of tinkering with mains power and the obvious risks of opening up expensive devices, this chapter will focus on hacking an external element that can control a broad range of devices: a general-purpose remote control. Hacking a remote control is a really simple job; you just open the remote and connect it to an Arduino board. You then use the Kinect to perform gesture recognition in order to activate different functions.
In this project you will learn how to use NITE's gesture recognition capabilities and you'll even implement a simple gesture recognition routine yourself. You will send the data gathered to Arduino using serial communication, so you will ultimately be able to control the remote control with your body movements. This application can be used on pretty much any infrared-based remote control just by connecting the Arduino pins to the correct buttons on the remote.
Parts List for Kinect Remote Control
All you need for this project is a general-purpose remote control (the guts of which you can see in Figure 5-1), your Kinect, an Arduino board, a prototyping shield, some relays, and a bunch of cables for connecting them. Table 5-1 has more details on the required parts.
Figure 5-1. Parts of the remote control
You'll start by hacking the remote control, connecting it to Arduino, and testing it to make sure it works properly. Then you'll develop the Kinect sketch that will track your hands and gestures. Once everything is neat and tidy, you will connect the Kinect to Arduino using serial communication, as you did in the previous chapter.
There is a wide range of remote controls available and they all look quite different from each another, but as long as they are infrared-based they all work pretty much in the same way.
Hacking a Remote Control
Remote controls are completely inactive until you push a button. When you hit one of the buttons, the remote will send a signal to a receiver device, which should interpret the signal and perform the action you intend it to perform. Some remote controls use radio signals to communicate with the receiver devices, but most communicate via infrared light. You'll be focusing on infrared remotes.
When you hit a button on your remote, you are actually closing one of the open circuits inside it. At that moment, the remote starts to send a signal that goes to the receiver device. This carrier signal consists of a series of precisely spaced infrared pulses constituting a code that is processed by the receiver device. Once decrypted, the receiver triggers different functions according to the signal received. Apart from sending out pulses that encode the instructions for the receiver, the remote control sends a short code that identifies the device for which the message is intended.
The first thing you need to do is pair the universal remote control with the device that you want to control. Put the batteries in and follow the instructions. Pay attention to how long you have to press a button for a command to be transmitted because this information is useful for refining the Arduino code that controls the remote. Once you have tested the device (by changing channels and volume), it's time to move on to the most interesting part: opening up the remote (see Figure 5-2)!
Figure 5-2. Opening the remote
Remove the batteries and the screws so you can see the printed circuit board. Observe that the buttons are made of plastic and have a conductive material on the inner side. You have to locate the solder pads, which are the areas where you can solder your cables (see Figure 5-3). When you connect them back, you bridge the circuit in the same way as you do by holding a button.
Figure 5-3. Wires soldered to the remote pads
Normally two pins connect when the button is pressed, but some remotes have four to make the connection stronger. Sometimes it is not easy to find which solder pads you want to connect in order to transmit the right command. There are two main ways of finding them. The most straightforward method is by putting the batteries back in the remote and using a short cable to try to connect the right pads. You can sit in front of the TV, try the connections you think are right, and check what happens! A second, more scientific option is by using a multimeter to measure the resistance between the pads. If you are testing the right pads, you will see a change from high values of resistance to very low values when you press a button.
In this project, you need to find the right pads for sending the commands Channel+, Channel -, Volume +, and Volume -. Once you find them, you can connect the cables so that they are ready to be soldered to the prototype shield. Use cables of different colors so you don't lose track of them!
Connecting the Remote to Arduino
We're going to explain the basics of how to build a prototype shield and how to connect it to Arduino. Here you will find enough information for building it, but if you want a deeper explanation and information on how to build other Arduino applications, look to the book Practical Arduino by Jonathan Oxer and Hugh Blemings (Apress, 2009).
The way that the remote control circuit operates is by closing a circuit. When you hold a button, it works as a mechanical switch. You are going to perform the same function by using relays. A relay is an electrically operated switch that controls a circuit with a low-power signal; both circuits remain electrically isolated. In your case, a low current coming from Arduino controls the flow of current in the remote control. 5v relays require about 20 mA to operate, meaning that you can connect them directly to your Arduino outputs (Figure 5-4).
Figure 5-4. Remote control switches testing a relay
First, you're going to do a test on a breadboard in order to understand how to connect a relay to your Arduino board: you're going to control a LED through a relay.
Figure 5-5. Testing the relay on a breadboard
Relays have their input (coil connection) in the central pair of pins and the output on the outer pair of pins. First you insert the relay in the breadboard, making sure it spans unconnected rows. Connect power (5V) and ground to the breadboard. Then insert the LED into the breadboard, connecting the short leg to ground. After that, connect the relay. Input pins are connected to pin 2 of the Arduino board and ground. For the output, connect power in one end and a resistor in the other that connects to the longer leg of the LED. This constitutes the basic circuit, as shown in Figures 5-5 and 5-6. Then you upload into Arduino the example Blink, changing the output pin to pin 2.
void setup() {
pinMode(2, OUTPUT); // Set the digital pin as an output.
}
void loop() {
digitalWrite(2, HIGH); // set the LED on
delay(1000); // wait for a second
digitalWrite(2, LOW); // set the LED off
delay(1000); // wait for a second
}
Your Arduino sends a signal to the relay, and the relay allows the current to pass during the same time that your signal is High (which is 1 second, according to the code). You will hear a little noise, meaning the relay is switching on and off.
Figure 5-6. Schematics of the relay circuit
Tip It's necessary to introduce a diode in order to protect your circuit. This is due to the way a relay operates; it generates a magnetic field to mechanically close the circuit. When the relay is turned off, the magnetic field stops and generates a reverse spike. This is potentially dangerous for your circuits, as you are receiving a negative voltage on the same pin where you were supplying positive voltage. If the spike is big enough, it can damage the circuit. In order to solve this problem, you can connect a diode across the relay coil, protecting your Arduino. Diodes work only in one direction, stopping the negative voltage from going to your Arduino. To connect them, you need to connect the cathode lead (the end with the stripe) towards the Arduino pin output and the anode lead to the ground. It's important to take this into consideration when you are using high voltage. However, due to the range you are using in this exercise (very low power), it's unlikely that the relay will damage your Arduino, so you're not going to incorporate them in your prototype shield.
Assembling the Prototype Shield
Now that you understand the principles behind a relay, you can start building stuff! The first thing is to mount four relays on the prototype shield. Observing the shield, you can identify one line for power and other line for ground. There should be enough space for assembling four relays (Figures 5-7 and 5-8).
Figure 5-7. Four relays on the prototype shield
Figure 5-8. Back of prototype shield
Start by fitting the relays and soldering the necessary connectors (two legs of inputs and two legs of outputs). Then, using short cables, connect one of the coils of each relay together and to the ground (see Figure 5-9).
Figure 5-9. Relays connected to Arduino shield
You can also incorporate the breakaway headers to connect the prototype shield to the Arduino (see Figure 5-9). Once the headers are fitted, turn the prototype shield around and connect the other coil to the matching Arduino digital pin. In this example, you're using pins 5, 6, 7, and 8, but you can use any of them; just make sure they're coordinated with the code (see Figure 5-10)!
Figure 5-10. Breakaway headers
You have the relays' inputs connected to Arduino, so now you connect the outputs; just connecting a pair of cables to each output connection will do (see Figure 5-11). Make sure you leave enough length to connect them to the remote control. You can screw one connector in each cable to make it easy to connect them with the cables coming from the remote control.
Figure 5-11. Finished shield
You're almost done. Attach the prototype shield to your Arduino board and connect each pair of cables with the corresponding ones on the remote control. Table 5-2 lists the correct connections and Figure 5-12 shows the end result.
Figure 5-12. Remote connected to Arduino
Now your prototype shield is ready to be tested! If your remote control runs on 3V, instead of using its own batteries you can run it from Arduino. Just remove the batteries and solder a pair of wires in the positive and ground terminals, connecting them to the 3.3V pin and ground respectively (+ goes to 3.3V!). This allows the remote control to take its power from the Arduino so you don't have to worry about batteries running down.
Testing the Circuit
Now that you have the prototype shield ready and connected to the remote control, it's time to program your Arduino in order to do the first tests! Once you're sure that it's working and ready to receive data, you can develop the Processing application that will drive the remote.
Let's start from the basic Arduino code for testing. You begin by defining a set of variables, assigning a symbolic name to each pin (5,6,7,8). You also define a pulse that is the time that the pin is going to be High, equivalent to the time that you hold a button in the remote control.
int ChannelPlusPin = 5;
int ChannelLessPin = 6;
int VolumePlusPin = 7;
int VolumeLessPin = 8;
int pulse = 250;
After defining the variables, start the setup() function. Describe each pin as outputs and start the serial communication.
void setup(){ pinMode(ChannelPlusPin, OUTPUT);
pinMode(ChannelLessPin, OUTPUT);
pinMode(VolumePlusPin, OUTPUT);
pinMode(VolumeLessPin, OUTPUT);
Serial.begin(9600);// Start serial communication at 9600 bps
}
Before starting the loop(), define an external function to perform a series of commands. This function is called from the loop(), modifying the state of each pin to High (on), waiting some time (pulse previously defined), and returning Low (off). The function takes two arguments for the pin number and the pulse. The Serial.print commands write text to the serial port (ON, name of the pin used, and OFF) so you can be aware of which function has been called at every moment.
void updatePin (int pin, int pulse){
Serial.println("RECEIVED");
Serial.println(pin);
digitalWrite(pin, HIGH);
delayMicroseconds(pulse);
digitalWrite(pin, LOW);
Serial.println("OFF");
}
The main loop() function is where all the action happens. Arduino is listening for available serial data. If there is data, it will be stored inside a char variable; according to the value, it will call the updatePin() function to the pin assigned. If the message is “1”, Arduino will modify the state of the pin ChannelPlus (Pin 5), activating it for a period of time. In other words, this will turn on the relay connected to this pin, activating the remote control and changing the channel to the next one. After the pulse time has elapsed, the pin will be brought back to low voltage, so the relay will be turned off and the mechanism will come to a steady state, waiting for new data to come in.
void loop(){
if (Serial.available()) {
char val=Serial.read();
if(val == '1') {
updatePin(ChannelPlusPin, pulse);
} else if(val == '2') {
updatePin(ChannelLessPin, pulse);
} else if(val == '3') {
updatePin(VolumePlusPin, pulse);
} else if(val == '4') {
updatePin(VolumeLessPin, pulse);
}
}
}
This code is written for four commands, so you're using just four pins; feel free to add more pins and more commands in order to have a full set of instructions for your remote control.
Once you have finished writing your code, plug the Arduino into the USB port, select the correct Arduino (in our case Arduino Uno), verify the sketch, and upload it!
In physical computing it's a good idea to get accustomed to frequently checking if everything is connected correctly. So take a moment now to check the following:
- The Arduino is connected with a USB cable to the computer.
- The remote control has working batteries.
- The Arduino is plugged into your prototype shield.
- The TV is on.
- The remote control is pointing towards the TV.
For testing purposes, use the serial communication console to introduce your values by typing on the keyboard (see Figure 5-13). Double-check that serial communication is at the same baud rate as the one specified in the code (9600 in our case).
Figure 5-13. Using the Serial Monitor to input data
Enter a value of 1 in the text area. You should read a response from your Arduino saying that the data has been received and the pin number being used, followed by an off message. If everything worked as expected, the remote should have changed the channel up to the next one; you should be able to send the other commands by manually introducing other values as 2, 3, or 4.
Kinect Hand Tracking and Gesture Recognition
You now have a hacked remote control that is connected to an Arduino board and is just waiting for data to stream in. Your next goal is to easily communicate the commands “next channel,” “previous channel,” “volume up,” and “volume down” to the remote using hand gestures.
We have figured out that the easiest way to tell the Kinect that we want to change the channel is by doing a quick movement with our hand to the left or right, so you're going to implement a right or left “swipe” gesture for that purpose. For the volume, we could have done the same with “up” and “down”, but it is not that easy to “swipe” upwards or downwards (try it!), and it's nicer to have a continuous control over the volume, so we have decided that the good old-fashioned potentiometer will do the deal. You will use the circle detector included with NITE to allow you to smoothly change the volume.
Libraries and Setup
Start by importing the core Serial and OpenGL libraries and the Simple-OpenNI library to your sketch; also, declare the variables for the Serial and Simple-OpenNI objects.
import SimpleOpenNI.*;
import processing.opengl.*;
import processing.serial.*;
SimpleOpenNI kinect;
Serial myPort;
In the previous project you used NITE through Simple-OpenNI callbacks, but you didn't declare the NITE objects explicitly. In this project, you're going to use two NITE functions: hand tracking and gesture recognition for the waving, hand raising, and circle detection. For this purpose, you declare an XnVSessionManager object to deal with NITE gesture recognition, an XnVPointControl object to keep track of the hand points, and an XnVCircleDetector to trigger events when it recognizes a circle being drawn with a hand.
Note All classes, objects, and methods starting with “XnV” refer to NITE commands.
XnVSessionManager sessionManager;
XnVPointControl pointControl;
XnVCircleDetector circleDetector;
This project requires that you display text on screen, so you need to declare a text font. In this project, we used SansSerif-12, so you need to create the font first by going to Tools menu > Create Font.
PFont font;
The sketch tracks your hand as it moves on screen, and then it triggers events whenever it recognizes a “channel change” or “volume change” gesture, which each have their own timing. This requires the main draw() loop to run in three different modes. It remembers these modes in the variable called mode.
int mode = 0;
Now declare the variables for the hand tracking and volume control functions. For hand tracking, you need a signal that a hand is being tracked (handsTrackFlag), variables for hand positions (real position and projected on screen), and the arrayList of PVectors containing previous positions.
boolean handsTrackFlag = true;
PVector screenHandVec = new PVector();
PVector handVec = new PVector();
ArrayList<PVector> handVecList = new ArrayList<PVector>();
int handVecListSize = 30;
For volume control and channel, you declare previous and current rotations of the hand in the circle, a radius, an angle for the volume control gizmo, and its center in absolute and relative coordinates. For channel change, you need a changeChannel integer to define the direction of the change and a channelTime timer to keep track of the channel change gizmo.
float rot;
float prevRot;
float rad;
float angle;
PVector centerVec = new PVector();
PVector screenCenterVec = new PVector();
int changeChannel;
int channelTime;
In setup(), initialize all the objects and then enable the functions that you expect to be using through the execution of the program. You need the depth map, gestures, and hands enabled in the kinect object.
void setup(){
// Simple-openni object
kinect = new SimpleOpenNI(this);
kinect.setMirror(true);
// enable depthMap generation, hands + gestures
kinect.enableDepth();
kinect.enableGesture();
kinect.enableHands();
This is important: initialize the NITE session manager by adding the two gestures you'll be working with, and then initialize the hand point controller and circle detector. Call the methods RegisterPointCreate, RegisterPointDestroy, and RegisterPointUpdate in the point control object (passing the current applet “this” as a parameter) so the XnVPointControl object will invoke your callback functions. Now you have a control in the processes triggered at the creation, destruction, and update intervals of a hand point.
In the same way, call the RegisterCircle and RegisterNoCircle in the circle detector so it invokes your callbacks for circle detection. Finally, add the listeners to the NITE session so they get initialized when you start your session.
// setup NITE
sessionManager = kinect.createSessionManager("Wave", "RaiseHand");
// Setup NITE.s Hand Point Control
pointControl = new XnVPointControl();
pointControl.RegisterPointCreate(this);
pointControl.RegisterPointDestroy(this);
pointControl.RegisterPointUpdate(this);
// Setup NITE's Circle Detector
circleDetector = new XnVCircleDetector();
circleDetector.RegisterCircle(this);
circleDetector.RegisterNoCircle(this);
// Add the two of them to the session
sessionManager.AddListener(circleDetector);
sessionManager.AddListener(pointControl);
Wow, that was a little bit difficult! Don't worry if you didn't quite understand what was going on with these functions; it will all become clearer when you get to the callback functions. The only things you have left to do in your setup() function are to set the size of the sketch to match the depthMap and to initialize the font and the serial communication.
size(kinect.depthWidth(), kinect.depthHeight());
smooth();
font = loadFont("SansSerif-12.vlw");
String portName = Serial.list()[0]; // This gets the first port on your computer.
myPort = new Serial(this, portName, 9600);
}
Now, before you get into the draw() loop, you need to write the callbacks from NITE so all the previous declaring and calling of weird functions makes some sense.
NITE Callbacks
What you have done by setting up these objects is take advantage of NITE's gesture recognition framework. Now you can include a series of functions (the ones you specified!) that will be invoked when specific events are triggered by NITE.
Regarding hand recognition, you call functions when the hand points are created, destroyed, and updated (see Figure 5-14). Upon creation of a hand point, you set the flag to true so you know that a hand is being tracked, set the handVec PVector to the position of the hand by using the parameter coming from NITE's call of the function, clear the list, and add the first point.
void onPointCreate(XnVHandPointContext pContext){
println("onPointCreate:");
handsTrackFlag = true;
handVec.set(pContext.getPtPosition().getX(),
pContext.getPtPosition().getY(),
pContext.getPtPosition().getZ());
handVecList.clear();
handVecList.add(handVec.get());
}
On hand destruction, set the flag to false.
void onPointDestroy(int nID){
println("PointDestroy: " + nID);
handsTrackFlag = false;
}
Figure 5-14. Hand tracked by XnVPointControl
And whenever the hand point is updated, you update the hand vector and add another point to the list. If the list is larger than the size specified, get rid of its oldest element.
void onPointUpdate(XnVHandPointContext pContext){
handVec.set(pContext.getPtPosition().getX(),
pContext.getPtPosition().getY(),
pContext.getPtPosition().getZ());
handVecList.add(0, handVec.get());
// remove the last point
if (handVecList.size() >= handVecListSize) {
handVecList.remove(handVecList.size()-1);
}
}
The circle detector has its own callback functions as well: one for the circle “being there” and one when there is no circle. Simple.
When there is a circle, you get your rotation from the parameter fTimes. This parameter gives you the number of rotations your hand has made around the circle, so a 90 degree rotation will mean fTimes is equal to 0.25. This is very useful when you want to control the volume with a constant rotation of your hand. You need to extract your angle as well, which you also get from fTimes with a little function. You need the circle's center as well, so you extract it from the circle object with circle.getPtCenter(). But be careful! The center you are given doesn't include a z-coordinate! You get the z-coordinate from the hand vector, and then you convert to projective to get the projection of the circle on screen for visualization. You then extract the radius of the circle and set your sketch mode to “1”, or volume control mode.
void onCircle(float fTimes, boolean bConfident, XnVCircle circle)
{
println("onCircle: " + fTimes + " , bConfident=" + bConfident);
rot = fTimes;
angle = (fTimes % 1.0f) * 2 * PI - PI/2;
centerVec.set(circle.getPtCenter().getX(), circle.getPtCenter().getY(), handVec.z);
kinect.convertRealWorldToProjective(centerVec, screenCenterVec);
rad = circle.getFRadius();
mode = 1;
}
If there is no circle, you just tell your program to go back to mode “0”, or waiting mode.
void onNoCircle(float fTimes, int reason)
{
println("onNoCircle: " + fTimes + " , reason= " + reason);
mode = 0;
}
Draw Loop and Other Functions
The previous callback functions give you all the data you need from your Kinect device. Now you transform the data into clear signals to be sent to the Arduino, and you implement a simple interface to understand on screen what is happening in your code.
The main draw() loop, as usual, updates the kinect object, and draws the depth map on screen so you can follow what the Kinect is seeing. Then the loop runs into a switch statement that breaks the loop into three different paths.
void draw(){
background(0);
kinect.update();
kinect.update(sessionManager);
image(kinect.depthImage(), 0, 0); // draw depthImageMap
The switch structure chooses one of the cases' blocks according to the value of the mode variable. While mode is 0 (waiting mode), you keep checking the x-component of the hand to identify the swipe gesture. Then you draw the hand position and tail.
If mode is 1 (volume control mode), you call the function volumeControl(), which will draw the gizmo and send the volume signal to Arduino.
If mode is 2 (channel change mode), you get stuck in a loop that sends the channel change signal to Arduino, draws the gizmo, and avoids any other signal to be attended to for a number of frames.
switch(mode) {
case 0:
checkSpeed();
if (handsTrackFlag) { drawHand(); }
break;
case 1:
volumeControl();
break;
case 2:
channelChange(changeChannel);
channelTime++;
if (channelTime > 10) {
channelTime = 0;
mode = 0;
}
break;
}
}
The checkSpeed() function is where you have implemented your simple swipe gesture recognition. If you think about it, what is different in that gesture from the normal behavior of the hand? Its horizontal speed! Actually, just by analyzing the current horizontal speed of the hand, you can detect the channel change gesture. And as you can extract the current hand position and the previous one from your handVecList, you can measure the difference in their x-coordinates to set a threshold that defines when the hand is in normal movement and when it is swiping. Set a threshold of 50mm between frames (this may need to be fine-tuned for faster or slower computers).
If you recognize a gesture, you send the program to channel change mode and then set the channelChange variable to 1 for right direction and -1 for left direction.
void checkSpeed() {
if (handVecList.size() > 1) {
PVector vel = PVector.sub(handVecList.get(0), handVecList.get(1));
if (vel.x > 50) {
mode = 2;
changeChannel = 1;
}
else if (vel.x < -50) {
changeChannel = -1;
mode = 2;
}
}
}
When the main loop is in channel change mode, it calls channelChange and passes in a parameter defining the side of the gesture. If it's the first loop (channelTime is equal to 0), you send the channel change signal to the Arduino. In any case, you draw an arrow with some text to indicate that you are changing the channel (see Figure 5-15). You limit the sending of the signal to the first loop to make sure you're not sending repeated signals while the arrow is displayed on screen.
void channelChange(int sign) {
String channelChange;
pushStyle();
if (sign==1) {
stroke(255, 0, 0);
fill(255, 0, 0);
// Send the signal only if it's the first loop
if (channelTime == 0)myPort.write(1);
textAlign(LEFT);
channelChange = "Next Channel";
}
else{
stroke(0, 255, 0);
fill(0, 255, 0);
// Send the signal only if it's the first loop
if (channelTime == 0)myPort.write(2);
textAlign(RIGHT);
channelChange = "Previous Channel";
}
// Draw the arrow on screen
strokeWeight(10);
pushMatrix();
translate(width/2,height/2);
line(0,0,sign*200,0);
triangle(sign*200,20,sign*200,-20,sign*250,0);
textFont(font,20);
text(channelChange,0,40);
popMatrix();
popStyle();
}
Figure 5-15. Next channel and previous channel gestures
When you enter volume control mode in the main draw loop, you call the volumeControl method. This function draws the circle that NITE has detected by making use of the radius, center, and angle variables updated on every circle update.
You need to check if the present rotation of the hand is bigger than the previous one (meaning you are rotating clockwise) or smaller (counter-clockwise rotation). In the first case, you send the “volume up” signal to the Arduino, and in the second one, “volume down.” You don't need to worry that you are sending repeated signals because that is exactly what you do when you hold the volume button down on your remote! You also display the circle and the current rotation of the hand on screen, as shown in Figure 5-16.
void volumeControl() {
String volumeText = "You Can Now Change the Volume";
fill(150);
ellipse(screenCenterVec.x, screenCenterVec.y, 2*rad, 2*rad);
fill(255);
if (rot>prevRot) {
fill(0, 0, 255);
volumeText = "Volume Level Up";
myPort.write(3);
}
else {
fill(0, 255, 0);
volumeText = "Volume Level Down";
myPort.write(4);
}
prevRot = rot;
text(volumeText, screenCenterVec.x, screenCenterVec.y);
line(screenCenterVec.x, screenCenterVec.y, screenCenterVec.x+rad*cos(angle),
screenCenterVec.y+rad*sin(angle));
}
Figure 5-16. Volume control gizmo turning the volume up
Finally, add a drawHand function very similar to the one you implemented in the previous chapter (thus we won't go into the details again here).
void drawHand() {
stroke(255, 0, 0);
pushStyle();
strokeWeight(6);
kinect.convertRealWorldToProjective(handVec, screenHandVec);
point(screenHandVec.x, screenHandVec.y);
popStyle();
noFill();
Iterator itr = handVecList.iterator();
beginShape();
while ( itr.hasNext ())
{
PVector p = (PVector) itr.next();
PVector sp = new PVector();
kinect.convertRealWorldToProjective(p, sp);
vertex(sp.x, sp.y);
}
endShape();
}
To sum up, once the sketch is up and running you can wave your hand in front of the Kinect (saying hello with your hand) and a red dot will appear in the center of your hand. This means NITE has recognized your hand and it's tracking it. You'll also see a long tail of the 30 last positions of your hand.
If you swipe your hand horizontally (much in the same way as Tom Cruise in the movie Minority Report), you'll see the channel change signal on screen. If you do a circular movement with your hand, you'll see the volume control gizmo. If you want to stop the volume control, you need only move your hand away from the virtual circle that you have drawn in space.
It's fun to see how awkward it is at the beginning, but you quickly get used to the different movements. If you find that the swipe gesture is too sensitive or too narrow, you can change the threshold until you feel more comfortable with it.
You have to run this code with the Arduino connected to your computer; otherwise you'll get a serial error. If you want to run it without an Arduino, you can comment out all the calls to the serial object.
Connecting the Processing Sketch to Arduino
So now you have a Processing sketch that recognizes several different gestures and an Arduino programmed to control a remote. And you have cunningly prepared your Processing sketch to send the same serial data that you are using to control your Arduino.
So that's it! If you connect both the Kinect and the Arduino to your computer, upload the previous code to your Arduino, and run the Processing sketch, you should be able to stand back, wave at your Kinect, and start controlling your TV with hand gestures! Play with it, and get to ready impress your guests with a session of channel zapping without a remote. They'll surely be most impressed.
Summary
This chapter led you through the process of opening a remote control, examining its inner workings, and connecting it to your Arduino board in order to control it from your computer. Then you used NITE's gesture recognition capabilities, complemented with your own implementation of a gesture recognition routine, to alter the state of the remote by using body gestures.
As a result, you have built a system that allows you to control a home appliance (a TV set in this case) by waving, swiping, and circling your hand in space without the need of any electronic device in your hand.
This example aims to constitute a base for hacking any other appliances controlled by a remote. This is only the principle. Once you have opened a TV remote control, you can investigate other connections and see which commands are sent. Most infrared remote devices work by the same principles, so you can also hack a remote-control car or helicopter and control these devices with the movement of your hands or your body! You'll see more of this in the following chapters so stay tuned!