C H A P T E R 7
Mood Lamps
Light greatly affects the spaces we inhabit. We all try control the lighting levels and quality in our homes by installing specific lighting fixtures, wall lamps, spotlights, etc. to create the desired ambience. In this chapter, you are going to learn how to build your own wireless RGB LED lamp and how to install a lighting control system that lets you modify the state of several lamps via body gestures. This lighting system will also be aware of your position in the room and will respond by increasing the brightness of the lamp closest to you.
You will learn about the RGB color mode so you can build lamps like the one in Figure 7-1 that will cast a wide spectrum of colors using four LED lights: three colored (red, green, and blue) and one white. You will use wireless communication via two XBee modules, and you will work with a smaller version of your board, the Arduino Nano. The use of spatial body gesture recognition will take you to a deeper exploration of three-dimensional NITE’s skeleton tracking capabilities. Finally, you will have a first encounter with data storage and retrieval from external files, so you can save and load the layout of your lamps. Table 7-1 lists the required parts for this chapter.
RGB Color Space
Throughout this chapter, we will frequently talk about colors, hues, and color spaces. You will build the whole spectrum of colors using a combination of three LEDs (red, green, and blue), and you will add one extra white LED for intensity at certain moments. If you are familiar with any graphics software, you will easily recognize the RGB palette. If you are not so familiar with the RGB color space, you will find the following introduction pretty useful.
The RGB color model is an additive color model in which a series of basic colors (red, green, and blue) are added together to reproduce a broad spectrum of colors. The original purpose of the RGB color model was sensing, representing, and displaying images in electronic devices such as television sets and computers screens.
To form a color on an RGB display, you need three colored lights superimposed. By modifying the intensity of each color, you achieve the right mixture to represent the intended color. Zero intensity of each component forms the darkest color (no light means black), and full intensity on the three forms white. (Note that it’s difficult to achieve real white by physically mixing the light of three LEDs. You can buy RGB LED lights that are quite good at achieving this effect, but for this project you’ll use three different lights for a more dramatic effect.)
If one of the lamps has a much higher intensity than the others, the resulting light will have a hue similar to this color. If two lamps are shining with full intensity, the result will be a secondary color (cyan, magenta, or yellow). Secondary colors are formed by the sum of two primary colors with the same intensity: cyan is green + blue, magenta is red + blue, and yellow is red + green, as shown in Figure 7-2.
Figure 7-2. RGB chart
Common applications of the RGB color model are the display of colors on television sets and computer monitors. Each pixel of the screen is composed by three small and very close but still separate RGB light sources. If you stand close enough to the screen, you can distinguish the source’s colors (this means really close). Otherwise, the separate sources are indistinguishable, which tricks the eye into seeing a given solid color.
In computing, the component values are often stored as integer numbers in the range of 0 to 255, and this is the way you will work with color intensities in Arduino and Processing: going from values 0 (light off) to 255 (full intensity).
Color changes the quality of an ambience. By modifying lighting effects and hues, you can alter your perception of a space. The Chinese theory of Feng Shui takes the use of color to a more profound level, associating energies and moods according to the right combination of colors. You can apply different colors to a room according to different situations or moods, whether by following the directions of millenary Feng Shui or just your personal taste.
Arduino Nano
Due to the size of your lamp, the use of an Arduino Nano is more appropriate for this project. An Arduino Nano, shown in Figure 7-3, is a small and complete Arduino board designed and produced by Gravitec. It has some minor differences from the Arduino Uno you used in previous projects. Instead of having a standard USB socket, it has a Mini-B USB. The pins are the same as on the Arduino Uno, just male instead of female.
As you saw in Chapter 1, each of the 14 digital pins on the Nano can be used as input and output. You also have an LED on pin 13, and you can use digital pin 0 (RX) for receiving and 1(TX) for transmitting serial communication. Remember to select the correct Arduino board in the Arduino IDE.
For powering the Arduino Nano you can use the Mini-B USB connection, a 6-20V unregulated external power supply in pin 30, or a 5V regulated external power supply in pin 27. This is important because you will use a wireless connection, so both the wireless module and the Arduino board will be externally powered.
Building the Circuit
Let’s build your circle step by step. First, you will build the circuit that allows the control of the four LEDs from Arduino, and then you will make the whole installation wireless by adding a battery pack and an XBee radio module. Figure 7-4 shows the whole setup.
Get started by connecting the four LEDs (white, red, green, and blue) to your Arduino Nano, building the circuit using a strip board. The components are shown in Figure 7-5.
As usual, you start by cutting the strip board into a small enough size so you can insert the circuit inside your pencil holder but large enough to connect all the components. For the moment, you are keeping it simple—just controlling the LEDs using your Mini USB connection—but you will increase the amount of components later when you add the wireless XBee module.
Once you have cut the piece, scratch the board before soldering the pins that will hold your Arduino Nano, as shown in Figure 7-6. Double-check the direction of the copper strips on the back of the board so you don’t connect the two rows of pins to your Arduino.
Figure 7-6. Scratched strip board
After that, position your breakout headers and solder them to the strip board. You could actually place the Arduino Nano and solder it to your strip board if you want to make a permanent installation, but let’s use the breakout headers so you can easily detach the Arduino board for other projects.
In principle, you don’t need to connect all the pins to the board; you can add more later if you need them. First, solder the breakout headers that correspond to the analog pins, ground, and serial communication, and some of the digital pins on the other side to give some stability to the Arduino Nano, as shown in Figure 7-7.
Figure 7-7. Breakout headers (detail)
Resistors
The next step is to solder the resistors for the LEDs. In Chapter 4, we asked you to trust our choice of a resistor. Now that you are a bit more experienced, you will learn how to choose the right resistor according to the kind of LED you are using.
LEDs are semiconductors. The flow in an LED is an exponential function of the voltage across it, meaning that a small change in voltage can produce a huge change in current. Using a resistor, you will manage to have current and voltage linearly related. This means that a change in voltage will now produce a proportional change in current, thus limiting the current in the LED to a safe value.
Note In order to know which resistor is adequate, you can use an online resistor calculator, like http://led.linear1.org/1led.wiz, in which you can introduce the supply voltage, diode-forward voltage, and diode-rated current to obtain a suitable resistance value and the equivalent resistor color code.
Understanding Resistor Color Codes
Resistors have a series of color stripes that denote their resistance value. Normally, resistors have a four-band code. The first two bands represent the most significant digits of the resistance value. Colors are assigned to the numbers between 0 and 9, and the color bands basically translate the numbers into a visible code. Table 7-2 provides the code.
So if a resistor has brown and red as the first two bands, the most significant digits will be 1 and 2 (12). The third band is the multiplier using the power of ten (or how many zeros to add to your most significant values), using the same assigned value for each color as in the previous step. For example, if you have a resistor that has brown, red and red, that means 12 (significant digits brown =1, red =2) × 100 (multiplier, red =2 meaning 102) = 1200Ω (1.2kΩ).
The last band is called the tolerance band (the deviation from the specified value) and it’s usually spaced away from the others. Gold means 5% of tolerance and silver means 10% of tolerance; if there’s no tolerance band that means a 20% of tolerance.
The values of your LEDs from the distributor’s specifications are as follows:
- White LED: Supply voltage 5V, forward voltage 3.3V, rated current 30mA, meaning a resistor of 68 ohms (color code: blue, grey, black).
- Red LED: Supply voltage 5V, forward voltage 1.85V, rated current 30mA, meaning a resistor of 120 ohms (color code: brown, red, brown).
- Green LED: Supply voltage 5V, forward voltage 3.3V, rated current 25mA, meaning a resistor of 150 ohms (color code: brown, green, brown).
- Blue LED: Supply voltage 5V, forward voltage 3.3V, rated current 30mA, meaning a resistor of 68 ohms (color code: blue, grey, black).
Now that you know your resistors, you can solder them to your board. You’re going to use pins 5, 9, 10, and 11 for your LEDs (white, red, green, and blue), and scratch the strip board to separate all grounds, as shown in Figures 7-8 and 7-9.
Figure 7-8. Resistors soldered on board
Figure 7-9. Resistors soldered on board (back)
The next step is to connect all grounds together and back to Arduino’s ground, and then solder the LEDs to your circuits, as shown in Figure 7-10 and 7-11. Make sure that the right color is connected to the right pin; also remember that the shorter legs on the LEDs (cathode) connect to ground and the longer legs connect to the resistors.
Figure 7-11. LEDs on circuit (back)
Testing the Circuit
Now that everything is soldered, you can use a simple piece of code to test the circuit. This code programs the LEDs to dim at different speeds so you can appreciate the richness and variety of the colors resulting from the mix. The code is based on the Fade example from Arduino; it’s been modified to drive more than one LED.
int whiteLED = 5;
int redLED = 9;
int greenLED = 10;
int blueLED = 11;
int Wbr = 0;
int Rbr = 0;
int Gbr = 0;
int Bbr = 0;
int Wspeed = 0;
int Rspeed = 2;
int Gspeed = 3;
int Bspeed = 5;
void setup() {
pinMode(whiteLED, OUTPUT);
pinMode(redLED, OUTPUT);
pinMode(greenLED, OUTPUT);
pinMode(blueLED, OUTPUT);
}
void loop() {
analogWrite(whiteLED, Wbr);
analogWrite(redLED, Rbr);
analogWrite(greenLED, Gbr);
analogWrite(blueLED, Bbr);
Wbr += Wspeed;
if (Wbr == 0 || Wbr == 255) {
Wspeed *= -1 ;
}
Rbr += Rspeed;
if (Rbr == 0 || Rbr == 255) {
Rspeed *= -1 ;
}
Gbr += Gspeed;
if (Gbr == 0 || Gbr == 255) {
Gspeed *= -1 ;
}
Bbr += Bspeed;
if (Bbr == 0 || Bbr == 255) {
Bspeed *= -1 ;
}
delay(30);
}
Run this code and watch your lamp change colors smoothly (Figure 7-12). Each LED changes its intensity at a different speed. You can tweak the code and change the speed of each lamp at will. The lamp looks good now, but you still have a Mini USB cable attached to it. The USB cable brings power and data to the Arduino, so removing it means you need to plug an external power source and a wireless module to the board, as shown in Figure 7-13.
Figure 7-13. XBee and battery clip
To substitute the USB power on your freestanding module, connect a battery clip to your circuit. But first you need to solder a new header connecting pin 30 in your Arduino (Vin, a 6-20V unregulated external power supply). Then connect the red cable in the battery clip (power) to this new pin and the black cable (ground) to the other grounds (Figure 7-14).
Figure 7-14. Battery clip
Connect the battery holder to the battery clip and observe that the lights behave the same way as when the mini USB cable is connected. The code was already uploaded from the previous test and Arduino just followed the routine without receiving any external data from the computer.
XBee Wireless Module
You’re now going to add an XBee to the circuit. XBee is a brand name from Digi International for a family of radio modules. These wireless modules allow your Arduino to communicate wirelessly with your computer, avoiding the inconvenience of having cables sticking out of your beautiful lamps.
XBee pins are spaced 2 mm apart, which means that XBee can’t be directly connected to a breadboard. The XBee modules must be first connected to an XBee Explorer USB (broadcaster) and to an XBee Explorer Regulated (receiver).
The first step is to install the drivers in order for your computer to recognize the XBee module. The XBee Explorer uses a FTDI chip to convert the serial transmission to USB signals. The FTDI drivers can be found at http://www.ftdichip.com/FTDrivers.htm. Select the right platform and install the drivers. After installing the drivers and plugging in your XBee Explorer mini USB, you should be able to see your device using Terminal (ls /dev/tty.*) if you are using a Mac or by going to Start/Control panel/Hardware and Sound/Device Manager/ Universal Serial Bus controllers if you are a Windows user.
Once you have your broadcaster installed, you need to connect your receiver to the Arduino board. You could plug your XBee into a shield that connects the module directly to Arduino, but for this project you are going to use the normal XBee Explorer so you can analyze the connections.
XBee has many pins but it takes only five of them to create a working wireless connection with Arduino via its built-in serial communication protocol. The five connections that you have to make between the Arduino board and the XBee Explorer Regulated are described in Table 7-3 and shown in Figures 7-15 through 7-17.
Figure 7-15. XBee Explorer connected to Arduino
Figure 7-16. Strip board with serial connections
Figure 7-17. Strip board with serial connections (back)
The last step is to power your XBee, so pick a cable for power from the 3V Arduino pin to the XBee Explorer’s VCC or 3.3V, and connect Arduino’s ground to the XBee Explorer’s ground.
Once you have the circuit assembled (Figure 7-18), pile it so it fits it inside your plastic pencil holder, building your lamp (Figure 7-19).
Tip A piece of advice: perforate a hole on top of the pencil holder so if the components get stuck inside, you can push them out.
Figure 7-19. Pencil holder
After assembling the lamp, the next step is to use the Kinect to control the light’s colors. You will learn how to control several lights at the same time, implementing code that will work on any number of lamps, from one to as many as you can build. If you feel patient, build a couple more of them!
Arduino Programming
Now it’s time to implement and upload the Arduino code to the board, so you can move on to programming the user-lamp interaction in Processing. The following Arduino code declares the LED pin variables and four integers (Wval, Rval, Gval, and Bval) that store the brightness level of each LED. You are building several lamps, so you will declare an integer called lampNumber that determines which lamp it is. You need to change this number before uploading the code to every lamp’s Arduino, starting from zero and counting up.
int whiteLED = 5;
int redLED = 9;
int greenLED = 10;
int blueLED = 11;
int WVal, RVal, GVal, BVal;
int lampNumber = 0;
Within the setup() function, start the serial communication and set the four pins as output.
void setup() {
Serial.begin(9600);
pinMode(whiteLED, OUTPUT);
pinMode(redLED, OUTPUT);
pinMode(greenLED, OUTPUT);
pinMode(blueLED, OUTPUT);
}
Within the loop() function, check if you have enough data in the serial buffer and read the first value. If this value is the event trigger ‘S’, read the next piece of data, which you interpret as the number of the lamp for which the communication is intended. If it’s equal to this lamp’s lampNumber, read the following four numbers and apply their values directly to the four LEDs connected to the Arduino.
void loop(){
if (Serial.available()>5) { char val = Serial.read();
if(val == 'S'){
int messageNumber = Serial.read();
if(messageNumber == lampNumber){
WVal = Serial.read();
RVal = Serial.read();
GVal = Serial.read();
BVal = Serial.read();
}
}
}
analogWrite(whiteLED, WVal);
analogWrite(redLED, RVal);
analogWrite(greenLED, GVal);
analogWrite(blueLED, BVal);
}
The Lamp Class
The goal of this chapter is to create a lighting installation that is aware of the users and behaves according to their position in space. The lamps will also be controlled by intentional body gestures performed by the users. You’re going to spend the rest of this chapter developing the software that will gather user information from Kinect and translate it into precise orders for each of your lamps, which will be broadcast with an XBee module attached to your computer.
At this moment, you have presumably fabricated a few lamps, each equipped with an XBee wireless module, a battery clip, and an Arduino Nano with specific code on board. It would be very useful for you to create a piece of code in Processing that could define the present state of each lamp and allow you to set each of the lamps’ parameters according to the information from the Kinect. This code structure should be implemented as a class.
This is the first class you are implementing from scratch in the book, so it’s a good moment to give you a refresher on object-oriented programming (OOP). If this sounds scary, don’t panic. You have been using OOP throughout all of the previous chapters and projects; we just haven’t talked about it explicitly. OOP is a vast and complex subject, and it would be out of the scope of this book to attempt a thorough introduction, but we’re hoping this will be a reasonable starting point for you to get acquainted with the topic so you can understand the following chapters.
Object-Oriented Programming
First of all, object-oriented programming is a programming paradigm, or programming style, based on the use of objects for the design of computer programs. There is a long list of available programming paradigms, and they all differ in the abstractions used to represent the elements of a program. This means they are different in the way they think about the computation, not in the type of computation they perform. You can tackle the same problem using different programming paradigms, and thus many programming languages allow the implementation of code using several of them.
OOP uses classes and objects, which are instances of classes, to structure the code. Classes are used as blueprints to create objects to represent the elements in the code. In the next pages, you will use classes, fields, constructors, and methods to implement the class Lamp, which represents the lamps you have just built and programmed in the previous sections. This class allows you to create instances of your lamp objects, and store and affect their positions and RGB values in a structured way.
Java, the programming language on which Processing is based, is an object-oriented programming language and as such, every time you programmed in Processing, you were using OOP. In Java, the convention is to store each class in a separate file, and in Processing this translates to writing each class in a separate tab (Figure 7-20). This will automatically create a new .pde file in your sketch folder with the name of the tab.
Figure 7-20. Creating a new tab in Processing
Class Declaration
The first line of code for the new class starts with the word “class,” followed by the name you have chosen for the class. It is good coding practice to start class names with a capital letter. All the rest of the class code is contained within the curly brackets that you open after the class’s name.
class Lamp {
// Your code goes here
}
Field Description
After declaring your class, you declare the fields, or properties of the class. In your case, you want to be able to control the position of each lamp object and the brightness of its four LEDs, so you declare one PVector to keep track of the lamp’s position and four integers for LED brightness.
class Lamp {
PVector pos;
int R, G, B, W;
In Java, classes, fields, and methods can be preceded by access specifiers defining the visibility of the members. The access specifier is a keyword preceding the declaration of the element. There are three main types of visibility in Java.
private: The field can only be accessed only by the class.protected: the field can be accessed by the class and its subclasses.public: The member can be accessed from anywhere.
Note In Processing, the using of public members is encouraged at all times, so the visibility of every member will be public unless otherwise specified. This is the reason why you normally won’t be explicitly defining the visibility of a member in your code.
Constructor
The constructor is a subroutine or method that has the same name as the class and doesn’t have an explicit return type. The constructor is called at the creation of an object, and it initializes the data members (fields) of the class. There can be multiple constructors for the same class as long as they take different sets of parameters at the creation of the objects.
In your Lamp class, you only need one constructor; it will take three floats as parameters. The constructor initializes the PVector defining the position of the lamp to the x, y, and z parameter values and sets the blue LED to full brightness so you can see that the lamp has been initialized.
Lamp(float x, float y, float z) {
this.pos = new PVector(x, y, z);
this.B = 255;
}
Methods
Methods are functions declared within the class. Public methods can be accessed from other classes, and they can be declared as void if you don’t need them to return a value or any kind of data. Getters and setters are just special ways of calling methods used to change or acquire a specific property of the class. They are sometimes called accessor methods.
The first method in your class sets the lamp’s RGB parameters. It takes three integers and directly applies them to the R, G, and B fields of your object.
public void setColor(int r, int g, int b) {
R = r;
G = g;
B = b;
}
You also need the current state of the LEDs in the lamp, so you implement a function that returns an array of three integers containing the RGB values.
public int[] getColor() {
int[] colors = { R, G, B };
return colors;
}
Although you won’t be using this function in your code, it could be useful for some other application to be able to change the lamp’s position from another class, including your main Processing draw() loop, so implement a setter for your lamp’s position PVector.
Implement a draw() method that takes care of the display of the lamp on screen. It’s called from the main draw() loop, so you can see your lamps at their right position and colors. Note that this function could be called render(), display(), or any other name that you feel is more descriptive.
public void draw() {
pushMatrix();
translate(pos.x, pos.y, pos.z);
pushStyle();
if (W == 255) {
fill(W);
}
else {
fill(R, G, B);
}
noStroke();
box(100, 150, 100);
popStyle();
popMatrix();
}
Figure 7-21. The lamp closer to the user will glow at full intensity
The method updateUser() takes the user’s position in space and checks if the lamp is closer than a certain threshold distance. If it is, it makes the white LED glow at full brightness, assuming that the user could use a bit of light for reading or finding his way around (Figure 7-21).
public void updateUserPos(PVector userCenter) {
float dist = pos.dist(userCenter);
if (dist < 1000) {
W = 255;
}
else {
W = 0;
}
stroke(200);
line(pos.x, pos.y, pos.z, userCenter.x, userCenter.y, userCenter.z);
}
Finally, drawSelected() is called when the user has successfully selected a lamp by pointing at it, and it will display a gizmo around the lamp so you know the lamp is selected.
public void drawSelected() {
pushMatrix();
translate(pos.x, pos.y, pos.z);
pushStyle();
noFill();
stroke(R, G, B);
box(150, 225, 150);
popStyle();
popMatrix();
}
Remember that all this code should be within the curly braces you opened after your class declaration. After you close the last method, there should be a closing curly brace to close the class.
User Control Sketch
You have implemented a Lamp class that you can use to represent your physical lamps in the code. Now you need to write the necessary routines to allow for gesture-based user control and the sending of serial data from your computer. You’re going to use NITE’s user and skeleton tracking capabilities for this purpose.
You encountered skeleton tracking in previous projects, but this time you are going to be using three-dimensional skeleton tracking to check where the user is pointing in space. You’re also going to implement a simple gesture-recognition routine to trigger the creation of lamps.
First and foremost, you need to tell Processing to import all the libraries that you will be using in the project. These are Processing’s OpenGL and Serial libraries, KinectOrbit, and Simple-OpenNI.
import processing.opengl.*;
import kinectOrbit.KinectOrbit;
import processing.serial.Serial;
import SimpleOpenNI.SimpleOpenNI;
Variable Declaration
You now need to declare all the libraries’ objects and create a File object where you store your lamp’s positions and colors. This is detailed in the data storage and retrieval section.
SimpleOpenNI kinect;
KinectOrbit myOrbit;
Serial myPort;
File lampsFile;
The Boolean serial defines if you are initializing and using serial communication. The userID integer keeps track of the current user, and the PVector userCenter contains the current position of the user. You also need to create an ArrayList of Lamp objects so it’s easy to add more lamps in runtime.
boolean serial = true;
int userID;
PVector userCenter = new PVector();
ArrayList<Lamp> lamps = new ArrayList<Lamp>();
Setup() Function
The setup() function sets the frame’s size and renderer, initializes the Simple-OpenNI object, and enables the Mirror, depth map, and user capabilities.
public void setup() {
size(800, 600, OPENGL);
smooth();
kinect = new SimpleOpenNI(this);
kinect.setMirror(true);
kinect.enableDepth();
kinect.enableUser(SimpleOpenNI.SKEL_PROFILE_ALL);
The KinectOrbit object is initialized now, passing this as a parameter, and you set the gizmo drawing to true. The file lampsFile is also initialized now; it’s important that you do it now instead of doing it in the Variable Declaration section, as otherwise the data path won’t refer to the data folder in your Processing sketch.
myOrbit = new KinectOrbit(this, 0);
myOrbit.drawGizmo(true);
lampsFile = new File(dataPath("lamps.txt"));
To finish the setup(), initialize the serial communication if you have set your serial variable to true.
if (serial) {
String portName = Serial.list()[0];
myPort = new Serial(this, portName, 9600);
}
}
Draw() Function
The first lines in your draw() function update the data from Kinect and set the background color to black. You then push the KinectOrbit so you can control the camera view with your mouse and call the function drawPointCloud(), which draws the raw point cloud from Kinect on screen. You only display one every six points for speed.
Figure 7-22. User recognized by Kinect
Then you check if there is a user in the Kinect’s field of view, and if so, you get his center of mass using the Simple-OpenNI function getNumberOfUsers(), which sets the second parameter (your PVector userCenter) to the center of mass of the user userID (Figure 7-22). You use the center of the user’s mass to update each of the lamps in your ArrayList.
if (kinect.getNumberOfUsers() != 0) {
kinect.getCoM(userID, userCenter);
for (int i = 0; i < lamps.size(); i++) {
lamps.get(i).updateUserPos(userCenter);
}
Then, if apart from having detected a user, you are also tracking his skeleton, you call the function userControl(), and you draw the skeleton on screen. If you’re not tracking a skeleton, you simply print the user’s point cloud on screen for reference (Figure 7-23).
if (kinect.isTrackingSkeleton(userID)) {
userControl(userID);
drawSkeleton(userID);
}
else {
drawUserPoints(3);
}
}
Figure 7-23. Use point cloud (left) and user skeleton (right)
Now you draw each lamp on screen and the Kinect camera frustum, and you pop the KinectOrbit object. If the serial Boolean is true, you call the sendSerial() function.
for (int i = 0; i < lamps.size(); i++) {
lamps.get(i).draw();
}
kinect.drawCamFrustum();
myOrbit.popOrbit();
if (serial) {
sendSerialData();
}
}
User Control
The function userControl() performs the computations that are most specific to this project. This function gets the user’s right arm position from Kinect and uses the joint alignment to check whether the user is pointing at a lamp or not. Then it uses the left hand position to change the color of the selected lamp. This function also contains the implementation of the lamp creation gesture recognition.
First, you create PVectors to store the three-dimensional positions of the body joints you need for your user control. Then you use the Simple-OpenNI function getJointPositionSkeleton() to get the joint’s coordinates and set your vectors to these positions.
private void userControl(int userId) {
PVector head = new PVector();
// Right Arm Vectors
PVector rHand = new PVector();
PVector rElbow = new PVector();
PVector rShoulder = new PVector();
// Left Arm Vectors
PVector lHand = new PVector();
// Head
kinect.getJointPositionSkeleton(userId, SimpleOpenNI.SKEL_HEAD, head);
// Right Arm
kinect.getJointPositionSkeleton(userId, SimpleOpenNI.SKEL_RIGHT_HAND,
rHand);
kinect.getJointPositionSkeleton(userId, SimpleOpenNI.SKEL_RIGHT_ELBOW,
rElbow);
kinect.getJointPositionSkeleton(userId,
SimpleOpenNI.SKEL_RIGHT_SHOULDER, rShoulder);
// Left Arm
kinect.getJointPositionSkeleton(userId, SimpleOpenNI.SKEL_LEFT_HAND,
lHand);
Lamp Control
The way you check if the user is pointing at a lamp is by checking that his right arm is stretched; if it is, you check where in space it is pointing. For this, you need the vectors defining the user’s forearm and arm; these are easily found by subtracting the vector defining the shoulder position from the one defining the elbow position, and doing the same operation with the elbow and the hand PVectors (Figure 7-24).
PVector rForearm = PVector.sub(rShoulder, rElbow);
PVector rArm = PVector.sub(rElbow, rHand);
Figure 7-24. Right arm angles and lamp
Once you have these vectors, you can check whether the user’s right arm is stretched. You do this by testing if the angle between the two PVectors (the complementary of the elbow angle) is lesser than the threshold you have set to PI/8, or 22.5 degrees. If the right arm is indeed stretched, you test the angle between the right arm’s PVector and the angle defining the direction from the right hand to the lamp’s position. The smaller the arm-lamp angle is, the more directly the right hand is pointing at the lamp. You have set this threshold to PI/4, or 45 degrees, so it’s easier for the user to select a lamp.
If you are under the threshold, you use the Lamp object’s method setColors(), passing as a parameter the halved x, y, and z distances of the left hand to the user’s head. This way the user can change the color parameters of the lamp using unique body gestures (Figure 7-25).
Figure 7-25. User control of a lamp’s RGB values
if (PVector.angleBetween(rForearm, rArm) < PI / 8f) {
for (int i = 0; i < lamps.size(); i++) {
PVector handToLamp = PVector.sub(rHand, lamps.get(i).pos);
if (PVector.angleBetween(rArm, handToLamp) < PI / 4) {
PVector colors = PVector.sub(head, lHand);
lamps.get(i).setColor((int) colors.x / 2,
(int) colors.y / 2, (int) colors.z / 2);
lamps.get(i).drawSelected();
}
}
}
Lamp Creation
Until now, we have been talking about lamps that you haven’t created yet. You have a series of physical lamps that you want to be able to control from your code, and you have implemented a way to control virtual lamps in your program. Now, if you can make the coordinates of your virtual lamps coincident to the physical lamps, you’ll be getting somewhere.
The way you do this is by implementing gesture recognition that allows the user to tell the program where the lamps are placed physically. You basically place the lamps in the space and then stand by each of the lamps and tell the computer, “Hey, here is a lamp!”
This gesture should be something the user can do without too much effort but one that is not common enough to trigger a random series of lamps every minute. The gesture we came up with is touching your head with two hands, which is a reasonably good balance between easiness and rareness; it also has the advantage of being extremely easy to detect. You only need to check the distances of the right and left hands to the head; if both are smaller than 200mm you create a lamp positioned at the user’s center of mass. This is not their exact location, but it’s a reasonably good approximation.
There is one more test you need to perform. If you don’t explicitly avoid it, the program will create lamps at every loop the whole time your hands are on your head. To avoid this, set a proximity limit so if the lamp about to be created stands closer than 200mm to any other lamp, it won’t be created. This way, even if you keep your hands on your head for a minute, only one lamp will be created until you move away by the specified distance (Figure 7-26).
if (head.dist(rHand) < 200 && head.dist(lHand) < 200) {
boolean tooClose = false;
for (int i = 0; i < lamps.size(); i++) {
if (userCenter.dist(lamps.get(i).pos) < 200) {
tooClose = true;
}
}
if (!tooClose) {
Lamp lampTemp = new Lamp(userCenter.x, userCenter.y,
userCenter.z);
lamps.add(lampTemp);
}
}
}
Figure 7-26. Lamp generation by body gestures
Data Storage and Retrieval
You have installed your lamps all over your living room, and you have told your program where each of them is located by using your lamp-creation gesture. But you don’t want to have to do this every time you restart your sketch. It would be great to be able to save the current lamp layout together with their individual states to an external file that can be retrieved in future sessions. This is the purpose of the functions saveLamps() and loadLamps().
You use the previously defined lampsFile file to store the information, employing the Processing function saveStrings() for this purpose. This function allows you to save an array of strings into a file with just one line of code. The first parameter that the function takes is the file you want to save your data to and the second one is the array of strings.
In your case, you need first to transform each of your lamps’ position coordinates and RGB values to strings, and then store the information for each lamp in one line of code, separating the data with blank spaces so you can find them later.
void saveLamps() {
String[] lines = new String[lamps.size()];
for (int i = 0; i < lamps.size(); i++) {
lines[i] = String.valueOf(lamps.get(i).pos.x) + " "
+ String.valueOf(lamps.get(i).pos.y) + " "
+ String.valueOf(lamps.get(i).pos.z) + " "
+ Integer.toString(lamps.get(i).getColor()[0]) + " "
+ Integer.toString(lamps.get(i).getColor()[1]) + " "
+ Integer.toString(lamps.get(i).getColor()[2]);
}
saveStrings(lampsFile, lines);
}
The function loadLamps() performs the inverse task to the previous one. You use the Processing function loadStrings() to save each line in the file to a string in the array of strings stringArray, and then you split each line at the blank spaces and convert each substring into floats defining the position of the lamp and three integers defining its color. You then create the necessary lamps and set their color values to the ones loaded from the file.
void loadLamps() {
String lines[] = loadStrings(lampsFile);
for (int i = 0; i < lines.length; i++) {
String[] coordinates = lines[i].split(" ");
Lamp lampTemp = new Lamp(Float.valueOf(coordinates[0]),
Float.valueOf(coordinates[1]),
Float.valueOf(coordinates[2]));
lampTemp.setColor(Integer.valueOf(coordinates[3]),
Integer.valueOf(coordinates[4]),
Integer.valueOf(coordinates[5]));
lamps.add(lampTemp);
}
}
These two functions are called when you press the s and l (the letter l, not number 1) keys on your keyboard. The Processing callback function keyPressed() takes care of invoking them when a new keyboard event is triggered.
public void keyPressed() {
switch (key) {
case 's':
saveLamps();
break;
case 'l':
loadLamps();
break;
}
}
Serial Communication
The serial function performs the task of sending the necessary data to coordinate the physical lamps to the ones you are seeing on your screen. For each lamp you send the event trigger ‘S’, followed by the number of the lamp the message is intended for, and then the W, R, G, and B values.
void sendSerialData() {
for (int i = 0; i < lamps.size(); i++) {
myPort.write('S'),
myPort.write(i);
myPort.write(lamps.get(i).W);
myPort.write(lamps.get(i).R);
myPort.write(lamps.get(i).G);
myPort.write(lamps.get(i).B);
}
}
Display Functions
The following functions are used to print the Kinect point clouds and user skeleton on screen. You saw how to display the Kinect point cloud in Chapter 3.
void drawPointCloud(int steps) {
int[] depthMap = kinect.depthMap();
int index;
PVector realWorldPoint;
stroke(255);
for (int y = 0; y < kinect.depthHeight(); y += steps) {
for (int x = 0; x < kinect.depthWidth(); x += steps) {
index = x + y * kinect.depthWidth();
if (depthMap[index] > 0) {
realWorldPoint = kinect.depthMapRealWorld()[index];
point(realWorldPoint.x, realWorldPoint.y, realWorldPoint.z);
}
}
}
}
The drawing of the user points builds on the same principles as the previous function, but you sort the depth map points according to the color of the equivalent point in the user map. The user map is an array of pixels with value 0 if the Kinect determines they are background pixels, and a higher value, starting at 1, for the pixels defining the users.
This function allows you to print the user’s pixels in a different color, so you can observe on screen what Kinect is considering to be your user.
void drawUserPoints(int steps) {
int[] userMap = kinect.getUsersPixels(SimpleOpenNI.USERS_ALL);
// draw the 3d point depth map
PVector[] realWorldPoint = new PVector[kinect.depthHeight() * kinect.depthWidth()];
int index;
pushStyle();
stroke(255);
for (int y = 0; y < kinect.depthHeight(); y += steps) {
for (int x = 0; x < kinect.depthWidth(); x += steps) {
index = x + y * kinect.depthWidth();
realWorldPoint[index] = kinect.depthMapRealWorld()[index].get();
if (userMap[index] != 0) {
strokeWeight(2);
stroke(0, 255, 0);
point(realWorldPoint[index].x, realWorldPoint[index].y,
realWorldPoint[index].z);
}
}
}
popStyle();
}
The next two functions are from Simple-OpenNI’s 3d skeleton tracking example. They both use functions specific to Simple-OpenNI to draw the user’s limbs on screen.
void drawSkeleton(int userId) {
strokeWeight(3);
// to get the 3d joint data
drawLimb(userId, SimpleOpenNI.SKEL_HEAD, SimpleOpenNI.SKEL_NECK);
drawLimb(userId, SimpleOpenNI.SKEL_NECK,
SimpleOpenNI.SKEL_LEFT_SHOULDER);
drawLimb(userId, SimpleOpenNI.SKEL_LEFT_SHOULDER,
SimpleOpenNI.SKEL_LEFT_ELBOW);
drawLimb(userId, SimpleOpenNI.SKEL_LEFT_ELBOW,
SimpleOpenNI.SKEL_LEFT_HAND);
drawLimb(userId, SimpleOpenNI.SKEL_NECK,
SimpleOpenNI.SKEL_RIGHT_SHOULDER);
drawLimb(userId, SimpleOpenNI.SKEL_RIGHT_SHOULDER,
SimpleOpenNI.SKEL_RIGHT_ELBOW);
drawLimb(userId, SimpleOpenNI.SKEL_RIGHT_ELBOW,
SimpleOpenNI.SKEL_RIGHT_HAND);
drawLimb(userId, SimpleOpenNI.SKEL_LEFT_SHOULDER,
SimpleOpenNI.SKEL_TORSO);
drawLimb(userId, SimpleOpenNI.SKEL_RIGHT_SHOULDER,
SimpleOpenNI.SKEL_TORSO);
drawLimb(userId, SimpleOpenNI.SKEL_TORSO, SimpleOpenNI.SKEL_LEFT_HIP);
drawLimb(userId, SimpleOpenNI.SKEL_LEFT_HIP,
SimpleOpenNI.SKEL_LEFT_KNEE);
drawLimb(userId, SimpleOpenNI.SKEL_LEFT_KNEE,
SimpleOpenNI.SKEL_LEFT_FOOT);
drawLimb(userId, SimpleOpenNI.SKEL_TORSO, SimpleOpenNI.SKEL_RIGHT_HIP);
drawLimb(userId, SimpleOpenNI.SKEL_RIGHT_HIP,
SimpleOpenNI.SKEL_RIGHT_KNEE);
drawLimb(userId, SimpleOpenNI.SKEL_RIGHT_KNEE,
SimpleOpenNI.SKEL_RIGHT_FOOT);
strokeWeight(1);
}
void drawLimb(int userId, int jointType1, int jointType2) {
PVector jointPos1 = new PVector();
PVector jointPos2 = new PVector();
float confidence;
// draw the joint position
confidence = kinect.getJointPositionSkeleton(userId, jointType1,
jointPos1);
confidence = kinect.getJointPositionSkeleton(userId, jointType2,
jointPos2);
stroke(255, 0, 0, confidence * 200 + 55);
line(jointPos1.x, jointPos1.y, jointPos1.z, jointPos2.x, jointPos2.y,
jointPos2.z);
}
Simple-OpenNI Callbacks
The Simple-OpenNI callbacks are the same functions you used in the previous chapter. The only point to notice is that you update the userID integer to the last user you detected in theonNewUser() function.
Speaking of classes and member visibility, you should add the keyword “public” before every callback function. This is unnecessary in Processing, but if you are using another IDE, such as Eclipse, you will need to set all the callbacks as public or Simple-OpenNI won’t invoke them.
public void onNewUser(int userId) {
println("onNewUser - userId: " + userId);
println(" start pose detection");
kinect.startPoseDetection("Psi", userId);
userID = userId;
}
public void onLostUser(int userId) {
println("onLostUser - userId: " + userId);
}
public void onStartCalibration(int userId) {
println("onStartCalibration - userId: " + userId);
}
public void onEndCalibration(int userId, boolean successfull) {
println("onEndCalibration - userId: " + userId + ", successful: " + successfull);
if (successfull) {
println(" User calibrated !!!");
kinect.startTrackingSkeleton(userId);
}
else {
println(" Failed to calibrate user !!!");
println(" Start pose detection");
kinect.startPoseDetection("Psi", userId);
}
}
public void onStartPose(String pose, int userId) {
println("onStartdPose - userId: " + userId + ", pose: " + pose);
println(" stop pose detection");
kinect.stopPoseDetection(userId);
kinect.requestCalibrationSkeleton(userId, true);
}
public void onEndPose(String pose, int userId) {
println("onEndPose - userId: " + userId + ", pose: " + pose);
}
Summary
You were introduced to a broad range of topics in this chapter. You learned how to implement three-dimensional skeleton tracking and use it to perform custom gesture recognition. You also had a refresher on object-oriented programming, and you learned how to write a new class from scratch.
On the hardware side of things, you built Arduino-based autonomous devices, each with its own power source and wireless module, which permitted the communication of the units with your computer. You also learned about the RGB color space and that you can reproduce the whole color spectrum with only three RGB LEDs (Figure 7-27). You also learned how to calculate the appropriate resistors for the LEDs. In the end, you were able to control a whole lighting system with body gestures and make it aware of your position in space so it could respond by lighting the closest lamps to you with a higher intensity.
Figure 7-3. RGB lamp color change
You can think of this chapter as a general introduction to building any sort of application for intelligent houses. Imagine applying this sort of gesture control to any of your home appliances. The introduction of the XBee wireless module adds a new set of possibilities for using Arduino remotely, and you will explore these possibilities in future chapters of this book.