C H A P T E R 11
3D Modeling Interface
So far you’ve used Kinect to accurately track your movements in space, but when it comes to “clicking,” you can’t seem to dispense with your mouse. In this project, you’re going to build a wearable user interface that will complement Kinect’s tracking capabilities with accurate multi-clicking and you will use it to build 3D geometries by drawing directly in space.
Your user interface will be mounted on a standard glove and will use an Arduino LilyPad as its computational core. You will learn how to use the bending of your fingers as an input device via flex sensors. You will then learn how to develop your own simple CAD program to translate the position of your hand and the bending of your fingers into 3D geometries (as shown in Figure 11-1) that you will then export as .dxf files readable by other CAD and 3D-modeling packages.
Figure 11-1. 3D modeling with the glove interface
The Interface
In previous chapters, you explored several ways of interacting with your computer using Kinect’s 3D scanning plus NITE’s hand and skeleton tracking capabilities. But the “clicking” precision using gesture recognition is rather inaccurate and can be discouraging for some applications.
In this chapter, you are going to bridge this gap by building a wearable wireless user interface that can be used as a 3D interface for several applications (Figure 11-2). You will take advantage of Kinect’s excellent 3D hand-tracking capability, and you will complement it with a way to trigger different behaviors in your program without using conventional keyboard and mouse devices.
Figure 11-2. 3D modeling interface
Table 11-2 lists all the components necessary for this project and Figure 11-3 shows what they look like. If you need to buy all of the Arduino LilyPad stuff, we recommend the LilyPad Beginner’s Kit from SparkFun, which includes most of the components for this project plus other parts you may find useful later.
Note You are using the Arduino LilyPad Simple in this project. As a result, you also need to use Arduino’s SofwareSerial library to communicate with the XBee radio module, as you can’t access pins 0 and 1 on the LilyPad Simple. This way of transmitting information is actually much slower than using pins 0 and 1, but it’s good enough for this project. If you are planning on adding more demanding transmission of information to this project, you should consider using the standard Arduino LilyPad board. For more information, see the “Going Wireless” section.
Figure 11-3. Project parts
Arduino LilyPad
When working on wearable technology with Arduino, it’s difficult to find a better option than the Arduino LilyPad, an Arduino board developed cooperatively by Leah Buechley and SparkFun Electronics specifically to be sewn to fabric and clothing for wearable applications.
For this project, you are going to use the Arduino LilyPad Simple board. This board has fewer connections than the standard Arduino LilyPad, but it includes a built-in power socket for plugging in a LiPo battery plus an on/off switch. This will simplify the circuit by eliminating the need to sew an extra power supply to your glove.
Note that there’s no USB connection on the LilyPad board (Figure 11-4). To connect the board to the computer, you will need to use an FTDI Basic Breakout that plugs into the 6-pin male header on the top of the LilyPad board. The FTDI Basic Breakout (Figure 11-5) can be plugged to the computer with a USB MiniB cable.
Figure 11-4. Arduino LilyPad Simple board
Figure 11-5. FTDI Basic Breakout
Flex Sensors
Flex sensors are electronic components that change resistance when bent. As with any other kind of variable resistors, they must be used with a voltage divider to get changing voltages into your analog input pins (Figure 11-6). This kind of sensors has been used extensively in the fabrication of virtual reality gloves.
Figure 11-6. Voltage divider for flex sensor
For this project, you will use four two-directional Bi-Flex Sensors from Images Scientific Instruments. These sensors decrease their resistance as they are bent in either direction. (This feature is something you don’t really need for the project, as your fingers only bend in one direction, so you can choose any other flex sensors as long as they fit nicely into the glove’s fingers.)
There are three versions of the FLX-01 sensors, depending on the resistance ranges. This project calls for the FLX-01-M (medium resistance range of 20K-50K) and 10K resistors for each sensor, which provides a reasonable input range. The kind of circuit you are using for each sensor is called a voltage divider (see Figure 11-6). We discussed the calculation of output voltages and resistors for voltage dividers in the section “Feedback from Arduino” in Chapter 4. Go back to this section if you need a refresher.
The first thing you need to do is to install the flex sensors in the glove. There are many ways of doing this. Our gloves had a double skin, so we decided to insert the sensors between the fake-leather exterior skin and the fluffy interior one. After measuring the length of the sensor from the tip of each finger, cut some small incisions, just large enough for the sensors to slip into. Then insert one sensor in each finger, leaving the cables to protrude from the slits. Only use sensors for the thumb, index, middle, and ring fingers (Figure 11-7). Leave the little finger out because of the difficulty of controlling it independently (unless you are a piano player!) and because there are only four accessible analog pins on the LilyPad Simple.
Figure 11-7. Flex sensors installed in the glove
Sewing the Circuit
Yes, that’s the right word. This time you won’t be “building” your circuit; you’ll be “sewing” it (Figure 11-8). For wearable applications, you use conductive thread instead of normal wires to connect the components in the circuit; the result is a more flexible, stable, and lightweight design. A piece of advice: if you have never sewn a button in your life, you may want to get some help from more experienced relatives or friends before you put a needle in your hand!
Figure 11-8. Sewing the circuit
The main idea is to link the LilyPad pins to the right components with the conductive thread. Note that you want to avoid the threads touching each other, just as you would do on a normal circuit. Sometimes you will need to cross threads but you don’t want them to join electrically. The best way around this is to arrange for them to be on different sides of the skin at the crossing point.
Start by sewing a test circuit using your Arduino LilyPad and the four flex sensors. After proper testing, you can add other components.
First, sew the Arduino LilyPad to the glove, and then sew the connections to the flex sensors and the resistors (Figure 11-9). To connect the flex sensors, expose a section of the cables and loop around the exposed wires with your conductive thread. Then cut off the rest of the cable. If you look carefully at Figure 11-10, you can see that the ground thread connecting all the resistors together goes under the exterior skin most of the time so that it doesn’t connect to the cables going from the flex sensors to the Arduino LilyPad pins.
Figure 11-9. Flex sensors circuit
Figure 11-10. The circuit sewn to the glove
Test that the connections are correct using a multimeter and then connect the board to your computer using the FTDI Basic Breakout. Now you can load and test your first program (Figure 11-11).
Figure 11-11. The LilyPad connected to the FTDI Basic Breakout
Testing the Circuit
Now that you have sewn the circuit, you need to check that everything works properly. To do so, you will upload an Arduino program to your board to read the input from the flex sensors and send it via serial so you can see the values on the screen.
Arduino Serial Sender
Open a new Arduino sketch and declare four integer variables, one for each finger. In the setup() function, start serial communication at 9600 baud.
int finger1, finger2, finger3, finger4;
void setup() {
Serial.begin(9600);
}
In the loop() function, read the four sensors using analogRead() and send their values to the serial buffer, separating them by blank spaces and finishing with a println() function call to get a newline character at the end of each cycle.
void loop() {
finger1 = analogRead(A5);
finger2 = analogRead(A4);
finger3 = analogRead(A3);
finger4 = analogRead(A2);
Serial.print(finger1);
Serial.print(" ");
Serial.print(finger2);
Serial.print(" ");
Serial.print(finger3);
Serial.print(" ");
Serial.println(finger4); }
Upload this code and open the Serial Monitor. If you put the glove on and bend your fingers, you should see the numbers changing within a range of 100 to 700 or similar.
Processing Glove Data Receiver
It will be much easier to understand this data if you can get a visual representation of it, so you’re going to build a simple Processing sketch that will receive the values and draw them on screen as variable-width rectangles. You will reuse some of these elements in the final project.
Open a new Processing sketch and import the Serial library. Then initialize the serial port and an array of four floats to contain the data from the flex sensor for each finger. The data from Arduino is actually integers, but you declare the fingerData array as a float data type because you are smoothing out the incoming data and the result may turn out to be floating point numbers. Then, within the setup() function, set the sketch size and initialize the serial communication.
import processing.serial.*;
Serial myPort;
float[] fingerData = new float[4];
void setup() {
size(800, 300);
// Serial Communication
String portName = Serial.list()[0];
myPort = new Serial(this, portName, 9600);
myPort.bufferUntil('
'), }
The draw() function begins by defining a black color for the background and then drawing a rectangle for each finger, the width based on the finger data from Arduino.
void draw() {
background(0);
for (int i = 0; i < fingerData.length; i++) {
rect(0, 50+i*50, fingerData[i], 20);
}
}
The serialEvent() function is triggered every time you get a newline character in the serial buffer. You have programmed the Arduino to send the four values from the flex sensors separated by blank spaces, so you need to get the string and split it at every blank space.
public void serialEvent(Serial myPort) {
String inString = myPort.readStringUntil('
'),
if (inString != null) {
String[] fingerStrings = inString.split(" ");
If you have the correct amount of String values (four), you convert every String to a float and then you add it to the fingerData[] array. You place the conversion inside a try{} … catch{} block to avoid the program stopping completely when it gets an anomalous character from Arduino, which happens occasionally. Use the Processing function lerp() to smooth out quick oscillations in the incoming data. Figure 11-12 shows what the data could look like.
if (fingerStrings.length==4) {
for (int i = 0; i < fingerData.length; i++) {
try {
float intVal = Float.valueOf(fingerStrings[i]);
fingerData[i] = lerp(fingerData[i], intVal, 0.5);
}
catch (Exception e) {
}
}
}
}
}
Figure 11-12. Finger-bending test
Going Wireless
Now that you have the basics of the glove interface working, you are going to improve your device by eliminating the cables. The ultimate goal of this project is an intuitive interface that works via your hand movements, so having a cable attached to it is not quite ideal.
To free yourself from the inconvenient cables, you need to add a power source to your glove design and an XBee radio module to transmit serial data wirelessly. You have used the XBee radio module in several other projects, so you should be acquainted with it already. If you skipped the previous chapters, read the introduction to XBee in Chapter 7.
LilyPad XBee
Happily, there’s a LilyPad XBee breakout board that works on the same principles as the main LilyPad board. You are going to add this XBee breakout to your glove, and you will use an XBee Explorer USB connected to your computer to receive the information (Figure 11-13).
Figure 11-13. XBee Explorer USB
Connecting the XBee breakout to your LilyPad board isn’t difficult. The way you normally do it is by attaching the + and - pins on the XBee breakout to 5V and ground on the LilyPad and the rx and tx to pins 1 and 0. But that’s on a standard LilyPad board. You’re using a LilyPad Simple, and after close inspection, you see that pins 0 and 1 don’t exist. (Well, they do exist but the architecture of the LilyPad doesn’t expose them.) So what should you do? Can you use the XBee module with the LilyPad Simple? The answer to this problem is one of the Arduino’s core libraries: SoftwareSerial.
SoftwareSerial Library
Arduino’s built-in support for serial communication in pins 0 and 1 occurs via a piece of hardware called UART (Universal Asynchronous Receiver/Transmitter), which is built into the ATmega microcontroller.
You can’t access pins 0 and 1 on the LilyPad Simple, but you can replicate the serial reception and transmission functionality on other pins using the SoftwareSerial library, which is included with the Arduino IDE. This way of transmitting information is actually much slower than using pins 0 and 1, but it’s still good enough to transmit the information from the glove in this project. If you are planning on adding some more demanding transmission of information to this project, you should definitely consider using the standard Arduino LilyPad board.
Wireless Arduino Code
Let’s modify the previous Arduino code to work with the XBee through pins 5 and 6 using this library. First, you need to include the SoftwareSerial library and initialize a SoftwareSerial object, passing pin numbers 5 and 6 as parameters. Call the object XBee. This is the first time you’re using an Arduino library in this book. You have previously included libraries in Processing by selecting Import Library in the Sketch menu or typing something like import SimpleOpenNI.*; directly in the text editor. In Arduino, the syntax is different because the language is based on C instead of Java. You can still go to Sketch > Import Library and choose SoftwareSerial; you’ll see a new line of code in your text editor, #include <SoftwareSerial.h>, which is the Arduino equivalent of the Processing import statement.
So wherever you wrote Serial on your previous code, you now write XBee. The resulting code is the following:
#include <SoftwareSerial.h>
SoftwareSerial XBee(5,6);
int finger1, finger2, finger3, finger4;
void setup() {
XBee.begin(9600);
}
void loop() {
finger1 = analogRead(A5);
finger2 = analogRead(A4);
finger3 = analogRead(A3);
finger4 = analogRead(A2);
XBee.print(finger1);
XBee.print(" ");
XBee.print(finger2);
XBee.print(" ");
XBee.print(finger3);
XBee.print(" ");
XBee.println(finger4);
}
This is the Arduino code that you will be using for the rest of this chapter. Before trying this code with the previous Glove Receiver Processing sketch, you need to sew the XBee to the glove circuit. Figure 11-14 shows the final circuit for the project with the components arranged as in the physical device. It will take you some time to sew all the connections. Be sure to make the connections solid and reliable; otherwise, the movement of your hands will end up loosening them.
Apart from the XBee, you need to introduce the LiPo battery in order to make the glove completely autonomous. Just plug it in the LilyPad power supply socket and put the battery inside the glove (Figure 11-15). If your gloves have a double skin, this is the perfect place to hide it.
Figure 11-14. Glove circuit completed
Figure 11-15. The LilyPad Xbee on the circuit (left) and the LiPo battery (right)
Implementing the Modeling Interface
Now that the physical device is ready and the code is uploaded to the Arduino LilyPad and sending serial data, it’s time to write your 3D modeling interface program in Processing.
Let’s define 3D modeling as the creation of 3D geometries with specialized software. These geometries are defined by surfaces and lines, which in turn are defined by points. You begin by defining points in space and then linking these points together with lines and surfaces, as you can see in Figure 11-16. You implement this by writing a very simple computer-aided design (CAD) software program (detailed in the “Geometric Classes” section) that you can control with your glove. This software is based on a main Processing routine and three helper classes that deal with particular functions and data structures.
The first class, GloveInterface, is derived from your previous Glove Data Receiver example and deals with the information coming from the glove via serial. The other three classes are the Point, Line, and Shape classes, which are the basic blocks of your geometrical constructions.
Figure 11-16. 3D modelling with the glove interface
The GloveInterface Class
This class deals with the finger-bending information from the flex sensors. Due to the different sizes and forms of your fingers and the particularities of the different flex sensors, the ranges of the data coming from each one of the fingers will be slightly different. You want to control the incoming data with the same functions, so you need to map these values to standard ranges with a calibration function.
First, declare the GloveInterface class in its own tab, as you saw in Chapter 7. Then declare a File object to store the calibration information so you don’t need to repeat the calibration process every time you run the program.
public class GloveInterface {
File fingerCal;
You need three PVectors to handle the position of your hand in space, the position that you consider to be the origin of coordinates (which is determined by the position of the hand at the moment of recognition), and the position of the glove in your model space (pos).
PVector pos = new PVector();
PVector zeroPos = new PVector();
PVector handPos = new PVector();
You also need an array to store the bending state of each finger as per the data from the Arduino (fingerState[]) and then after calibration (fingerStateCal[]). The calibration data is stored in the fingerCalibration[] nested array, which you initialize to predetermined values.
The integer currentFinger defines the finger that is “clicked” at any specific moment, which is determined using the setFingerValues() function. The default value (99) doesn’t correspond to any of the four fingers you are following and it’s identified as no finger clicked.
float[] fingerState = new float[4];
float[] fingerStateCal = new float[4];
float[][] fingerCalibration = {{0,600},{0,600},{0,600},{0,600}};
public int currentFinger = 99;
You only implement one constructor taking no arguments. Within this constructor, specify the file containing your calibration data. If the file already exists, get the data in the file using the function getCalibration().
GloveInterface() {
fingerCal = new File("data/fingerCal.txt");
if (fingerCal.exists()) {
getCalibration();
}
}
Setter Functions
You implement three setter functions: one to set the fingers’ state, another to set the current glove position, and a third one to set the initial or zero position.
The function setFingerValues() is called from the main routine every time you get new data from Arduino via the serial interface. This function takes an array of strings as its first argument. The first step in the function is to convert each String in the array into a float and store them in the fingerState[] array. This code is very similar to the code from the Glove Data Receiver sketch, but it uses a try{} … catch{} block to avoid the execution, stopping when incorrectly formatted strings are received.
public void setFingerValues(String[] fingerStrings) {
for (int i = 0; i < fingerState.length; i++) {
try {
float intVal = Float.valueOf(fingerStrings[i]);
fingerState[i] = PApplet.lerp(fingerState[i], intVal, 0.2f);
}
catch (Exception e) {
}
}
After acquiring the fingers’ states, use the fingerCalibration values to remap the incoming data from the calibration values to the desired range of 0-600. The way you calibrate the interface is by opening all your fingers and storing the “stretched” values into the zero index element of each array in fingerCalibration. Then you close your fingers and store the “maximum bending” state into the index 1 of each array in fingerCalibration. By mapping each of the incoming values from these ranges to the desired range (0-600), you make sure that a stretched finger means a value close to 0 and a fully bent finger casts a value close to 600. You analyze these calibration functions at the end of the class.
for (int i = 0; i < fingerStateCal.length; i++) {
fingerStateCal[i] = PApplet.map(fingerState[i], fingerCalibration[i][0],
fingerCalibration[i][1], 0, 600);
}
Finally, you determine if one of the fingers is “clicked” by checking which one of the fingers presents a higher bending state and if its value is over a threshold (300). The current finger is stored in the integer currentFinger.
float maxBending = 0;
currentFinger = 5;
for (int i = 0; i < fingerStateCal.length; i++) {
if (fingerStateCal[i] > maxBending && fingerStateCal[i] > 300) {
maxBending = fingerStateCal[i];
currentFinger = i;
}
}
}
The function setPosition() takes a PVector defining the current position of your hand in global coordinates, which is stored in the handPos PVector. Then you translate this value to your modeling coordinate system by subtracting the zeroPos PVector from it. The zeroPos PVector updates from the main routine by calling the setZeroPos() function.
public void setPosition(PVector inputPos) {
this.handPos = inputPos;
this.pos = PVector.sub(inputPos, zeroPos);
}
public void setZeroPos(PVector handPos) {
this.zeroPos = handPos;
}
Display Functions
The draw() function simply draws a point on screen of the coordinates of your glove interface. The switch() statement changes the color of the point depending on the current finger.
public void draw() {
pushStyle();
switch (currentFinger) {
case 0:
stroke(255, 0, 0);
break;
case 1:
stroke(0, 255, 0);
break; case 2:
stroke(0, 0, 255);
break;
case 3:
stroke(255, 255, 0);
break;
case 5:
stroke(255, 255, 255);
break;
}
strokeWeight(20);
if(!record){
point(pos.x, pos.y, pos.z);
}
popStyle();
}
Note that you put the point() function within an if() statement that only draws if the Boolean record is not true. This is due to the dxf export library that you will be bringing in later; this Processing library can’t export points so you need to avoid the printing of points when exporting. You do this every time you draw points in this program.
The drawData() function draws the same information on screen as the Glove Data Receiver sketch, letting you see the calibrated bending state of each finger.
public void drawData() {
for (int i = 0; i < fingerStateCal.length; i++) {
rect(0, 50 + i * 50, (int) fingerStateCal[i], 20);
}
}
Calibrating the Interface
We have already explained how the calibration of the device works. The following three functions take care of setting the calibration data, storing it in the external file, and retrieving it on demand.
The function calibrateFinger() is called from the main Processing routine by pressing a key on the keyboard. It takes the finger to be calibrated (i) and the state being calibrated (j), which is 0 if calibrating the stretched state or 1 if calibrating the bent state. You first store the present non-calibrated state of the finger into the fingerCalibration[] array and then call the saveCalibration() function.
public void calibrateFinger(int i, int j) {
fingerCalibration[i][j] = fingerState[i];
saveCalibration();
}
saveCalibration() converts the fingerCalibration double array into a linear array of strings and uses the saveStrings() function to store all the values into the fingerCal file.
private void saveCalibration() {
String[] mySettings = new String[8];
for (int k = 0; k < fingerCalibration.length; k++) {
mySettings[k] = String.valueOf(fingerCalibration[k][0]);
mySettings[k + 4] = String.valueOf(fingerCalibration[k][1]);
}
PApplet.saveStrings(fingerCal, mySettings);
}
Lastly, the function getCalibration() performs the opposite process, using loadStrings() to retrieve the strings from the fingerCal file and storing these values into the fingerCalibration array after converting them into floats.
private void getCalibration() {
String[] mySettings = PApplet.loadStrings(fingerCal);
for (int k = 0; k < fingerCalibration.length; k++) {
fingerCalibration[k][0] = Float.valueOf(mySettings[k]);
fingerCalibration[k][1] = Float.valueOf(mySettings[k + 4]);
}
}
}
Geometric Classes
In order for you to be able to draw in three dimensions with your new glove interface, you need to implement your own modeling software or link your Processing interface to existing CAD software. You’re going to take the first approach by implementing a very, very simple and limited CAD program.
CAD, or computer-aided design, is the software that allows engineers, architects, and designers to draft with a computer. There are plenty of commercial CAD and 3-D modeling software packages—and a few good open source options. The very basic blocks of the geometrical constructions generated with modeling software are the notions of point, line, and surface. You are going to create a separate class for each of these basic geometric concepts, which you will then use to generate the geometry (Figure 11-17).
Figure 11-17. Points, lines, and surfaces defined by your program
Point Class
As you build your geometry, a point is defined as a specific location in 3D space. This location is described by three coordinates: x, y, and z. For your purposes, this location can be treated as a vector (x,y,z), which means you could use the PVector class to define your points. Instead, you are going to write your own Point class to be able to add a couple more functions to your point objects.
Note that the class Point will have two fields: a PVector describing its position and a Boolean variable that is true if the element has been selected. You implement only one constructor taking a PVector as parameter that is used as the point’s position.
The draw() function displays the point on screen. Note that you again put the point() function within an if() statement that only draws if the Boolean record is not true.
void draw() {
pushStyle();
strokeWeight(10);
if (selected) {
stroke(255, 255, 0);
}
else {
stroke(200);
}
if (!record){
point(pos.x, pos.y, pos.z);
}
popStyle();
}
Finally, implement three more functions that allow you to change the position of the point, select it, and unselect it (Figure 11-18).
public void setPos(PVector newPos) {
this.pos = newPos;
}
public void select() {
this.selected = true;
}
public void unSelect() {
this.selected = false;
}
}
Figure 11-18. Selected point
Line Class
As you saw in the previous section, a line is a one-dimensional geometrical element. A line is by definition infinite, but in this case, whenever we talk about lines, we’re actually referring to line segments.
A line segment is a portion of a line, and it can be defined by two points. You have probably heard line defined as “the shortest path between two points,” which is pretty close to the sense the word line will have in your code.
Your Line class, then, has two Point objects as fields. These two points define the line and are used within the draw() function to display the line on screen by using Processing’s line() function.
public class Line {
Point p1;
Point p2;
Line(Point p1, Point p2) {
this.p1 = p1;
this.p2 = p2;
}
void draw() {
pushStyle();
stroke(255);
line(p1.pos.x, p1.pos.y, p1.pos.z, p2.pos.x, p2.pos.y, p2.pos.z);
popStyle();
}
}
Shape Class
The Shape class represents triangular surfaces in your program. A surface is a two-dimensional geometrical element. Your Shape objects represent portions of planes, or flat surfaces, defined by three points (Figure 11-19). Triangular shapes are the basis of many polygonal modeling programs; for you, they constitute the only way in which your software will represent surfaces.
The Shape class uses an ArrayList of Point elements to store the points defining it. You could have used three points instead, but this way lets you extend the concept further and draw complex shapes consisting of more than three points. The draw() function makes use of Processing’s beginShape(), vertex(), and endShape() functions to draw your shapes on screen. In the future, you might want to dynamically add points to this function in runtime, so add a function addPoint() for this purpose.
public class Shape {
ArrayList<Point> points = new ArrayList<Point>();
Shape( ArrayList<Point> points) {
for (Point pt : points) {
this.points.add(pt);
}
}
void draw() {
pushStyle();
fill(255);
stroke(150);
beginShape();
for (int i = 0; i < points.size(); i++) {
vertex(points.get(i).pos.x, points.get(i).pos.y, points.get(i).pos.z);
}
endShape(PConstants.CLOSE);
popStyle();
}
void addPoint(Point newPoint) {
points.add(newPoint);
}
}
Figure 11-19. Freeform three-dimensional geometry built with triangular shapes
Modeling Interface Main Program
Now that you have your helper classes implemented, you can write the main routine. This code drives the program’s workflow and makes use of the previous classes to achieve the ultimate goal of allowing you to make 3D geometries using your glove interface. Add more of Processing’s core libraries so you can export the geometries you create to .dxf files and import them into other CAD software.
Imports and Fields
You need to import five libraries for this project: three core libraries (OpenGL, Serial, and DXF) and two external libraries (Simple-OpenNI and KinectOrbit).
import processing.opengl.*;
import processing.serial.*;
import processing.dxf.*;
import SimpleOpenNI.*;
import kinectOrbit.*;
KinectOrbit myOrbit;
SimpleOpenNI kinect;
Serial myPort; // Initialize the Serial Object
boolean serial = true; // Define if you're using serial communication
You are implementing hand recognition using NITE’s XnVSessionManager, as you did in Chapter 5, so you must initialize the appropriate objects.
XnVSessionManager sessionManager;
XnVPointControl pointControl;
You then declare the GloveInterface object glove and four ArrayList collections defining the geometry’s points, selected points, lines, and shapes. Use ArrayList collections because you don’t know how many points, shapes, or lines the user will create; ArrayList collections provide you with the flexibility you need for the program. The integer thresholdDist defines how close the glove interface needs to be from a point for it to be selected.
private GloveInterface glove;
ArrayList<Point> points = new ArrayList<Point>();
ArrayList<Point> selPoints = new ArrayList<Point>();
ArrayList<Line> lines = new ArrayList<Line>();
ArrayList<Shape> shapes = new ArrayList<Shape>();
int thresholdDist = 50;
The boolean record serves to indicate when you want to save your geometry as a .dxf file. It is set to false by default, so you avoid creating a new file in the first loop.
boolean record = false;
Setup Function
The setup() function starts by defining the sketch size and renderer, and it then initializes the KinectOrbit object. You then set KinectOrbit to draw the orbiting gizmo on screen and the Coordinate System axes. You also use the function setCSScale() to set the size of the CS axes to 200 mm. (You can change these settings to any other configuration.)
public void setup() {
size(800, 600, OPENGL);
myOrbit = new KinectOrbit(this, 0, "kinect");
myOrbit.drawGizmo(true);
myOrbit.drawCS(true);
myOrbit.setCSScale(200);
Now you initialize the Simple-OpenNI object and all the capabilities you need, including the sessionManager and pointControl objects.
// Simple-openni
kinect = new SimpleOpenNI(this);
kinect.setMirror(false);
kinect.enableDepth();
kinect.enableGesture();
kinect.enableHands();
sessionManager = kinect.createSessionManager("Wave", "RaiseHand"); // setup NITE
pointControl = new XnVPointControl(); // Setup NITE's Hand Point Control
pointControl.RegisterPointCreate(this);
pointControl.RegisterPointDestroy(this);
pointControl.RegisterPointUpdate(this);
sessionManager.AddListener(pointControl);
And finally, you initialize the serial communication and the GloveInterface object.
if (serial) {
String portName = Serial.list()[0]; // Get the first port
myPort = new Serial(this, portName, 9600);
myPort.bufferUntil('
'),
}
glove = new GloveInterface(); // Initialize the GloveInterface Object
}
Draw Function
As usual, you pack most of the processes into additional functions, so the draw() function presents a clear scheme of the program’s workflow. The first step is to update the Simple-OpenNI object (kinect) and the sessionManager. Then you set the background to black.
public void draw() {
kinect.update(); // Update Kinect data
kinect.update(sessionManager); // update NITE
background(0);
This program behaves differently depending on which finger is bent. You do this by means of a switch() statement with four case handlers. The four possible behaviors are moving an existing point, creating a new point, adding a line between two points, or adding a new shape (Figure 11-20).
switch (glove.currentFinger) {
case 0:
movePoint();
break;
case 1:
addPoint();
break;
case 2:
addLine();
break;
case 3:
addShape();
break;
}
Figure 11-20. Triggering Actions with your Fingers and thumb
After this switch() statement, all the geometrical operations have been performed, so you only need to draw the results on screen. You first push the KinectOrbit object so you can control your point of view of the geometry.
myOrbit.pushOrbit(this); // Start Orbiting
After pushing the KinectOrbit, check if the boolean record is true; if so, use the beginRaw() method to save the geometry into the file output.dxf. Then draw the glove and the points, lines, and shapes. The drawClosestPoint() function highlights the closest point to your glove interface if the distance is smaller than thresholdDist.
if (record == true) {
beginRaw(DXF, "output.dxf"); // Start recording to the file
}
glove.draw();
drawClosestPoint();
for (Point pt : points) {
pt.draw();
}
for (Line ln : lines) {
ln.draw();
}
for (Shape srf : shapes) {
srf.draw();
}
Before you pop the KinectOrbit, check again the state of the record variable. If it’s true (and you are recording the .dxf file), you end the recording and set the record variable to false. After this, you pop the KinectOrbit and, if you want to see the finger data printed on screen, call the method drawData() from the glove object.
if (record == true) {
endRaw();
record = false; // Stop recording to the file
}
myOrbit.popOrbit(this); // Stop Orbiting
glove.drawData();
}
Additional Functions
The movePoint() function is called when you bend your thumb. It checks all the existing points; if their distance to your glove object is under thresholdDist, it changes the point’s position to the position of your glove (which is your hand’s position). In this way, you can actually move the points around with your hand, just as if they were standing around you in space.
You use a Java for-each loop to run through the ArrayList points. This is equivalent to using a standard for loop, simplifying this common form of iteration. You can read it as “For each Point object in the ArrayList points, do { }.” The Point instance pt is set to a different Point object inside points in every loop.
void movePoint() {
for (Point pt : points) {
if (glove.pos.dist(pt.pos) < thresholdDist) {
pt.setPos(glove.pos);
}
}
}
The addPoint() function creates a new point at the position of your glove. Bending your index finger triggers this function. To avoid the creation of multiple adjacent points, you won’t create a point if there is another point closer than thresholdDist.
void addPoint() {
Point tempPt = new Point(glove.pos.get());
boolean tooClose = false;
for (Point pt : points) {
if (tempPt.pos.dist(pt.pos) < thresholdDist) {
tooClose = true;
}
}
if (!tooClose) {
points.add(tempPt);
}
}
Bending your middle finger calls the addLine() function. This function proceeds in two steps. First, it checks if there is a point closer than the defined threshold. If there is one and it’s not already contained in the ArrayList selPoints, it selects the point and adds it to selPoints.
Then, if there is more than one point selected, it creates a new line between the two points in selPoints and adds it to the ArrayList lines. Finally, it unselects everything using the unSelectAll() function.
void addLine() {
for (Point pt : points) {
if (glove.pos.dist(pt.pos) < thresholdDist) {
pt.select();
if (!selPoints.contains(pt)) {
selPoints.add(pt);
}
if (selPoints.size() > 1) {
Line lineTemp = new Line(selPoints.get(0), selPoints.get(1));
lines.add(lineTemp);
unSelectAll();
}
}
}
}
The last geometry creation function, addShape(), is called when you bend your third finger. This function works in the same way as addLine(), but it waits until you have three selected points to create a new Shape object, adds it to the ArrayList shapes, and unselects all the points.
private void addShape() {
for (Point pt : points) {
if (glove.pos.dist(pt.pos) < thresholdDist) {
pt.select();
if (!selPoints.contains(pt)) {
selPoints.add(pt);
}
if (selPoints.size() > 2) {
Shape surfTemp = new Shape(selPoints);
shapes.add(surfTemp);
unSelectAll();
}
}
}
}
The unSelectAll() function runs through all the points in your sketch and unselects them; then it clears the selPoints ArrayList.
void unSelectAll() {
for (Point pt : points) {
pt.unSelect();
}
selPoints.clear();
}
The previously used drawClosestPoint() function takes care of changing the onscreen appearance of the points that are close enough to the glove to be modified with the other functions.
void drawClosestPoint() {
for (Point pt : points) {
if (glove.pos.dist(pt.pos) < thresholdDist) {
pushStyle();
stroke(0, 150, 200);
strokeWeight(15);
point(pt.pos.x, pt.pos.y, pt.pos.z);
popStyle();
}
}
}
Processing Callbacks
You use two Processing callback functions in this project. First, the serialEvent() function calls every time you receive a newline character from serial. This function acquires the string from the serial communication, splits it, and passes the resulting array of strings as a parameter to the glove object using the method setFingerValues(), which updates the finger-bending values.
public void serialEvent(Serial myPort) {
String inString = myPort.readStringUntil('
'),
if (inString != null) {
String[] fingerStrings = inString.split(" ");
if (fingerStrings.length == 4) {
glove.setFingerValues(fingerStrings);
}
}
}
You use the keyPressed() Processing callback function to calibrate your glove interface. Numbers 1 to 4 set the stretched values for your fingers, and q, w, e, and r characters set the maximum bending values. Additionally, the character d triggers the saving of the geometry into the previously defined .dxf file.
public void keyPressed() {
switch (key) {
case '1':
glove.calibrateFinger(0, 0);
break;
case '2':
glove.calibrateFinger(1, 0);
break;
case '3':
glove.calibrateFinger(2, 0);
break;
case '4':
glove.calibrateFinger(3, 0);
break;
case 'q':
glove.calibrateFinger(0, 1);
break;
case 'w':
glove.calibrateFinger(1, 1);
break;
case 'e':
glove.calibrateFinger(2, 1);
break;
case 'r':
glove.calibrateFinger(3, 1);
break;
case 'd':
record = true;
break;
}
}
Simple-OpenNI Callbacks
Finally, you add the Simple-OpenNI callback functions. Within onPointCreate(), you set the zero position of your glove. This has the effect of setting the origin of coordinates to the point in space where the waving gesture was recognized. This makes it easier for the user to be drawing at an acceptable distance from the origin.
public void onPointCreate(XnVHandPointContext pContext) {
println("onPointCreate:");
PVector handVec = new PVector(pContext.getPtPosition().getX(), pContext
.getPtPosition().getY(), pContext.getPtPosition().getZ());
glove.setZeroPos(handVec.get());
}
public void onPointDestroy(int nID) {
println("PointDestroy: " + nID);
}
Within the onPointUpdate() function, you call the setPosition() method from your glove object to update the position of the glove image on screen to reflect your current hand position.
public void onPointUpdate(XnVHandPointContext pContext) {
PVector handVec = new PVector(pContext.getPtPosition().getX(), pContext
.getPtPosition().getY(), pContext.getPtPosition().getZ());
glove.setPosition(handVec);
}
Figure 11-21. 3D geometries created with the interface
Run this program and start playing with the position of your hands and the bending of your fingers (Figure 11-21). Soon you will develop an intuitive understanding of the functioning of the program; after some time, you will be able to create some pretty complex 3D geometries without having touched your mouse.
When you are satisfied with your results, press the d key on your keyboard and you’ll get a file called output.dxf in your Processing sketch folder. Note that .dxf is a file format created by Autodesk (the company behind AutoCAD) for importing and exporting data from AutoCAD to other programs.
You have chosen .dxf because there is a Processing core library that does all the exporting for you, but if you are curious enough to read the next chapter, you will learn how to implement your own routine to export .ply files, which are compatible with many other modeling programs.
Summary
This project led you through the building of a wearable, lightweight, wireless user interface designed to work in combination with Kinect’s hand-tracking capabilities that ultimately allowed you to create 3D geometries in an intuitive way, without using any conventional human interface devices.
You built the interface on a glove, using the Arduino LilyPad board in combination with flex sensors to read the bending of your fingers and a couple of XBee radio modules to transmit the information to your computer wirelessly. You also learned how to use the Arduino SoftwareSerial library to use any pin on the Arduino for serial communication.
Finally, you implemented your own simplified CAD software to deal with the basic geometrical entities and data structures that constituted your constructions. This software is controllable by your glove interface.
This software and interface are just the bare bones of what could be the beginning of a new standard in human-computer interfaces. Thanks to the Kinect, you are no longer limited to the two dimensions of conventional devices. Now you can use the full mobility of your body as a way to input data into your applications.