C H A P T E R 10
Biometric Station
In 1487, Leonardo da Vinci created a drawing that he entitled “The Vitruvian Man,” depicting what he believed to be the ideal human proportions. But these proportions are not only a matter of beauty. Height, weight, and other body measures can actually be an indicator of appropriate development and general health in both children and adults. Kinect has a device that can precisely measure all your body proportions in real time. If you combine this ability with other sensors, you can create a general-purpose biometric station for your personal use and delight.
Throughout this chapter, you will learn how to hack a bathroom scale and connect it wirelessly to your computer, how to combine the incoming data with the Kinect point cloud and skeleton data, and how to compare this information to external databases for different purposes. Table 10-1 shows the parts required for this project. You will be able to identify people, automatically get body mass indexes, and check whether they fall within desirable ranges, as shown in Figure 10-1. You will store that information so you can create graphs of the evolution.
This is an advanced chapter in terms of programming techniques, and it will introduce some Java file parsing techniques and the use of Java AWT components. Welcome to Kinect biometrics!
Figure 10-1. A Kinect user being scanned and recognized
Hacking a Bathroom Scale
In this project you will combine weight information with volume information from the Kinect sensor. You learned how to acquire Kinect data in a previous chapter, so now you need to figure out how to get the weight information into the computer. Well, one option is to type the information in, but that would be too easy, wouldn’t it? You’re going to go a little further and hack a standard bathroom scale to communicate wirelessly with your computer, so you can get the volume and weight information at the same time (Figure 10-2).
Figure 10-2. Scale after removing the cover and LCD
Hacking a bathroom scale would seem to be a simple task. If you have previously worked with pressure sensors, you might think that it’s just a matter of finding the analog output, reading it from your Arduino, and mapping the values to the real weight. Well, it happens to be slightly more complicated than that. Most modern scales use the combined output of one to four strain gauges. The main PCB processes this information and displays it on the LCD.
If, after a close inspection of your PCB, you don’t find any analog output that makes sense regarding weight data, look at the back of the PCB. There’s often a three-pin breakaway header (Figure 10-3) that provides an analog signal that changes depending on the state of the scale (off, weighing, weight found). You will see how to use this information later.
Figure 10-3. PCB breakaway headers connected to an analog sensor cable
Trying to get information from the strain gauges directly seems downright impossible, so why not let the PCB do its job and just sniff the weight information as it’s passed to the LCD? This lets the PCB do all computation and means that you get the same numbers as are shown on the LCD.
LCD hacking is a very handy technique indeed; the same approach can be applied to any other appliance or device with a built-in LCD (which is pretty much every single device you can buy today). You might as well be getting the minutes remaining before you can eat your microwaved meal or the temperature in your home boiler. But for the moment, let’s stick to the weight on your bathroom scale.
Seven-Segment LCD
Most bathroom scales, and actually most built-in LCDs, are seven-segment displays (Figure 10-4). This includes calculators and all kind of home appliances. They are widely used because this is a very optimized way of displaying decimals compared to dot-matrix displays, as every number can be encoded in one byte of information; seven bits defining the number, and one extra for the decimal point.
Figure 10-4. Individual segments (from Wikipedia), and the LCD on your scale.
The information is usually encoded in the order of gfedcba or abcdefg, but you can find other encodings, as you will see in this chapter. You have to check the signals from the PCB to the LCD to figure out the encoding of your scale, so first let’s hack the scale.
Hacking the LCD
Once you have removed the cover and exposed the guts of your scale, you’re probably looking at an LCD display attached to a printed circuit board (PCB). Unscrew the LCD from the PCB. There will be some form of connection between the LCD and the 16 pins on the PCB (Figure 10-5). In our case, it was connective rubber that we kept for future repositioning.
The ultimate goal here is to acquire and decode the information sent from the PCB to the LCD, so you solder one wire to each of the LCD pins. (Later you will try to figure out which ones define the eight bits of information relevant to you.) Use flexible wrap wire or a 16-wire ribbon cable, preferably; hook-up wire is too sturdy and could strip the PCB connectors off.
If there are connections on the PCB that you can use without impeding the repositioning of the LCD, use those so you can still read the numbers on your scale as depicted in Figure 10-6. This will be extremely helpful when analyzing the information from the pins.
Figure 10-5. Main PCB with LCD pin connections (at the top)
Test that all pins are properly connected and there is no electrical flow between them. Then replace the LCD and the scale cover, leaving all the LCD wires and the analog cable you previously connected outside. You will probably need to carve an indent on the plastic cover with a scalpel so you can pass the wires through. Connect all the cables to an Arduino prototyping shield so they are ready to be easily plugged into your board (Figures 10-6 through 10-8).
Figure 10-6. Wires soldered to LCD pins. The rubber connection can still be plugged in.
Figure 10-7. Arduino and XBee module plugged and LCD repositioned
Figure 10-8. Arduino Prototype Board connected to all the LCD pins and the PCB
Add an XBee module to the board, as explained in Chapter 7, and put everything into a transparent plastic box along with an external power supply (Figure 10-9). This way, you won’t need to depend on the USB cable’s length to use your scale.
Figure 10-9. Scale finished and plugged to the Arduino and XBee module
Acquiring the LCD Signal
You have a pretty exciting cryptographic exercise ahead. When you connect your shield to the Arduino and read the values coming from the LCD pins, you will be overwhelmed by a constant flow of ones and zeros passing through your Serial Monitor. Don’t despair; you are going to make sense of all this in the next few pages!
Connect your Arduino to your computer and start a new Arduino sketch. If you found a breakout header like the one shown previously, you might be able to power your scale from the Arduino; connect the pins to ground and 3.3V so you can get by without using a battery.
By now you should be acquainted with Arduino code, so you should have no problem reading through the following code.
#define dataSize 16
int data [dataSize];
void setup() {
Serial.begin(9600);
pinMode(2,INPUT);
pinMode(3,INPUT);
pinMode(4,INPUT);
pinMode(5,INPUT);
pinMode(6,INPUT);
pinMode(7,INPUT);
pinMode(8,INPUT);
pinMode(9,INPUT);
pinMode(10,INPUT);
pinMode(11,INPUT);
pinMode(12,INPUT);
}
void loop() {
for(int i = 2; i<13; i++){
data[i-2] = digitalRead(i);
}
data[11] = digitalRead(A5);
data[12] = digitalRead(A4);
data[13] = digitalRead(A3);
data[14] = digitalRead(A2);
data[15] = digitalRead(A1);
for(int i = 0; i<dataSize; i++){
Serial.print(data[i]);
}
Serial.println();
}
You are defining your pins as inputs (only the digital ones) and then storing their current digital readings in the array of integers you have called data. Note that the order is defined by the connections you used. In your Arduino, you connected the LCD pins left to right from Arduino pin 2 to 12 and then from analog 5 to 1, hence the order of the readings.
You then print the resulting array to your serial port. If you open the Serial Monitor, you will be confronted with something similar to the following lines:
0000000011100000
1000000000000000
0000000001010000
0010000000000000
0000000011110000
0000000011100000
0100000000000000
This won’t really make any sense to you yet, but you can start to identify a pattern. There are a series of sparse ones written on the left side of the string, and then some more tightly grouped at the center-right. Not much, but it’s a start.
Reading the LCD Signal
Every line of code seems to be different, but an LCD should keep the pins high for a while, turn them off, and them on again for a certain time. Arduino should be faster than that, so you should get at least a couple of consecutive lines that look the same every time. You might have guessed the answer already: Arduino can indeed loop faster than the LCD cycle, but the serial printing slows it down considerably, so you don’t get the same reading twice. Let’s test this with some more sophisticated code.
You defining a bi-dimensional array named serial that will store fifty lines of LCD signals. Whenever it acquires the fifty lines, it prints them to the Serial Monitor. This way, you’re not slowing down your Arduino every cycle, so you should be able to gather the information faster. When you print the information out, you will, of course, spend some time and certainly skip some cycles, but you will still have a consistent stream of data in your serial array.
#define dataSize 16
int data [dataSize];
#define lines 50
int serial [lines][dataSize];
int index;
void setup() {
Serial.begin(9600);
pinMode(2,INPUT);
pinMode(3,INPUT);
pinMode(4,INPUT);
pinMode(5,INPUT);
pinMode(6,INPUT);
pinMode(7,INPUT);
pinMode(8,INPUT);
pinMode(9,INPUT);
pinMode(10,INPUT);
pinMode(11,INPUT);
pinMode(12,INPUT);
}
void loop() {
for(int i = 2; i<13; i++){
data[i-2] = digitalRead(i);
}
data[11] = digitalRead(A5);
data[12] = digitalRead(A4);
data[13] = digitalRead(A3);
data[14] = digitalRead(A2);
data[15] = digitalRead(A1);
for(int i = 0; i<dataSize; i++){
serial[index][i] = data[i];
}
index++;
if(index==lines){
index=0;
for(int i = 0; i<lines; i++){
for(int j = 0; j<dataSize; j++){
Serial.print(serial[i][j]);
}
Serial.println();
}
Serial.println("__________________");
}
}
If you upload this sketch and open the Serial Monitor, you will get a surprisingly different panorama:
0000000001010000
0000000001010000
0000000001010000
0000000011110000
0000000011110000
__________________
1000000000000000
1000000000000000
0000000000000000
0010000000000000
0010000000000000
You can see that the LCD pins are sending a pattern for a time and then changing to another pattern. You can’t synchronize your Arduino to the LCD, but you can write code that reads the numbers only if they have changed and if they are maintained for at least two cycles (to avoid misreading at the transition point).
#define dataSize 16
int data [dataSize];
int previousData [dataSize];
#define lines 50
int serial [lines][dataSize];
int index;
// Booleans defining the change
boolean changed;
boolean scanning;
void setup() {
Serial.begin(9600);
pinMode(2,INPUT);
pinMode(3,INPUT);
pinMode(4,INPUT);
pinMode(5,INPUT);
pinMode(6,INPUT);
pinMode(7,INPUT);
pinMode(8,INPUT);
pinMode(9,INPUT);
pinMode(10,INPUT);
pinMode(11,INPUT);
pinMode(12,INPUT);
}
void loop() {
for(int i = 2; i<13; i++){
data[i-2] = digitalRead(i);
}
data[11] = digitalRead(A5);
data[12] = digitalRead(A4);
data[13] = digitalRead(A3);
data[14] = digitalRead(A2);
data[15] = digitalRead(A1);
Check if the numbers have changed from the previous reading
int same = 0;
for(int i = 0; i<dataSize; i++){
if(data[i]==previousData[i])same++;
previousData[i] = data[i];
}
if(same<dataSize){
changed=true;
scanning=true;
}
else{
changed=false;
}
If the numbers have NOT changed and if they have occurred at least twice (so scanning is true), store the the data into the serial[] array.
if(changed == false && scanning == true){
for(int i = 0; i<dataSize; i++){
serial[index][i] = data[i];
}
index++;
scanning=false;
}
Finally, if you have reached the buffer limit, reset the index to 0 and run a for loop to print each line in your serial[] array to serial.
if(index==lines){
index=0;
for(int i = 0; i<lines; i++){
for(int j = 0; j<dataSize; j++){
Serial.print(serial[i][j]);
}
Serial.println();
}
Serial.println("__________________");
}
}
This will actually give you the key to reading the scale’s LCD signals. If you open your Serial Monitor, you now see a clear pattern in the flow of data streaming from the scale.
1000000000000000
0010000000000000
0001000000000000
0000000011100000
0000000011110010
0000000001010000
0000000011110000
0100000000000000
This is your data stream when the scale is showing a 0.0 kg signal. If you settle on another weight, the signal will look differently. Let’s try 67.4kg.
1000000000000000
0010000000000000
0001000000000000
0000001001000000
0000001101010010
0000001001110000
0000001111100000
0100000000000000
Well, that’s good; you can now see that there is a pattern that repeats itself at the far-left of your strings, and then some numbers that change at the center. The ones appearing on the right side are actually defining if the kg or lb symbol is turned on.
After quite a lot of testing, we were able to identify the pins that carried the information on the digits and the ones that dealt with other elements. You should run a similar trial and error process to find out the order in your particular scale. This was the one in ours.
Pin 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
1 0 0 0 | 0 0 | 0 0 | 0 0 | 0 0 | 0 0 0 0
0 0 1 0 | 0 0 | 0 0 | 0 0 | 0 0 | 0 0 0 0
0 0 0 1 | 0 0 | 0 0 | 0 0 | 0 0 | 0 0 0 0
The first three rows don’t change with the weight, so they can be used to identify the start of the message when pin 3 is turned on.
0 0 0 0 | 0 0 | 1 0 | 0 1 | 0 0 | 0 0 0 0
0 0 0 0 | 0 0 | 1 1 | 0 1 | 0 1 | 0 0 1 0
0 0 0 0 | 0 0 | 1 0 | 0 1 | 1 1 | 0 0 0 0
0 0 0 0 | 0 0 | 1 1 | 1 1 | 1 0 | 0 0 0 0
After the three first rows have passed, the fifth to eighth rows carry the important information. Columns 4-5 each consist of 8 bits, defining the hundreds. Columns 6-7 define the tens, 8-9 the units, and 10-11 the decimals. Column 14 provided information on the unit used (kg or lbs).
0 1 0 0 | 0 0 | 0 0 | 0 0 | 0 0 | 0 0 0 0
The last row is always constant and can be used to identify the end of the message.
Sending the Signal to Processing
You have acquired the information sent to the LCD display. You now want to decode it so you can work with the weight. You are going to send the raw information to Processing and write a decoder to transform this information into numbers.
Stop scanning columns 12, 13, and 15 because they don’t add any useful information. Write a routine that checks the start of the message (one on column 3) and the end of the message (one on column 2) and sends to serial all the rows in between. Also send the information coming from analog pin 0 connected to the PCB’s analog output that changes with the state of the weighing process.
#define dataSize 13
int data [dataSize];
int previousData [dataSize];
#define lines 4
int serial [lines][dataSize];
int index;
// Booleans defining the change
boolean changed;
boolean scanning;
boolean start;
int state;
int previousState;
void setup() {
Serial.begin(9600);
pinMode(2,INPUT);
pinMode(3,INPUT);
pinMode(4,INPUT);
pinMode(5,INPUT);
pinMode(6,INPUT);
pinMode(7,INPUT);
pinMode(8,INPUT);
pinMode(9,INPUT);
pinMode(10,INPUT);
pinMode(11,INPUT);
pinMode(12,INPUT);
}
void loop() {
for(int i = 2; i<13; i++){
data[i-2] = digitalRead(i);
}
data[11] = digitalRead(A5);
data[12] = digitalRead(A2);
state = analogRead(A0);
// Check if the numbers have changed from the previous reading
int same = 0;
for(int i = 0; i<dataSize; i++){
if(data[i]==previousData[i])same++;
previousData[i] = data[i];
}
if(same<dataSize){
changed=true;
scanning=true;
}
else{
changed=false;
}
if(changed == false && scanning == true){
if(data[1] == 1){
start=false;
Serial.println('S'),
for(int i = 0; i<4; i++){
for(int j = 0; j<dataSize; j++){
Serial.print(serial[i][j]);
}
Serial.println();
}
Serial.print('X'),
Serial.println(state);
}
if(start){ // If you have started the message
for(int i = 0; i<dataSize; i++){
char number[1];
// Serial.print(itoa(data[i], number, 10));
// Serial.print(" ");
serial[index][i]=data[i];
}
index++;
}
scanning=false;
if(data[3] == 1){
start=true;
index = 0;
}
}
}
This code will output the following data stream to the serial buffer:
S
0000000001100
0000000001111
0000000011110
0000000010110
X532
S
0000000001100
0000000001111
0000000011110
0000000010110
X532
You have enough data to identify the start and end of the information package, and all the necessary data to recompose the weight information at the other end of the line.
Decoding the LCD Signal
The decoding process is far simpler than the acquisition of the information. At this point, you can easily write a Processing program that receives the information from Arduino and decodes it into numbers you can understand. You are going to pack this process into a class so you can reuse the implementation in different applications.
This class is called LCD. You input a two-dimensional array defining the individual number on the seven-segment display and you get a float defining the weight.
public class LCD {
private float data;
int lcdArray[][] = new int[4][13];
private String units = "kg";
private int state;
boolean foundWeight;
boolean scanning;
int[] zero = { 1, 1, 1, 0, 1, 1, 1 };
int[] one = { 0, 0, 1, 0, 1, 0, 0 };
int[] two = { 1, 1, 0, 1, 1, 0, 1 };
int[] three = { 1, 0, 1, 1, 1, 0, 1 };
int[] four = { 0, 0, 1, 1, 1, 1, 0 };
int[] five = { 1, 0, 1, 1, 0, 1, 1 };
int[] six = { 1, 1, 1, 1, 0, 1, 1 };
int[] seven = { 0, 0, 1, 0, 1, 1, 1 };
int[] eight = { 1, 1, 1, 1, 1, 1, 1 };
int[] nine = { 1, 0, 1, 1, 1, 1, 1 };
Figure 10-10. Seven-segment display code
The arrays you just defined are a blueprint to check your data against. They are based on the distribution of the seven-segment bits coming from Arduino. After, again some intensive testing sessions, we found the pattern depicted in Figure 10-10.
The method getReading() returns the real value of the weight.
public float getReading() {
return data;
}
Use the method setLcdArray() to input the new values from Arduino into your LCD class to keep the numbers updated. You convert every character to an integer, store them in the lcdArray, and run the update() method to update your real value.
public void setLcdArray(char[][] stringData) {
if (stringData[1][12] == '1') {
units = "kg";
}
else if (stringData[2][12] == '1') {
units = "lb";
}
for (int i = 0; i < stringData.length; i++) {
for (int j = 0; j < stringData[0].length; j++) {
lcdArray[i][j] = Character.digit(stringData[i][j], 10);
}
}
this.update();
}
Within the update function, you reorganize your main two-dimensional array into a three-dimensional array that stores the four two-dimensional arrays defining each of the digits.
public void update() {
int[] digits = new int[4];
int[][][] segments = new int[4][4][2];
for (int i = 0; i < lcdArray.length; i++) {
for (int j = 4; j < 6; j++) {
segments[0][i][j - 4] = lcdArray[i][j];
}
for (int j = 6; j < 8; j++) {
segments[1][i][j - 6] = lcdArray[i][j];
}
for (int j = 8; j < 10; j++) {
segments[2][i][j - 8] = lcdArray[i][j];
}
for (int j = 10; j < 12; j++) {
segments[3][i][j - 10] = lcdArray[i][j];
}
}
Then you use the function getNumber(), passing each of the bi-dimensional arrays and thus getting an array of the individual numbers displayed on the LCD.
for (int i = 0; i < digits.length; i++) {
digits[i] = getNumber(segments[i]);
}
Lastly, you recompose the individual digits into the real value of the weight by multiplying the hundreds by 100, the tens by 10, the units by 1, and the decimals by 0.1. This real value is stored in the float variable data.
data = digits[0] * 100 + digits[1] * 10 + digits[2] + digits[3] * 0.1;
}
You use the function getNumber() to get the decimal number represented by each of your seven-segment bi-dimensional arrays. This process starts by extracting the seven relevant bits of the incoming array and storing them in a linear array. You compare this array to each one of the number blueprints that you defined previously. If it matches one of them, the corresponding number will be returned.
public int getNumber(int segments[][]) {
int flatSegment[] = new int[7];
for (int i = 0; i < segments.length; i++) {
for (int j = 0; j < segments[0].length; j++) {
if (i + j == 0) {
flatSegment[i * 2 + j] = segments[i][j];
}
else if (!(i == 0 && j == 1)) {
flatSegment[i * 2 + j - 1] = segments[i][j];
}
}
}
if (Arrays.equals(flatSegment, zero)) { return 0; }
else if (Arrays.equals(flatSegment, one)) { return 1; }
else if (Arrays.equals(flatSegment, two)) { return 2; }
else if (Arrays.equals(flatSegment, three)) { return 3; }
else if (Arrays.equals(flatSegment, four)) { return 4; }
else if (Arrays.equals(flatSegment, five)) { return 5; }
else if (Arrays.equals(flatSegment, six)) { return 6; }
else if (Arrays.equals(flatSegment, seven)) { return 7; }
else if (Arrays.equals(flatSegment, eight)) { return 8; }
else if (Arrays.equals(flatSegment, nine)) { return 9; }
else {
return 0;
}
}
The setState() function is used to set the state of the scale from the main sketch. You perform a check of the state of the scale and set the scanning and foundWeight Boolean values accordingly.
public void setState(int state) {
this.state = state;
if (state < 518 && state > 513) {
scanning = true;
}
if (state < 531 && state > 526 && scanning) {
foundWeight = true;
}
}
The following function displays the raw data in array form on screen.
public void displayArray(int x, int y) {
pushMatrix();
translate(x, y);
for (int i = 0; i < lcdArray[0].length; i++) {
for (int j = 0; j < lcdArray.length; j++) {
text(lcdArray[j][i], 20 * i, j * 20);
}
}
popMatrix();
}
}
Note that the last curly brace the closing curly brace for the whole class.
Using the Weight Data
You are going to write a first simple sketch to test the decoding process. Then you will use it with your Kinect data, so let’s check that it works properly before starting to make things more complex.
Start a new sketch and create a separate tab in which you paste the LCD class. Then go back to the main tab and add all the necessary code to acquire serial communication. Then declare and initialize the LCD object, and declare a character array to store the incoming data.
import processing.serial.*;
Serial myPort;
PFont font;
LCD lcd;
char[][] stringData = new char[4][13];
int currentLine;
public void setup() {
size(400, 300);
font = loadFont("SansSerif-14.vlw");
textFont(font);
String portName = Serial.list()[0]; // This gets the first port on your computer.
myPort = new Serial(this, portName, 9600);
myPort.bufferUntil('
'),
lcd = new LCD();
}
The main draw() function updates the LCD with the latest data from Arduino, displays the array data, and draws the result weight on the screen, as shown in Figure 10-11.
public void draw() {
background(0);
lcd.setLcdArray(stringData);
lcd.displayArray(50, 50);
text(lcd.data + " " + lcd.units, 50, 200);
}
Figure 10-11. Testing the LED data decoder
You have worked with the serialEvent() function previously in this book, so it shouldn’t be completely unknown to you. Trim the newline character from your incoming string, and then check whether you have received a start or end of communication character.
public void serialEvent(Serial myPort) {
String inString = myPort.readString();
inString = inString.substring(0, inString.indexOf('
') - 1);
If you receive an S, set the current line to 0, so all new data is stored on the first line of your data array.
if (inString != null) {
if (inString.equals("S")) {
currentLine = 0;
}
If the first character of your incoming string is an X, this means you are done with this reading. The numbers following the X are the reading of your LCD state. Pass that value to the LCD object.
else if (inString.charAt(0) == 'X') {
String newState = inString.substring(1);
lcd.setState(Integer.valueOf(newState));
}
Otherwise, check if the length of the array is correct, that you are not beyond the fourth line, and store the values of the incoming characters into the stringData array.
else {
if (inString.length() == 13 && currentLine<4) {
for (int i = 0; i < stringData[0].length; i++) {
stringData[currentLine][i] = inString.charAt(i);
}
currentLine++;
}
}
}
else {
// increment the horizontal position:
println("No data to display");
}
}
And that’s all! When you run this you should see the array of ones and zeros on the screen defining the LCD pins information and the real value of the weight displayed on the scale. You’re done with the weight data acquisition; let’s have fun with it and your Kinect data!
Implementing the Biometric Recognition
You have probably had enough of playing with your scale by now, so you’re going to plunge straight into the final piece of code that will use skeleton tracking to recognize users and the weight input to generate body mass index (BMI) values. The basic functioning of the program is the following; whenever a user enters the field of vision of Kinect, you start tracking him. Find the highest and lowest points defining your user and extract the height of the person from those points. At the same time, you skeletonize the user and extract the proportions of their limbs to use as a footprint to match the user with previously saved information. If the footprint is similar to an existing user, you recognize him and retrieve the data from previous readings. You also have an input interface that allows you to create new users, stating their names and dates of birth. Once you have all of the skeleton and height data, the user steps on the bathroom scale and you read his weight. You then save all the current information in an external CSV file and calculate the person’s BMI and display it on screen.
Note A CSV (comma-separated values) file is a file format that stores tabular data separated by commas. It is used extensively in the transfer of data between applications.
Controlling the height and weight of children from early life up to adulthood can be a good way to keep track of adequate development, so for the sake of this exercise, compare your data to the male BMI-for-age chart for children less than 20 years old. You can find this chart at http://www.cdc.gov/growthcharts/clinical_charts.htm. This chart applies to US citizens; you can probably find equivalent charts from your government’s health program.
Imports and Variable Declaration
The first thing you need to do in your Processing sketch is to import the external libraries that you need for this process. You are building a basic user interface, so import Java AWT and AWT.event, which are part of the Java Foundation Classes and allow you to interact with your sketch in runtime by introducing new users.
import processing.serial.*;
import SimpleOpenNI.*;
import java.awt.*;
import java.awt.event.*;
SimpleOpenNI kinect;
Serial myPort;
PFont font;
PFont largeFont;
LCD lcd;
You are going to be juggling quite a few external files, so next you define an array called files that stores all the previous user data files that exist in the data folder. You also declare a current user file, two AWT TextFields, and one button for the user interface.
// User Input and Files
File[] files; // Existing user files
File userFile; // Current user
TextField name;
TextField birthdate;
Button b = new Button("New User");
The serial variables are similar to previous examples. The stringData array stores the incoming information from Arduino and one integer, currentLine, which you use in the process of parsing the incoming data. The Boolean scan is set to true, and then it is changed to false once you have acquired the final weight information.
// Serial Variables
boolean serial = true;
char[][] stringData = new char[4][13];
int currentLine;
boolean scan = true;
The numeric user data is stored in the array of floats userData. You also store some user information in String format, such as the name and date of birth.
// User Data
float[] userData = new float[11];
int user; // Current user ID
String userName = "new User";
String userBirthDate = "00/00/0000";
int userAge;
float userBMI;
float userHeightTemp;
You declare a double array to keep the data in the chart that you want to display, and a File object defining the chart CSV file. The String today contains the current date.
// Charts
float[][] chart;
File chartFile;
String today;
Setup Function
The setup() function is slightly longer than usual due to the user interface elements and the objects managing the external files. The first half of the function deals with the Simple-OpenNI object and the Processing fonts.
Along with the depthMap and the user capabilities, you enable the scene from NITE. The scene allows you to separate the background from the foreground user. This means you can select the 3D points that make up the user’s volume and extract dimensions from it, namely the user height (Figure 10-12).
Figure 10-12. Kinect scene: the user is automatically recognized.
public void setup() {
size(1200, 480);
smooth();
kinect = new SimpleOpenNI(this);
kinect.setMirror(true);
kinect.enableDepth();
kinect.enableScene();
kinect.enableUser(SimpleOpenNI.SKEL_PROFILE_ALL);
font = loadFont("SansSerif-12.vlw");
largeFont = loadFont("SansSerif-14.vlw");
textFont(largeFont);
lcd = new LCD();
if (serial) {
String portName = Serial.list()[0]; // This gets the first port
myPort = new Serial(this, portName, 9600);
myPort.bufferUntil('
'), // don't generate a serialEvent() unless you get a newline
}
User Interface
Now you initialize the AWT objects that constitute your user interface (Figure 10-13). The input is based on two Java AWT TextFields and one button. You actually position these three elements out of the Processing frame, right under it. When one user is detected, you resize the height of the Processing frame, but not the applet, so you then expose the part of the frame containing the interface. The two text fields contain the name and date of birth of the new user.
// Setting Up the User Input Elements
name = new TextField(40);
birthdate = new TextField(40);
frame.add(name);
frame.add(birthdate);
frame.add(b);
name.setBounds(20, height + 27, 200, 20);
name.setText("Your Name");
birthdate.setBounds(260, height + 27, 200, 20);
birthdate.setText("Date of Birth dd/mm/yyyy");
b.setBounds(520, height + 27, 100, 20);
You need to add an action listener to your button so an action is triggered when you press it. In your case, you store the user name and date of birth and you call the function newUser(), passing these strings as parameters.
b.addActionListener(new ActionListener() {
public void actionPerformed(ActionEvent e) {
userName = name.getText();
userBirthDate = birthdate.getText();
newUser(userName, userBirthDate);
}
}
);
You want to be able to recognize users created in previous sessions. For this, you need to scan the data folder in search of CSV files starting with the characters “user_”. Use Java’s FilenameFilter for this purpose. The following lines first get a list of all files in the data folder and then apply the filter and store the resulting files into the File array files. You then print the file names to the console for debugging.
// Acquire the user files already on the folder
File directory = new File(dataPath(""));
FilenameFilter filter = new FilenameFilter() {
public boolean accept(File dir, String name) {
return name.startsWith("user_");
}
};
files = directory.listFiles(filter);
println("List of existing user files");
for (File f : files) {
println(f);
}
To finish the setup() function, you choose the chart you are going to be comparing your results to and trigger the function readChart(), which retrieves all the data from the file. You also get the current date with the function getDate().
// Acquire the chart file
chartFile = new File(dataPath("charts/Male_2-20_BMI.CSV"));
readChart(chartFile);
getDate();
}
Draw Function
The draw() loop, in contrast to the setup() function, is quite simple because you have implemented the main routines into separate functions that are detailed next. The first thing you do in each loop is to update the data from the Simple-OpenNI and LCD objects.
public void draw() {
kinect.update();
background(0);
lcd.setLcdArray(stringData);
image(kinect.sceneImage(), 0, 0);
Then, if you have detected a user, you update the height data, as you are taking it from the user point cloud.
if (kinect.getNumberOfUsers() != 0) {
updateUserHeight();
}
If you are tracking a skeleton as well, you draw it on screen and update the user data from his limbs proportions.
if (kinect.isTrackingSkeleton(user)) {
drawSkeleton(user);
updateUserData(user);
}
Then, if you are still scanning the weight (scan is true), and the LCD object’s variable foundWeight is true (meaning you got a steady weight), you try to recognize the user from previous users, and then you store the first piece of data (the weight from the scale) in the first position of your userData array and the height in its second position. You then update the BMI value and save the user for future retrieval. You set the scan Boolean to false so you stop updating the values.
if (lcd.foundWeight && scan) {
userRecognition();
userData[0] = lcd.data;
userData[1] = userHeightTemp;
userBMI = userData[0]/pow(userData[1] /1000, 2);
saveUser();
scan = false;
}
And finally, you take care of the visualization by printing the chart and the user data on the right side of your Processing screen.
printUserData();
drawChart();
}
Additional Functions
You probably guessed it: a short draw loop means a quite extensive list of additional functions! Some of these functions are very similar to code used in previous examples, so we won’t go into detail on those. Others are more complex and will be described exhaustively, explaining what they do and when they are called.
Before you get into the user functions, there is a helper function called getDate() that uses the Java Calendar class to fetch the current date and store it in the variable today. It also calculates the age of the current user by subtracting the user’s date of birth from the current date.
public void getDate() {
Calendar currentDate = Calendar.getInstance();
SimpleDateFormat formatter = new SimpleDateFormat("dd/MM/yyyy");
today = formatter.format(currentDate.getTime());
Date todayDate = currentDate.getTime();
Date userDate = null;
try {
userDate = formatter.parse(userBirthDate);
}
catch (ParseException e) {
e.printStackTrace();
}
long diff = todayDate.getTime() - userDate.getTime();
userAge = (int) (diff/31556926000l);
}
New User
The newUser() function is called whenever you press the New User button in your user interface. The user name and date of birth are passed as arguments.
You first need to check if you have pressed the button without entering a name or a date, in which case you prompt the user to insert the data. Then, you use the Java function DateFormat() to check whether the date that has been inserted corresponds to the format you are expecting. If it doesn’t, a parse exception is thrown and caught by your try-catch statement, and you prompt the user to insert a properly formatted date of birth (which doesn’t mean it is the real one anyway…).
If the data is correct, you proceed to check whether there is already a user with that name in the system, in which case you simply use that file for the rest of the time. If it doesn’t exist, you create it and add the user name and date of birth to the first line in the file. You finally flush and close the PrintWriter.
public void newUser(String name, String birth) {
// Start the user file
if (name.equals("Your Name") || birth.equals("Date of Birth dd/mm/yyyy")) {
println("Please insert your name and date of Birth");
}
else {
DateFormat format = new SimpleDateFormat("dd/MM/yyyy");
try {
format.parse(birthdate.getText());
PrintWriter output;
userFile = new File(dataPath("user_" + userName + ".csv"));
if (!userFile.exists()) {
output = createWriter(userFile);
output.println(name + "," + birth);
output.flush(); // Write the remaining data
output.close(); // Finish the file
getDate();
println("New User " + name + " created successfully");
}
else {
setPreviousUser(userFile);
}
}
catch (ParseException e) {
System.out
.println("Date " + birthdate.getText()
+ " is not valid according to "
+ ((SimpleDateFormat) format).toPattern()
+ " pattern.");
}
}
}
Updating User Data
You call updateUserData() from the draw() loop in every cycle if you are tracking a skeleton. This function first stores all the real world coordinates of the user’s joints; then it calculates the dimensions of arms, forearms, thighs, shins, and shoulder breadth; and then it stores all of this data in the userData array, starting from the third position (the first two are occupied by the weight and height).
This array serves later as an identifier for the user. The body proportions stored in the userData array can be matched and compared to other users’ identifiers and thus the person tracked can be recognized. The first step is to initialize all the PVectors that store the location of the user’s joints.
void updateUserData(int id) {
// Left Arm Vectors
PVector lHand = new PVector();
PVector lElbow = new PVector();
PVector lShoulder = new PVector();
// Left Leg Vectors
PVector lFoot = new PVector();
PVector lKnee = new PVector();
PVector lHip = new PVector();
// Right Arm Vectors
PVector rHand = new PVector();
PVector rElbow = new PVector();
PVector rShoulder = new PVector();
// Right Leg Vectors
PVector rFoot = new PVector();
PVector rKnee = new PVector();
PVector rHip = new PVector();
Then you use the getJointPositionSkeleton() function to get the real world coordinates into the vectors.
// Left Arm
kinect.getJointPositionSkeleton(id, SimpleOpenNI.SKEL_LEFT_HAND, lHand);
kinect.getJointPositionSkeleton(id, SimpleOpenNI.SKEL_LEFT_ELBOW, lElbow);
kinect.getJointPositionSkeleton(id, SimpleOpenNI.SKEL_LEFT_SHOULDER, lShoulder);
// Left Leg
kinect.getJointPositionSkeleton(id, SimpleOpenNI.SKEL_LEFT_FOOT, lFoot);
kinect.getJointPositionSkeleton(id, SimpleOpenNI.SKEL_LEFT_KNEE, lKnee);
kinect.getJointPositionSkeleton(id, SimpleOpenNI.SKEL_LEFT_HIP, lHip);
// Right Arm
kinect.getJointPositionSkeleton(id, SimpleOpenNI.SKEL_RIGHT_HAND, rHand);
kinect.getJointPositionSkeleton(id, SimpleOpenNI.SKEL_RIGHT_ELBOW, rElbow);
kinect.getJointPositionSkeleton(id, SimpleOpenNI.SKEL_RIGHT_SHOULDER, rShoulder);
// Right Leg
kinect.getJointPositionSkeleton(id, SimpleOpenNI.SKEL_RIGHT_FOOT, rFoot);
kinect.getJointPositionSkeleton(id, SimpleOpenNI.SKEL_RIGHT_KNEE, rKnee);
kinect.getJointPositionSkeleton(id, SimpleOpenNI.SKEL_RIGHT_HIP, rHip);
And finally, you store the distances between certain joints in the userData array.
userData[2] = PVector.dist(rHand, rElbow); // Right Arm Length
userData[3] = PVector.dist(rElbow, rShoulder); // Right Upper Arm Length
userData[4] = PVector.dist(lHand, lElbow); // Left Arm Length
userData[5] = PVector.dist(lElbow, lShoulder); // Left Upper Arm Length
userData[6] = PVector.dist(lShoulder, rShoulder); // Shoulder Breadth
userData[7] = PVector.dist(rFoot, rKnee); // Right Shin
userData[8] = PVector.dist(rHip, rKnee); // Right Thigh
userData[9] = PVector.dist(lFoot, lKnee); // Left Shin
userData[10] = PVector.dist(lHip, lKnee); // Left Thigh
}
Updating User Height
The detection of the user’s height is not based on skeleton tracking but directly on the point cloud, so you implement a separate function for it. This function is called from draw in case you’re tracking a user in the scene.
You have previously worked with functions that parsed and printed on screen the Kinect point cloud. This function is similar, but you won’t print any of the points. What you want to do is to find out the higher and lower points from the user point cloud so you can subtract their Y coordinates and thus find the real height of the user in millimeters (Figure 10-14).
Figure 10-14. User height from highest and lowest points
You will only use one of every three points to speed up the process; if you want a more precise measurement, you can decrease the steps to 1. Use userMap from Simple-OpenNI to get a flat image of the points that define the user.
void updateUserHeight() {
int steps = 3;
int index;
int[] userMap = kinect.getUsersPixels(SimpleOpenNI.USERS_ALL);
PVector userCenter = new PVector();
PVector[] realWorldPoint = new PVector[kinect.depthHeight() * kinect.depthWidth()];
Then you initialize the userTop and userBottom PVectors to the center of the mass of the user, so you can start looking for a top and bottom pixel from a neutral ground.
kinect.getCoM(1, userCenter); // Get the Center of Mass
PVector userTop = userCenter.get();
PVector userBottom = userCenter.get();
And you run a loop checking every point in the depth map looking for user points. You determine that a point belongs to a user if the equivalent pixel in the userMap is not black. If you find a user point, you check if it’s higher than the higher pixel found or lower than the lowest pixel found; in positive case, you set it as the highest or lowest point.
for (int y = 0; y < kinect.depthHeight(); y += steps) {
for (int x = 0; x < kinect.depthWidth(); x += steps) {
index = x + y * kinect.depthWidth();
realWorldPoint[index] = kinect.depthMapRealWorld()[index].get();
if (userMap[index] != 0) {
if (realWorldPoint[index].y > userTop.y)
userTop = realWorldPoint[index].get();
if (realWorldPoint[index].y < userBottom.y)
userBottom = realWorldPoint[index].get();
}
}
}
Once you have found the highest and lowest points, you store the value of the difference of their Y coordinates into userHeightTemp. But you won’t do it straight away. You want to smooth the noisy data to avoid continuous oscillations of the value, so you use Processing’s lerp() function to progressively make the transition from one value to the next one.
userHeightTemp = lerp(userHeightTemp, userTop.y - userBottom.y, 0.2f);
}
Saving the User
If you have successfully created a user and tracked his weight, height, and proportions, now you want to store this information into the previously created CSV file (Figure 10-15).
For this, you create a File object with the name of the current user, and if you find that it doesn’t exist yet, you prompt the user to insert his name and birth date. If the file does exist, use a Java BufferedWriter to append information to the file without wiping it out.
Figure 10-15. Structure of the user information CSV file
private void saveUser() {
userFile = new File(dataPath("user_" + userName + ".csv"));
if (!userFile.exists()) {
println("Please Enter your Name and Date of Birth");
} else {
try {
FileWriter writer = new FileWriter(userFile, true);
BufferedWriter out = new BufferedWriter(writer);
out.write(today);
for (int i = 0; i < userData.length; i++) {
out.write("," + userData[i]);
}
out.write("
");
out.flush(); // Write the remaining data
out.close(); // Finish the file
}
catch (IOException e) {
e.printStackTrace();
}
println("user " + userName + " saved");
}
}
User Recognition
Now you have all the data you need from your current user, and you want to know if the same user was previously registered in the system. This means that there is a CSV file containing very similar limb lengths to the ones of the current user.
In order to find this out, you make use of the generalization of the concept of vector subtraction to a higher-dimensional space. What? Well, the next few lines are a technical explanation of how to measure the similitude between two users based on the skeleton data. If you are not that interested in multidimensional vector operations, simply skip the math and use the code provided next!
Vector Subtraction in Higher Dimensions
You are using the Cartesian coordinate system with basis vectors.
If the subtraction of two two-dimensional vectors is
and the subtraction of two three-dimensional vectors is
then the subtraction of n-dimensional vectors will be
This means that you can treat your user data array as a multidimensional vector and calculate the distance or difference between two users as the magnitude of the vector resulting from subtracting the two.
This is performed by the function getUserDistance(), which takes an array as parameter and returns the magnitude of the vector resulting from the subtraction of the vector “pattern” and the one formed by the user data, excluding the two first parameters, so the recognition doesn’t get affected by changes in weight or height.
float getUserDistance(float[] pattern) {
float result = 0;
for (int i = 0; i < pattern.length; i++) {
result += pow(pattern[i] - userData[i + 2], 2);
}
return sqrt(result);
}
Figure 10-16. Users being compared to previously stored information
The function userRecognition() is the bit of code that deals with comparing the current user to the information previously stored in CSV files (Figure 10-16). First, you create an array of floats that stores the distances of the current user to the various previously stored users, hence the size of it being files.length.
Then, for each user file, you read its content; if the line contains more than two chunks of data (so it’s not the header), you store all the relevant floats in the array pattern, and you get its distance to your current user data using the previously defined function getUserDistance().
private void userRecognition() {
float[] distances = new float[files.length];
for (int i = 0; i < files.length; i++) {
String line;
try {
BufferedReader reader = createReader(files[i]);
while ( (line = reader.readLine ()) != null) {
String[] myData = line.split(",");
if (myData.length > 2) {
float[] pattern = new float[9];
for (int j = 3; j < myData.length; j++) {
pattern[j - 3] = Float.valueOf(myData[j]);
}
distances[i] = getUserDistance(pattern);
}
}
}
catch (IOException e) {
e.printStackTrace();
println("error");
}
}
Once you have all the distances, or differences, you only need to find the user that is closest to your current user; if this distance is small enough (after some testing, we set the minimum to 100), you set the user in the file as the current user (Figure 10-17).
int index = 0;
float minDist = 1000000000;
for (int j = 0; j < distances.length; j++) {
if (distances[j] < minDist) {
index = j;
minDist = distances[j];
}
}
if (minDist < 100)
setPreviousUser(files[index]);
println("Recognising User");
}
The function setPreviousUser() simply retrieves the user data from the selected file and updates the user name and birth date.
private void setPreviousUser(File file) {
String line;
try {
BufferedReader reader = createReader(file);
line = reader.readLine();
String[] myData = line.split(",");
userName = myData[0];
userBirthDate = myData[1];
getDate();
}
catch (IOException e) {
e.printStackTrace();
println("error");
}
println("User set to " + "userName, born on" + userBirthDate);
}
Figure 10-17. Users being compared
Reading and Drawing the Chart
The following functions follow the same principles and methods you used to read the user data from external files, but this time you read the chart files and transform the numeric data into graphs representing the percentiles of BMI.
private void readChart(File file) {
String line;
int lineNumber = 0;
try {
int lines = count(file);
chart = new float[lines][11];
BufferedReader reader = createReader(file);
while ( (line = reader.readLine ()) != null) {
String[] myData = line.split(",");
if (lineNumber == 0) {
}
else {
for (int i = 0; i < myData.length; i++) {
chart[lineNumber - 1][i] = Float.valueOf(myData[i]);
}
}
lineNumber++;
}
}
catch (IOException e) {
e.printStackTrace();
println("error");
}
}
The function count() is a helper function used to find out how many lines you will be reading from a CSV file.
public int count(File file) {
BufferedReader reader = createReader(file);
String line;
int count = 0;
try {
while ( (line = reader.readLine ()) != null) {
count++;
}
}
catch (IOException e) {
e.printStackTrace();
println("error");
}
return count;
}
The function drawChart() simply runs through the chart array and draws the lines defining the percentiles. You can apply this method to every chart you have included in the data folder of the sketch.
private void drawChart() {
float scaleX = 2;
float scaleY = 10;
pushStyle();
stroke(255);
noFill();
pushMatrix();
translate(0, height);
scale(1, -1);
translate(660, 20);
line(0, 0, 500, 0);
line(0, 0, 0, 300);
for (int i = 1; i < chart[0].length; i++) {
beginShape();
for (int j = 0; j < chart.length - 1; j++) {
vertex(chart[j][0] * scaleX, chart[j][i] * scaleY);
}
endShape();
}
strokeWeight(10);
stroke(255, 0, 0);
point(userAge*12*scaleX, userBMI*scaleY);
popMatrix();
popStyle();
}
Graphic Output
The following functions handle the on-screen representation of the user data in text form and the drawing of the user’s skeleton.
private void printUserData() {
fill(255);
text("Data at " + today, 660, 20);
text("Current User: " + userName, 660, 40);
text("Date of Birth " + userBirthDate, 660, 60);
text("Current Age " + userAge, 660, 80);
text("User Weight: " + userData[0] + " " + lcd.units, 660, 100);
text("User Height: " + userHeightTemp/1000 + " m", 660, 120);
text("User BMI: " + userBMI + " kg/m2", 660, 140);
fill(255, 255, 150);
text("BMI at ages 2-20", 1000, 20);
}
void drawSkeleton(int userId) {
pushStyle();
stroke(255, 0, 0);
strokeWeight(3);
kinect.drawLimb(userId, SimpleOpenNI.SKEL_HEAD, SimpleOpenNI.SKEL_NECK);
kinect.drawLimb(userId, SimpleOpenNI.SKEL_NECK, SimpleOpenNI.SKEL_LEFT_SHOULDER);
kinect.drawLimb(userId, SimpleOpenNI.SKEL_LEFT_SHOULDER, SimpleOpenNI.SKEL_LEFT_ELBOW);
kinect.drawLimb(userId, SimpleOpenNI.SKEL_LEFT_ELBOW, SimpleOpenNI.SKEL_LEFT_HAND);
kinect.drawLimb(userId, SimpleOpenNI.SKEL_NECK, SimpleOpenNI.SKEL_RIGHT_SHOULDER);
kinect.drawLimb(userId, SimpleOpenNI.SKEL_RIGHT_SHOULDER, SimpleOpenNI.SKEL_RIGHT_ELBOW);
kinect.drawLimb(userId, SimpleOpenNI.SKEL_RIGHT_ELBOW, SimpleOpenNI.SKEL_RIGHT_HAND);
kinect.drawLimb(userId, SimpleOpenNI.SKEL_LEFT_SHOULDER, SimpleOpenNI.SKEL_TORSO);
kinect.drawLimb(userId, SimpleOpenNI.SKEL_RIGHT_SHOULDER, SimpleOpenNI.SKEL_TORSO);
kinect.drawLimb(userId, SimpleOpenNI.SKEL_TORSO, SimpleOpenNI.SKEL_LEFT_HIP);
kinect.drawLimb(userId, SimpleOpenNI.SKEL_LEFT_HIP, SimpleOpenNI.SKEL_LEFT_KNEE);
kinect.drawLimb(userId, SimpleOpenNI.SKEL_LEFT_KNEE, SimpleOpenNI.SKEL_LEFT_FOOT);
kinect.drawLimb(userId, SimpleOpenNI.SKEL_TORSO, SimpleOpenNI.SKEL_RIGHT_HIP);
kinect.drawLimb(userId, SimpleOpenNI.SKEL_RIGHT_HIP, SimpleOpenNI.SKEL_RIGHT_KNEE);
kinect.drawLimb(userId, SimpleOpenNI.SKEL_RIGHT_KNEE, SimpleOpenNI.SKEL_RIGHT_FOOT);
popStyle();
}
Serial Event
The serialEvent() function is entirely derived from the previous example implemented in the “Using the Weight Data” section. This function acquires the data from Arduino and sets the LCD state to the incoming data.
public void serialEvent(Serial myPort) {
// get the ASCII string:
String inString = myPort.readString();
inString = inString.substring(0, inString.indexOf('
') - 1);
if (inString != null) {
if (inString.equals("S")) {
currentLine = 0;
}
else if (inString.charAt(0) == 'X') {
String newState = inString.substring(1);
lcd.setState(Integer.valueOf(newState));
}
else {
if (inString.length() == 13 && currentLine < 4) {
for (int i = 0; i < stringData[0].length; i++) {
stringData[currentLine][i] = inString.charAt(i);
}
currentLine++;
}
}
}
else {
println("No data to display");
}
}
Simple-OpenNI User Events
These are the same good old Simple-OpenNI callbacks as usual. The only hint of novelty is the last line in the onNewUser() function, which increases your Processing frame height by 54 pixels so the user interface is displayed.
public void onNewUser(int userId) {
println("onNewUser - userId: " + userId);
println(" start pose detection");
kinect.startPoseDetection("Psi", userId);
frame.setSize(width, height + 54);
}
public void onLostUser(int userId) {
println("onLostUser - userId: " + userId);
}
public void onStartCalibration(int userId) {
println("onStartCalibration - userId: " + userId);
}
public void onEndCalibration(int userId, boolean successfull) {
println("onEndCalibration - userId: " + userId + ", successfull: "
+ successfull);
if (successfull) {
println(" User calibrated !!!");
kinect.startTrackingSkeleton(userId);
user = userId;
}
else {
println(" Failed to calibrate user !!!");
println(" Start pose detection");
kinect.startPoseDetection("Psi", userId);
}
}
public void onStartPose(String pose, int userId) {
println("onStartdPose - userId: " + userId + ", pose: " + pose);
println(" stop pose detection"); kinect.stopPoseDetection(userId);
kinect.requestCalibrationSkeleton(userId, true);
}
public void onEndPose(String pose, int userId) {
println("onEndPose - userId: " + userId + ", pose: " + pose);
}
Summary
This has been quite a heavy project on the programming side. You had to acquire rather noisy data from a bathroom scale’s LCD, and you made sense of it by reverse-engineering its logic until you had the numeric data you wanted. You then built a whole framework for human biometric scanning and you implemented a user interface plus a data storage and retrieval system. You even managed to perform user recognition using NITE’s skeleton tracking capabilities, and you output all the gathered data on screen for visualization.
You used some Kinect biometrics in this project; obviously, there are innumerable spin-off projects to be generated from this. You could start by building a system to show all the available growth and weight charts on screen, or you could extract only the user-recognition implementation and build your own home security system. You could also implement a fitness evaluator from the user shape, and try to classify people according to body shapes and dimensions. If you have some commercial vision, you should definitely give a go to an online fitted clothes try-on service based on Kinect’s three-dimensional scanning!
Figure 10-18. Leonardo’s Vitruvian Man, and a not-so-Vitruvian Kinect user