C H A P T E R  6

Kinect Networked Puppet

by Enrique Ramos and Ciriaco Castro

In this chapter, you are going to make use of one of the most amazing capabilities of the Kinect: skeleton tracking. This feature will allow you to build a gesture-based puppeteering system that you will use to control a simple, servo-driven puppet.

Natural interaction applied to puppetry has an important outcome: the puppeteer (you) and the puppet don't need to be tied by a physical connection anymore. In the second part of the chapter, you will extend your project and implement network connectivity that will permit you to control the puppet remotely from any point in the world, as shown in Figure 6-1.

This project will introduce you to the ins and outs of skeleton tracking so you can detect the user's joints and limbs movements in space. The user data will be mapped to servo angles and transmitted via the Internet to a remote receiver program, which will in turn send them to the Arduino board controlling the servos of the robotic puppet.

Throughout the following pages, you will also be learning how to drive servos from Arduino and how to communicate over networks with a few lines of code using Processing's Net library.

Figure 6-1. Web puppet controlled by a remote user

The Puppet

We have set out to use only open source tools in this book, so we're going to use an open source design for the puppet. We have thus chosen the Android Robot by Google, the symbol for their open source mobile phone OS, which is licensed under the Creative Commons Attribution. The design will need to be slightly altered to add some joints to allow a higher range of movements. You will also design and build a stage were the robot will stand.

Servos

The movements of your puppet will be based on the rotation of nine small servos, so now is a good moment to understand the theory behind servo control. Servos are DC motors that include a feedback mechanism that allows you to keep track of the position of the motor at every moment. The way you control a servo is by sending it pulses with a specific duration. Generally, the range of a servo goes from 1 millisecond for 0 degrees to 2 milliseconds for 180 degrees. There are servos that can rotate 360 degrees, and you can even hack servos to allow for continuous rotation.

You need to refresh your control signals every 20ms because the servo expects to get an update every 20ms, or 50 times per second. Figure 6-2 shows the normal pattern for servo control.

Figure 6-2. Servo wave form

In Chapter 3, you learned about pulse width modulation (PWM) and you used it to dim an LED lamp. The way you control servos is very similar to PWM, but in this case, you will be dealing with the pulsing yourself instead of sending an analog value and allowing Arduino to break it down to pulses.

There is a Servo library included with Arduino, but you are going to write your own servo control functions. This will save you from a lot of problems when working with a number of servos. Let's see the differences in the two approaches to servo control with a simple example.

For the sake of this example, you will connect a servo to the digital pin 5 and control it with a potentiometer connected to analog pin 0. The following is the code you need for that using the Servo library. First, import the Servo library and declare the servo object.

In setup(), use the Servo.attach() function to set the servo to pin 5.

And in loop(), read the potentiometer value and map its values to a range of 0-180. Finally, write the angle value to the servo and add a delay to allow the servo to reach the new position.

If you were using your own servo control instead, this would look something like the following:

This requires a little explanation. The updating of the servos follows a very similar logic to the “blink without delay” Arduino example. (This is one of the examples included with Arduino IDE and is used to control time-depending events without the use of a delay. If you have never seen this sketch, go to the Arduino examples and have a good look at it. You will find it within 2. Digital in the Arduino examples.) You need to send a pulse to the servo every 20ms but you don't want to add delays to your code, so you're not interfering with other time-dependent routines included in the future project. When you have been running for longer than the specified interval, you update the servo. Note that the servo sends a HIGH pulse and then there is a delay. This one is inevitable because you need to send a very precise pulse, and anyway it's such a short delay (in microseconds) that you can live with it.

If you want to control several servos at the same time, you only need to add some more pins and update all of them at the same time! Let's add another servo and make it move just through half its rotational span (90 degrees).

This control technique is the one you will be using throughout the rest of the chapter and in further chapters that make use of servos. Try to familiarize yourself with the dynamics of servo control with a couple of servos only. When you are ready, you can move on to the following sections where you will be controlling up to nine servos at the same time.

Building the Stage

Now that you know what a servo is and how it works, let's move on to an easier bit: the stage. Basically you want a background and some support from which to hang the puppet. You're using black Perspex so your robot will stand out. The stage consists of a hollow base to hide all the mechanisms plus a backdrop, as you can see in Figure 6-4.

Figure 6-3 shows the stage parts as per the Illustrator file provided (C06_Stage_Parts.ai). The two small rectangular pieces have been added in order to support your Arduino and give stability to the stage. Note that the front is slightly raised to hide the gap between the robot's feet and the stage.

The two round holes on the central element permit the servo cables to reach the Arduino board at the base. The rectangular hole is for the servo that rotates the robot.

Figure 6-3. Stage parts

Figure 6-4. Assembled stage

The puppet rotation servo is attached to the base with four screws, as shown in Figure 6-5.

Figure 6-5. Servo fixed to the base

The robot is attached to two supports that are also made of black Perspex so they don't stand out from the background, as shown in Figure 6-6. These supports are connected to a disc, thereby allowing rotation. This disc will be screwed to a servo circular horn, and then attached to the rotation servo that you have previously fixed to the base.

Figure 6-6. Servo arm fixed to disc (left), and robot supports (right)

Building the Puppet

The puppet is made of laser-cut green Perspex. If you are thinking of using the same design, you can use the Illustrator vector files included with this book (C06_Puppet_Parts.ai) and shown in Figure 6-7.

Figure 6-7. Puppet pieces

There is a front body that covers up all the mechanisms and a back body that connects all the servos and is fixed to the support. Both bodies are connected by four bolts passing through the holes provided for that purpose, which you can see in Figure 6-8.

Figure 6-8. Assembling servos to the back body

Now you can start to assemble the legs and arms, screwing the double servo arm, and fixing a second servo, as shown in Figure 6-9.

Figure 6-9. Arms/legs with servomotors

Before screwing and fixing all arms to the corresponding servos, you should set all servos to the middle position (1500) so later you can map them in an easy way. The complete arm will look like that in Figure 6-10.

Figure 6-10. Arm assembled

Repeat the same process with the other arm and legs. Once you have all these parts assembled, attach the arms and legs to the back body piece and fix the puppet onto the stage, as shown in Figure 6-11. You're now ready to build the circuit!

Figure 6-11. The assembled puppet on the stage. The front of the body has not been attached yet.

The front body piece is the last thing that you assemble once everything is double-checked. Now you have the puppet up and running.

Building the Circuit

The circuit that you are going to prepare is going to be plugged into the Arduino board and is designed to control nine servomotors. You first run all servos using just the power of your USB connection, but you then add an external power source, as all servos working at the same time will exceed the power of the USB (Figure 6-12). You will use Arduino pins 3, 4, 5, 6, 7, 8, 9, 10, and 11, following the diagram in Figure 6-13.

Build your circuit on a strip board. Start by measuring the amount of strips that you need and soldering the breakaway headers. Make sure that the Arduino USB cable is at one extreme of the board so the cable can be plugged and unplugged without interfering with the breakaway headers.

Figure 6-12. Circuit components

Figure 6-13. Circuit diagram

Double-check the direction of the copper strips on the back of the strip board, and start scratching the strips so you are sure that you don't connect the two rows of pins on your Arduino.

You can use a potentiometer to check the resistance between strips to make sure that there is no connection. After that, you are ready to start soldering the breakaway header corresponding to Arduino pins 3, 4, 5, 6, 7, 8, 9, 10, and 11. Then you solder a breakaway header for ground and 5V.

After soldering, double-check that they are placed correctly by plugging the shield into your Arduino board and making sure that it attaches easily (Figure 6-14). You might need to bend some of the headers with a nipper to get the shield properly connected.

Figure 6-14. Strip board and Arduino

Now start soldering the breakaway headers for your servos. Before doing this, look at the servo cables. Servos have three cables: black for ground, red (middle one) for power, and yellow or brown for the signal. The signal comes from the Arduino pin; power and ground can come from Arduino 5V and ground or from an external power source. The important thing is that the circuit ground is connected to Arduino ground; otherwise you won't be able to control your servos, and they will behave in strange ways.

Due to the geometry of the strip board, you should arrange the headers in two columns, leaving a group of four headers together (2 for arms, 2 for legs) and one for the rotation of the robot. The circuit will look like that in Figure 6-15.

Figure 6-15. Strip board and headers

After completing the soldering, it's time to scratch the connection between the signals (yellow cable of the servo motors). With a little patience and a scalpel, remove the copper that connects the line (see Figure 6-16). After you have removed all the material, check that the connection is completely broken by using a multimeter.

Figure 6-16. Scratched strip board

The next step is to connect power and ground. Solder a cable that connects the 5V pin to the power column (middle pin of the header) and a small cable that connects the power of the other column (Figure 6-17). Repeat the same process with the ground (one cable form Arduino ground to the ground column, and a small cable connecting the two columns ground between them).

Figure 6-17. Strip board, power, and ground

Now, connect all the signals to the corresponding pin. You need to solder one cable connecting the Arduino pin strip to the header signal. It is important that you do this in a tidy and orderly manner (see Figures 6-18 and 6-19) so it's easy for you to remember which pin number connects to each servo.

Figure 6-18. Strip board, all pins connected front

Figure 6-19. Strip board, all pins connected back

Then you can plug in your Arduino (Figure 6-20) and do a first test.

Figure 6-20. Strip board, connected for test

Testing the Servos

The code for this first test is quite similar to the one in the servo introduction, but instead of using a potentiometer you are going to keep the servo moving from 0 to 180 degrees and back to zero. This way you can check if all of the servos work properly or if you have to debug the circuit (making sure that the soldering is right!).

The code calls all the servo pins as outputs and updates each pin with a pulse that will change between values 500-2500 (0-180 degrees). Once it arrives at 2500, it will decrease to 500.

Once you upload the code, all servos should start to move. If one servo doesn't move, check the circuit, the connections, and the Arduino pin (in case the fault is in the code).

Obviously, the Arduino 5V power isn't enough for running all nine servos at the same time. You can see that the Arduino power LED starts to fade out and blink; also, the servos aren't moving properly and instead make a hiss, like they are struggling. To avoid this, add an external power source to your circuit.

The first thing is to cut the cable that connects with the Arduino 5V pin (you don't want to fry your Arduino!). Then you connect a 3-battery clip to your circuit (see Figure 6-21). Solder the red cable power to the servo power stripe and the ground to the servo ground stripe. Note that ground is still connected to Arduino ground.

Figure 6-21. Circuit with battery clip

So now you can have a pack of four AA batteries in a battery holder connected to the battery clip. This should be enough power to move all your servos. Another option is to connect a power adapter plugged into a mains socket, but make sure that the output voltage is 5V. Later in the book, we will explain another way of getting a constant 5V current using a computer power supply.

Setting the Servos to the Starting Position

Reconnect your Arduino and the batteries and check that all servos move smoothly. You're almost ready to start coding. The last thing you need to do is set all servos to a starting position, so you can fix all of the servo arms.

You're going to send all shoulder and leg servos to 500 (0 degrees) or 2500 (180 degrees) depending if they are on the left side or right side, and the elbows to 1500 (90 degrees), allowing the movement in two directions. The last thing is to position the servo that will rotate the puppet to 1500 (90 degrees). Figure 6-22 shows all the angles.

Figure 6-22. Puppet servo angles

The code is similar to the previous; you're just commenting or deleting some lines (the ones that change the servo pulse) and adding new servo pulses for the 90 and 180 degrees position.

That servo position will be the starting position of the puppet. Once this position has been fixed, you know the orientation of all limbs so you can screw in your servo arms. Then you cover everything with the front body piece and use some cable ties to hold the cables in position (see Figure 6-23). Just make sure you keep enough length to allow the movement!

Figure 6-23. Puppet completed

Skeleton Tracking On-Screen

The first thing you need to get sorted for this project is tracking your skeleton with Kinect. We briefly described skeletonization in Chapter 3, but here you are going to delve deeply into it, so hang on!

You will start off by tracking the skeleton on-screen, and then you will attach the virtual puppet to the physical one you have just built. Simple-OpenNI provides a nice interface for skeleton tracking and a couple of callback functions to which you can add your own code. Start by importing and initializing Simple-OpenNI.

Set the mirroring to On so it's easy for you to relate to your on-screen image, and enable the depth image and the User. Pass the parameter SimpleOpenNI.SKEL_PROFILE_ALL to this function to enable all the joints. Set your sketch size to the depth map dimensions.

In the draw() loop, simply update the Kinect data, draw the depth map, and, if Kinect is tracking a skeleton, call the function drawSkeleton() to print your skeleton on screen.

We took this drawSkeleton() function from one of the examples in Simple-OpenNI. It takes an integer as a parameter, which is the user ID (it will be always 1 if you are just tracking a single skeleton). This function runs through all the necessary steps to link the skeleton joints with lines, defining the skeleton that you will see on screen. The function makes use of the public method drawLimb() from the Simple-OpenNI class.

Simple-OpenNI Events

Use the Simple-OpenNI callback functions to trigger the pose detection and skeleton tracking capabilities. The function onNewUser() is called when a user is detected. You know the user is there, but you haven't calibrated their skeleton yet, so you start the pose detection.

When the user is lost, you simply print it on the console.

When you detect a pose, you can stop detecting poses (you already have one!) and request a skeleton calibration to NITE.

You also display messages when the pose is finished and when the calibration has started.

If the calibration is finished and it was successful, you start tracking the skeleton. If it wasn't, you restart the pose detection routine.

Run this sketch and stand in front of your Kinect in the start pose. After a few seconds, your skeleton should appear on screen, and follow your every movement until you disappear from screen (see Figure 6-24).

Figure 6-24. Start position and calibrated user

Angle Calculation

With the previous program, you were able to track your skeleton and display it on screen. That is the base of your puppet control. Now you need to implement a routine that translates your poses to the angles of eight servos that will ultimately drive the puppet.

The eight servos are placed on the shoulders, elbows, hips, and knees of your puppet. If you can find out the current bending angles of your joints, you can map those angles to the servos and thus make the puppet move like you. You implement this with the cunning use of geometry.

Taking the previous code as a base, you include a series of routines that will use the skeleton to calculate all the joints' angles. First, add some PVectors at the beginning of the sketch (outside of any function) to store the position of all your joints.

You also need a PVector array to contain the angles of your joints.

You're interested in nine angles, one for each servo on your puppet.

• angles [0] Rotation of the body
• angles [1] Left elbow
• angles [2] Left shoulder
• angles [3] Left knee
• angles [4] Left Hip
• angles [5] Right elbow
• angles [6] Right shoulder
• angles [7] Right knee
• angles [8] Right Hip

You will be wrapping all these calculations into a new function called updateAngles() that you call from your main draw() function only in the case you are tracking a skeleton. Add this function to the if statement at the end of the draw() function, and then print out the values of the angles array.

You need a function that calculates the angle described by the lines joining three points, so implement it next; call it the angle() function.

The function updateAngles() performs the calculation of all these angles and stores them into the previously defined angles[] array. The first step is storing each joint position in one PVector. The method getJointPosition() helps you with this process. It takes three parameters: the first one is the ID of the skeleton, the second one is the joint you want to extract the coordinates from, and the third one is the PVector that is set to those coordinates.

Now you have all your joints nicely stored in the PVectors you defined for that purpose. These are three-dimensional vectors of real-world coordinates. You transform them to projective coordinates so you get the second angles (remember your puppet is pretty flat!), but first you need to extract your body rotation angle (see Figure 6-25), which is the only rotation of the plane that you need.

Figure 6-25. Rotation of the body

After this angle has been calculated, you can transform all the joints to projective coordinates (on screen coordinates).

And finally, you use the angle function you implemented previously to compute all the necessary angles for the control of the puppet.

Now you have all the data you need to control your puppet! If you run the angle calculation sketch and start the skeleton tracking, you will see the values of your joints' angles appear on the console. You could now connect the serial cable to your computer, add a serial communication protocol, and start driving the puppet straight away. But let's build some flexibility into this application. You want to be able to control the puppet remotely through the Internet, and to do this you need to learn some basics of network communication.

Network Communication

In previous chapters you used Serial communication to send messages from your computer to the Arduino board. Now you are going to follow an analogous process to let two machines communicate over the Internet.

Luckily, Processing includes a core library called Network that will help you go about the network communication with just a few lines of code. Start Processing and go to examples/Libraries/Network to open the SharedCanvasServer and SharedCanvasClient examples. (We won't explain the whole code because you are going to do that with your own project later, but we will use these sketches to illustrate the concepts.)

If you run the SharedCanvasServer sketch, you get a frame in which you can draw lines with your mouse. Not very impressive. If you open the SharedCanvasClient, you get the same but Processing throws a “Connection refused” error at you in the console. If you open the server sketch first, and then the client sketch, you have two canvases and no errors. Now, if you paint in one of them, the lines appear on the other one as well, as shown in Figure 6-26. You have two Processing sketches talking to each other; that's a good start!

Figure 6-26. Shared canvas

Communicating Within a Local Network

If you try the same over two computers, though, it won't work. If you look into the setup functions of the server sketch, you find the line in which you are starting the server.

And in the client sketch, you'll find the line in which you start the client.

The client is trying to connect to the computer with IP address 127.0.0.1 through port 12345. Well, this address happens to be the localhost address of the computer. It worked well as long as you had both sketches running on the same computer, but if you want to make them work on different machines, you need to provide the client with the IP address of the server. You can find the internal IP address of your computer easily. On a Mac, open Network Preferences; it's on the top right. You should see something like “AirPort is connected to xxxxxx and has the IP address 192.168.0.101.” On a PC, go to the Start menu. In “Search programs and files” write cmd and press Enter. You have opened a console. Now type ipconfig in the console and press Enter again. Many text lines will appear, one of which says something like “IPv4 Addres..:192.168.0.103”. This is your IP address.

If you go now to your client sketch and substitute the 127.0.0.1 with the IP address of the machine in which you are running the server (the other computer) and run both sketches (remember to run the server first), it should work smoothly.

Now you have two Processing sketches running on different computers talking to each other. But the ultimate goal is to be able to communicate over the Internet. Can this code make that happen? Can I go to my friend's house with the client sketch, run it, and be connected to my machine at home? Well, unfortunately not yet. The IP addresses you've been using are internal IP addresses. They are good as long as both computers are connected to the same local network. The local IP address is known only by the router, so if you go beyond the reach of your local network, these addresses won't make sense any more. What you need is the external IP address of your router.

Communicating over the Internet

The easiest way to find your external IP address is using a web site. If you go to www.whatsmyip.com or similar, you will find a line saying “Your IP Address Is: 24.28.51.56” or a similar number. This is you external IP address, the one you need to include in your Client sketch if you want to connect to your computer from a computer outside of your local network.

But you need to do just one more thing on the server side. The client is trying to connect to your computer using the external IP; now you need to tell the router that any incoming connection addressed to the port 12345 needs to be send to your machine. This is called port forwarding. If you are an online gamer, you probably know about port forwarding because you need to set up socket connections when you want to play online games. What you are trying to establish here is a synchronous connection, or socket, which is a bi-directional open communication channel between your machine and another machine over the Internet.

Port forwarding depends on the router you are using at home, but in general, if you type 192.168.0.1 in your browser, it will lead you to your router configuration menu. Provided that you have the user name and password (by default, it is often “admin”), you will find the port forwarding setting somewhere in the menu. You need to specify the IP address of the computer you are forwarding to (your server computer), and the port that the communication is sent to (12345 in our example), both in TCP and UDP. Dig your router's instructions manual from the dark place you hid it five years ago (or find the manual on the Internet, in the likely case you have no remote clue where your hard copy could be) if you need some help.

You should now be in a position where you can run your server sketch on your computer (where you forwarded the port) and have one of your geeky friends run the client sketch with your external address at his home computer. The two sketches should be communicating smoothly across neighborhoods, countries, and even continents!

Server Applet: Sending the Angles over a Network

So now that you know how to communicate across networks, you are going to add this functionality to your angle calculation sketch so you can send the angles over the network for another program to retrieve them from a remote location.

All your helper functions and callbacks stay the same, but you add some lines to the setup() and draw() functions. You first import the Network library and declare a Server object. Then you initialize all the PVectors and the angles array as per your previous code.

In the setup() function, you initialize the server object, passing the port you are communicating through as a parameter.

And finally, at the end of the draw() loop and only in case you are tracking a skeleton, you write a long string of characters to your port. The string is composed of all the joint angles separated by white spaces, so you can split the string at the other end of the line.

Client Applet: Controlling the Puppet

The server applet is finished and ready to start sending data over the Internet. Now you need a client sketch to receive the angle data and send it to Arduino to get translated into servo positions that will become movements of your puppet.

You're going to implement a virtual puppet resembling the physical puppet (Figure 6-27), so check that everything is working properly before sending data to the servos. You will implement the serial communication at the same time, so once everything is working you only need to check that the serial boolean is set to true for the data to be sent to Arduino.

Figure 6-27. Client applet

After importing the Net and Serial libraries, create a boolean variable called serial that you set to false if you haven't plugged the serial cable yet and want to try the sketch. You are going to be displaying text, so you also need to create and initialize a font.

You need to declare a client object called c, declare a String to contain the message coming from the server applet, and a data[] array to store the angle values once you have extracted them from the incoming string. You are going to use a PShape to draw a version of your physical puppet on screen so you can test all the movements virtually first.

In the setup() function, initialize the client object. Remember to replace the IP address with the external IP of the server computer if you are communicating over the Internet or the internal IP if you are communicating within the same local network. You will need to add the file Android.svg (provided with the chapter images) to your sketch folder so it can be loaded into the PShapes.

Within the draw() loop, check for incoming data from the server; in a positive case, read is as a string. Then trim off the newline character and split the resulting string into as many floats as there are sub-strings separated by white spaces in your incoming message. Store these floats (the angles of your limbs sent from the server sketch) into the data[] array.

Run a little validation test, making sure that all the values acquired fall within the acceptable range of -PI/2 to PI/2.

Once the data is nicely stored in an array, draw your puppet on screen and add the limbs in the desired position. You first draw the shape on screen in a specific position, which you have figured out by testing. Then use the function drawLimb() to draw the robot's limbs using the angles in the data array as parameters.

Then draw an array of circles showing the incoming angles in an abstract way so you can debug inconsistencies and check that you are receiving reasonable values.

And finally, if your boolean serial is true, use the sendSerialData() function to communicate the angles to Arduino.

The drawLimb() function is designed to draw an arm or leg on screen starting at the position specified by the parameters x and y. The angle0 variable specifies the angle that you have to add to the servo angle to accurately display its movement on screen. angle1 and angle2 determine the servo angles of the first and second joint angles on the limb. Limbsize is used as the length of the limb.

The function sendSerialData() works similarly to the ones used in previous chapters, sending a triggering character to Arduino and then writing a series of integer data values to the serial port. These values are the angles mapped to a range of 0 to 250, expressed as integers.

Final Arduino Code

If you run the server applet and then the client applet, you should be able now to control your virtual puppet by dancing in front of your Kinect, as shown in Figure 6-28. Once you have all of your Processing code ready, it's time to plug in your Arduino and type the last bits of code.

Figure 6-28. Server and client applets communicating over a network

You're going to store the values coming from the client Processing sketch, and remap these values from 0-255 to the range of the servo (500 to 2500). It is important to match the data coming from the Processing sketch with the right pin of Arduino, and map it in the right direction (clockwise or counterclockwise) in order to match the movements. The rest of the functions are those you have been using since the beginning of this chapter.

Next, remap the angles from the incoming range (0-255) to the servo range (500-2500).

And finally, move the servo to the new position.

Now upload your code into Arduino, call your most remote Kinect-enabled friend, and ask him to run the server code and dance in front of his Kinect. Then run your client applet, sit down, and enjoy the performance!

Summary

In this chapter, you learned how to use NITE's skeleton tracking capabilities from Processing and how to use the data acquired to control a physical puppet. You were introduced to servos and how to control them from Arduino; you also learned how to make use of network communication, both within a local network and beyond, spanning across the Internet.

The outcome of the chapter is a system with which you can control remote devices using natural interaction from home. In this example, you used it to control a puppet that literally copies your body gestures, but the concept can be endlessly extended. You could develop it to control more complicated devices, using other types of natural interaction algorithms and adapting it to your particular needs.