A depth image can be used for detecting the presence of an object on any rectangular flat surface, such as a rectangular part of a wall, a table, or a floor. Coupled with a projector or a TV panel, it lets us create an interactive system, sensitive to hands or feet, which move near the surface without touching.

The easiest way to make such a projector-camera interactive system is the following:

Let's demonstrate this technology in the example of the drawing application, which draws colors on the surface with dependence on the distance from the surface to the object. This project turns your surface (the surface with the projector's picture or TV panel) into a drawing surface, responding to your hands at a distance, without touching.

The example is based on the ImageAndDepth-Simple example of ofxOpenNI.

In the testApp.h file, add the following declarations in the testApp class declaration:

int W, H;                     //Screen size
vector<ofPoint> corners;      //Marked corners of the surface
ofShortPixels backgroundDepth;//Background depth image

ofPixels outPixels;         //Pixels array holding current drawing
ofImage outImage;           //Output image built from outPixels

bool calibrating;  //Working mode selector – calibrate or draw
ofPoint shift;     //Point used for drawing shifted color image

Keep the openNIDevice object declaration without changing it.

The calibrating variable is the mode selector. If its value is true, the application works in the calibration mode, where the user marks corners using a mouse. If its value is false, the application works in the drawing mode, turning the projected image into a drawing surface.

The setup() function sets up the depth camera, enables its depth and color images and the alignment between them, and starts the depth camera to capture. It also allocates pixels for user drawing, and finally turns on the full screen mode. The body of the function is the following:

//Depth camera setup
openNIDevice.setup();
openNIDevice.addDepthGenerator();
openNIDevice.addImageGenerator();
openNIDevice.setRegister(true);  //Enable alignment
                                 //of depth and color images
openNIDevice.start();            //Start depth camera to capture

//Set up drawing variables
W = 1024;                        //Desired screen size
H = 768;
outPixels.allocate( W, H, OF_IMAGE_GRAYSCALE );
calibrating = true;              //Set calibrating mode at start
shift = ofPoint( 100, 200 );     //The value of shifting
                                 //camera's image from the corner
                                 //of the screen

//Set full screen mode
ofSetWindowShape( W, H );
ofSetFullscreen( true );

The update() function updates the depth camera for obtaining its new images, and then analyzes the depth data, only if we are in the drawing mode and the user has specified all the four calibrating corners. The result of the analysis is written to the outPixels values. Finally, it loads the outPixels data into the outImage image for drawing it on the screen. The body of the function is the following:

openNIDevice.update();	  //Update depth camera
if ( !calibrating && corners.size() == 4 ) {
  //Analyze depth

  //Get current depth image
  ofShortPixels &input = openNIDevice.getDepthRawPixels();

  //Process pixels
  int w = input.getWidth();
  int h = input.getHeight();
  int minV = 30;    //Minimal distance in mm
  int maxV = 150;    //Maximal distance in mm
  for (int Y=0; Y<H; Y++) {
    for (int X=0; X<W; X++) {
      //Process pixel (X, Y)
      //See code below
    }
  }
  outImage.setFromPixels( outPixels );
}

The two parameters, minV and maxV, set the range in millimeters around the surface. We will analyze the objects lying on the distance in this range from the surface.

For compactness, we omit the computing part in the update() function, after the //Process pixel (X, Y) line. Now let's see the remaining code and discuss it. For processing the pixel (X, Y) in screen coordinates, it computes the uniform values a and b lying in [0, 1], and then computes the pixel (x, y) in depth image coordinates using bilinear interpolation of corners with weights (a, b). It is assumed that the corners are ordered clock-wise starting from the top-left corner:

The code for this (X, Y) → (x, y) transformation is as follows:

//Transform screen coordinates (X, Y) 
//to depth image coordinates (x, y)
float a = float(X) / W;
float b = float(Y) / H;
ofPoint p =
  (1-a) * (1-b) * corners[0]
  + a * (1-b) * corners[1]
  + a * b * corners[2]
  + (1-a) * b * corners[3];

int x = int( p.x );
int y = int( p.y );

Having (x, y) coordinates, we check if it actually lies in the depth image, and then get depth values inputZ and backgroundZ from the current depth image and the background depth image correspondingly. Though the original depth values are stored as unsigned short, we use the int type because we need to subtract the values:

if ( x >= 0 && x < w && y >= 0 && y < h ) {
  //Getting depth values
  int inputZ = input.getPixels()[ x + w * y ];
  int backgroundZ = backgroundDepth.getPixels()[ x + w * y ];

Now we compute the value of delta, which is the difference between backgroundZ and inputZ. Also, we check if any of these values is zero: it means that the depth camera does not measure the distance in this pixel, so we should not compute the difference:

  int delta;
  //Check for zero values - it means that depth camera
  //does not compute distance in the pixel
  if ( inputZ != 0 && backgroundZ != 0 ) {
    delta = backgroundZ - inputZ;
  }
  else {
    delta = -1;
  }

The computed value of delta is a distance between the object and the surface in millimeters. Now we check if it lies in the range between minV and maxV, and update outPixels correspondingly.

  //Output value
  if ( ofInRange( delta, minV, maxV ) ) {
    int value = ofMap( delta, minV, maxV, 0, 255, true );
    outPixels.getPixels()[ X + W * Y ] = value;
  }
}

We finished the code of the update() function. Let's consider the draw() function. In the calibrating mode, it draws a white screen with color and depth images, and also draws marked corners on the color image. In the drawing mode, it just draws the current drawing outImage. The body of the function is the following:

ofBackground( 255, 255, 255 );  //Set white background

if ( calibrating ) {
  
  //Draw color and depth image
  ofSetColor( 255, 255, 255 );
  int w = openNIDevice.getWidth();
  int h = openNIDevice.getHeight();
  openNIDevice.drawImage( shift.x, shift.y );
  openNIDevice.drawDepth( shift.x+w+20, shift.y, w/2, h/2 );

  //Draw corners
  ofSetColor( 255, 0, 0 );
  ofFill();
  int n = corners.size();
  for (int i=0; i<n; i++) {
    ofCircle( corners[i] + shift, 10 );
  }
  if ( n == 4 ) {
    for (int i=0; i<n; i++) {
      ofLine( corners[i] + shift,
              corners[(i+1)%n] + shift );
    }
  }
}
else {
  //Show current drawing
  ofSetColor( 255, 255, 255 );
  outImage.draw( 0, 0 );
}

Switching between the calibrating and drawing modes will be carried out by the Space key. Also, while switching to the drawing mode, we will store the current depth image as the background depth image, backgroundDepth. This is implemented in the keyPressed() function by the following code:

void testApp::keyPressed(int key){
  if ( key == ' ' ) {
    calibrating = !calibrating;
    if ( !calibrating ) {	//store background
      backgroundDepth = 
        openNIDevice.getDepthRawPixels();
    }
  }
}

Finally, we add to the body of the mousePressed() function the code for creating the corners when the mouse is clicked:

void testApp::mousePressed(int x, int y, int button){
  if ( calibrating && corners.size() < 4 ) {
    corners.push_back( ofPoint( x, y ) - shift );
  }
}

Note that we store the corners not as original (x, y) mouse coordinates, but shifted by the value of -shift, because the color image is shifted while rendering correspondingly.

The project is ready.

Now let's play with it.

Running the project

Working with the project comprises of the following steps:

1. Enable the projector or TV, and send to it the content of your screen.
2. Run the project. You will see the color and depth images on the white screen. Position the camera so that it can see the whole surface image. Then look at the depth image. It is a drawing smaller than the color image and is only used to ensure that the depth camera sees the surface properly. If the depth image is filled with some solid color—all is ok. But if the depth image has many black pixels, it means that the camera is too close or too far from the surface or the surface material is too dark, transparent, or reflecting. In this case, try to move the camera until you find a better position.
3. Use your mouse to mark four corners on the surface of the color image. It is assumed that the corners are ordered clock-wise starting from the top-left corner, as shown in the previous image. An example of the photo of such a surface with selected corners is presented as follows:
   
4. Note that this is the photo. The content of the screen is inside the white rectangle, where you can see the marked corners.
5. Go away from the surface and press Space. Then the application stores the current depth image as the background depth image and switches to the drawing mode.
6. Now it's time for you to enter. Go to the surface, and move the hands near it at a distance ranging from 30 mm to 150 mm (the distance range corresponds to the values of minV and maxV). You will see how your hands draw colors on the surface, resulting in a black and white abstract drawing.

Move the hands slowly, and you will obtain the picture with smoothing colors as shown in the following image:

Now move the hands faster and you will see a more stepping picture:

There are some additional notes on using this application:

• You can walk near the surface without disturbing it, because the application changes the drawing only when an object is at a distance between 30 mm and 150 mm from the surface.
• We use a low distance value equal to 30 mm (the parameter minV in code) instead of 0 mm, because the depth camera does not measure distances very accurately. So if you use smaller values for minV, it can give more noise in the resulting picture.
• Sometimes, due to small movements of the depth camera, the background image becomes inaccurate. In this case, just press Space twice. The application will switch to the calibration mode and then back to the drawing mode, and store a new background depth image.
• You may note that when you move your hand away from the surface, the tracked position of the hand does not coincide with the real hand properly. The reason is that our simple model does not take into account the relative geometrical positions of the projector and the camera. So our model works properly just near a flat surface.

Tip

For creating a more advanced model that will detect the object's position accurately, you should not use the depth image itself, but a 3D point cloud. See the Additional topics section.

You can use this example as a sketch for creating interactive tables and floor games. The starting point for this is replacing the following line in the update() function:

if ( ofInRange( delta, minV, maxV ) ) {

with just {. In this case, the application will show not the drawing, but the current distribution of distances over the surface. This distribution is held in pixel values of outPixels. You can use it directly, create mask of moving objects using thresholding algorithm, and so on.

We have finished the main example of the chapter. Now we will discuss the topics suggested for further study.