In this chapter, we worked only with objects in screen coordinates, by just using depth values of depth images. But in many applications, you need to use the 3D point cloud, which represents the depth image points in 3D space—in the world coordinate system centered at the camera. For obtaining a 3D point cloud from the depth image, you should convert all its pixels to 3D points using the openNIDevice.projectiveToWorld( p ) function. Namely, each pixel (x, y) should be represented as point p, where p.x = x, p.y = y, and p.z is equal to the value of the depth image's pixel (x, y): p.z = openNIDevice.getDepthRawPixels()[ x + w * y ].
The 3D point cloud can be used for:
- Accurately modeling the distance distribution over the flat surface. See the discussion at the end of the Creating interactive surface section.
- Making dynamic video mapping, which maps a projector image exactly on a moving object such as a performer. To achieve this, you should use geometrical information of relative positions of the projector and depth camera. For details see the ofxCamaraLucida addon available at http://ofxaddons.com.
- Stitching data from several depth cameras by combining their point clouds for extending the area of interaction.
Other topics to learn are user tracking, hand tracking, and gesture recognition. We suggest that you do this not just by exploring the ofxOpenNI source codes, but also by studying the original OpenNI library.