In this chapter we have considered the images mainly as building blocks of a visual scene. In this last section, we will see how images can be used in another way, as the source of data for internal calculations, not displaying on the screen directly. The main examples of such usage are masks and palettes, which we'll discuss now.
Color cameras and depth cameras give color and depth images that represent the scene they capture (for example, humans in front of the camera). Such images can be processed using pixel-by-pixel methods or with the computer vision library, OpenCV. The result is often a binary image, which is called mask , that contains black pixels denoting the background and white pixels denoting human silhouettes. The mask can be applied for controlling physics and changing any parameter of your interactive installation. The user never sees the mask image itself, but only perceives its effect on the interactive scene's behavior. See the example of the human silhouette mask obtained using the depth camera, Microsoft Kinect:
For details on getting a color image from a camera and its processing, see Chapter 5, Working with Videos, and Chapter 9, Computer Vision with OpenCV. For details on obtaining and processing data from depth cameras, see Chapter 10, Using Depth Cameras.
An image can be used as a palette, which sets the color of the brush or particles in your project. This means that you select some point p inside the image and slowly move it by some fixed rule of Perlin noise. After each move, you just read the pixel color from the image at the current position of p and use it for drawing with a brush or for a particle color. We often use such techniques and create many palettes for different objects and also for obtaining controllable coloring for our projects, which ensures predictable behavior.
An example of a palette used in our interactive installation and a dance performance Abstract Wall (made by Kuflex, produced by Ksenia Lyashenko, and shown at the Microsoft event at the Garage Center for Contemporary Culture, Moscow, 2013) are shown in the following screenshot:
The installation reacts to movements of the human body and draws flying particles with colors selected from the palette.
The method for changing colors of particles in the installation is the following: when a particle is created, its position in the palette is set randomly at the bottom of the palette image. During a particle's life, its position in the palette goes up, so the particle's color slowly changes.
Tip
Another frequently used method for changing the particle's position in the palette is using Perlin noise. See details on Perlin noise in Appendix B, Using Perlin Noise.