Chapter 7

Texturing

Texture mapping is by far the most used technique to make 3D scenes look real. In very simple terms, it consists of applying a 2D image to the 3D geometry as you would do with a sticker on a car. In this chapter we will show how this operation is done in the virtual world where the sticker is a raster image and the car is a polygon mesh.

7.1 Introduction: Do We Need Texture Mapping?

Suppose we want to model a checkerboard. With texture mapping we can make a box with six quads and then stick an image with the black and white pattern on the top (Figure 7.1, Right). However, we could obtain the same result even without texture mapping: we could make a mesh with 5 polygons for the base and the sides, and then 64 black and white quads to model each single tile (Figure 7.1, Left). Therefore, in principle, we could say we do not need texture mapping. However, note that in the second case we had to adapt the geometry of our model to comply with the color we wanted to put on it, by replacing the single top quadrilateral of the box with 8 × 8 = 64 quadrilaterals that represent the very same surface, for the sole goal of showing the color pattern. While at this scale it may seem an acceptable computational cost to avoid studying a new subject, consider what this process would be, even for a small image, say 1024 × 1024: you should modify geometry to introduce 1,048, 576 new quadrilaterals, one for each pixel of the image.

Although the efficiency of the representation alone would be enough, there are many other things that we can do with texture mapping. Back to the sticker on the car example: by using texturing we can move the sticker around, change its size, place a non-color information such as the normal of the surface, and so on.

7.2 Basic Concepts

A texture is a raster image, and its pixels are called texels (texture elements). A position in a texture is conventionally referred to as texture coordinates, or UV-coordinates, and is expressed in a reference system originating at the Bottom-Left of the image with axis (sx, 0), (0, sy), so that the texture corresponds to the rectangular space between (0, 0) and (1, 1), called texture space (see Figure 7.2(a)). A texture may be “stuck” onto a 3D surface providing a function M that maps the 2D texture space to the 3D surface. The way to specify and compute M depends very much on how the surface is represented (a polygon mesh, an implicit surface, an algebraic surface, etc.). In the case of a polygon mesh M is defined by assigning to each vertex of the model a pair of texture coordinates. Inside the each polygon, M is found by interpolating the values assigned to the polygon vertices. Defining a correspondence between texture space and a generic polygon mesh is commonly referred to as mesh parameterization and it is a wide topic on its own, on which we will provide some short notes in Section 7.9. In the scope of this book, either we will have simple cases where the parametrization is obvious, or we will use polygon meshes that already specify a parametrization. If you are particularly interested in polygon mesh parametrization, please refer to Floater et al. [12].

What if the value of texture coordinates is outside texture space, for example (−0.5, 1.2)? It is useful to define a complete mapping from [−∞, −∞] × [+∞ +∞], to [0, 1]2 in order to avoid dealing with exceptions and, as we will see shortly, to achieve some special behaviors. This mapping is usually referred to as texture wrapping and it is commonly defined in two alternative modes: clamp and repeat.

$\begin{array}{lll}\text{clamp}\left(\mathit{\text{x}}\right)\hfill & =\hfill & \text{min}\left(\text{max}\left(0.0,\mathit{\text{x}}\right),1.0\right)\hfill \\ \text{repeat}\left(\mathit{\text{x}}\right)\hfill & =\hfill & \mathit{\text{x}}-\lfloor \mathit{\text{x}}\rfloor \hfill \end{array}$

where x is the texture coordinate. Figure 7.2.(a) shows the values of texture coordinates outside the region [0, 1]2 and the images in Figure 7.2.(b,c,d) show the effect of applying the repeat mode for both coordinates, the clamp mode for both coordinates or the repeat mode for u and the clamp mode for v.

7.2.1 Texturing in the Pipeline

Texture mapping functionalities are done in two steps:

1. Texture coordinates are assigned to each vertex and then interpolated to obtain a per-fragment value. This involves the GT&AS stage and the Rasterization stage and it will be explained in detail in Section 7.4.
2. Each fragment is colored with the color contained in the texture at the location addressed by the fragment’s texture coordinates.

Figure 7.3 shows where texturing related operations take place in the rendering pipeline.

7.3 Texture Filtering: from per-Fragment Texture Coordinates to per-Fragment Color

Let us say we have a fragment and its texture coordinates, which means its corresponding position in texture space: how do we pick the right color? If we were in a continuum, with pixels and texels infinitely small, we should just pick up the color corresponding to the texture coordinates. Unfortunately, resources are limited and we deal with discrete domains, so we do not have a one-to-one correspondence between pixels and texels. The projection of a pixel in texture space is a quadrilateral that can be smaller than a texel, in which case we have texture magnification, or it can span over multiple texels, in which case we have texture minification (see Figure 7.4). In both cases we need a way to decide which is the color to assign to the fragment.

7.3.1 Magnification

The straightforward solution for magnification is to see in which texel the texture coordinates fall and to pick up its color. This is called nearest neighbor interpolation. Obviously the more the texture is magnified (the smaller the projection of the pixels is), the more the texels of the texture are visible.

To obtain a smoother result we can linearly interpolate the color of the four closest neighbors as shown in Figure 7.5. Texture coordinates (u′,v″) ∈ [0,1]2 are expressed in a local reference frame with origin in the center of the lower left texel. The formula in the left part computes the color c by interpolating the color of the four texels c00, c10, c11 and c01. The weight assigned to each texel is given by its barycentric coordinates. Since the interpolation is linear both along u and along v, it is called bilinear interpolation. The drawing in the right part of Figure 7.5 also shows another geometrical interpretation of the formula: if you center an imaginary texel to the texture coordinates, the weight of each texel corresponds to its intersection with the imaginary texel. Note that, althought the image produced by nearest neighbor interpolation is correct, bilinear interpolation gives in general more realistic results.

7.3.2 Minification with Mipmapping

With minification, the projection of a pixel in texture space spans over several texels. It is obvious that in this case nearest neighbor interpolation would cause adjacent fragments to pick up the color of arbitrarily far texels, so that we would have a meaningless sampling of the texture. Similarly, bilinear interpolation is pointless because we would have sparse sampling of a group of four texels.

What we would like to do is to assign to the pixel the combination of the texels covered by its projection. We could do it in principle but it would require many accesses to the texture, more precisely to all the texels that project on the view plane. A solution producing almost the same result in a much more efficient way is mipmapping, where mip stays for the latin multum in parvo, many in one.

Consider the ideal case depicted in Figure 7.6, where a quadrilateral with a 2 × 2 texture projects exactly on a pixel. In this case we could access the 4 texels to compute their average value and assign it to the pixel, or we can precompute an alternative version of the texture image made of only one pixel of which color is the average value of the four texels of the original texture and access it. The extension to the general case of an n × n texture and an arbitrary projection is quite straightforward. When we create the texture, we also create alternative versions of the same texture by iteratively halving its width and height. So if the original texture is, say, 1024 × 1024, we build a version of the same image with 512 × 512, 256 × 256, 128 × 128, 64 × 64, 32 × 32, 16 × 16, 8 × 8, 4 × 4, 2 × 2 and 1 × 1 texels, where each texel is obtained by averaging the values of the four corresponding texels at the lower level. Each of these images is a mipmap and the set of mipmaps is referred to as a mipmap pyramid (see Figure 7.7). A specific mipmap is indicated by its mipmap level, which is 0 for the original texture, 1 for the first mipmap and so on. Note that to build the whole hierarchy we need the original texture size to be power of two.

Choosing the Proper Mipmap Level

In the ideal example (in Figure 7.6) we can “see” that the correct mipmap level is the highest one because the projection of the pixel on the texture exactly matches with the texture. In the general the choice of mipmap level for each fragment is done by calculating how big is the projection of the pixel in texture space. If the pixel is as big as 4 texels the mipmap at level 1 will be used, if it is as big as 16 pixels the level 2 and so on. More formally setting ρ = texels/pixel the number of texels covered by a pixel the mipmap level is found as L = log2ρ.

To compute ρ we could project the four corners of the pixel in texture space and then compute the area of the relative parallelogram, but that would be too computationally demanding. Instead, the implementations of graphics API perform an approximate computation of ρ. Figure 7.8 shows a pixel with coordinates (x, y)T and its projection in texture space (u, v). For the sake of simplicity, let us say the (u, v) is expressed in textels ([0, 0] × [sx, sy]) and not in canonical texture coordinates ([0,0] × [1,1]). If we compute:

$\begin{array}{lll}\left(\begin{array}{c}\frac{\partial \mathit{\text{u}}}{\partial \mathit{\text{x}}}\\ \frac{\partial \mathit{\text{v}}}{\partial \mathit{\text{x}}}\end{array}\right)\hfill & =\hfill & \left(\begin{array}{c}\frac{\mathit{\text{u}}\left(\mathit{\text{x}}+\mathrm{\text{Δ}}\mathit{\text{x}},\mathit{\text{y}}\right)-\mathit{\text{u}}\left(\mathit{\text{x}},\mathit{\text{y}}\right)}{\mathrm{\text{Δ}}\mathit{\text{x}}}\\ \frac{\mathit{\text{v}}\left(\mathit{\text{x}}+\mathrm{\text{Δ}}\mathit{\text{x}},\mathit{\text{y}}\right)-\mathit{\text{v}}\left(\mathit{\text{x}},\mathit{\text{y}}\right)}{\mathrm{\text{Δ}}\mathit{\text{x}}}\end{array}\right)\hfill \end{array}$

we know how much the texture coordinates change if we move along x in screen space. For Δx = 1 the length of this vector is a measure of the distance between two adjacent pixels (which is the same as the size of a pixel) in texture space. Of course we can do the same along y and take the maximum of the two lengths:

$\mathit{\text{ρ}}=\text{max}\left(||\left[\begin{array}{c}\frac{\partial \mathit{\text{u}}}{\partial \mathit{\text{x}}}\\ \frac{\partial \mathit{\text{v}}}{\partial \mathit{\text{x}}}\end{array}\right]||,||\left[\begin{array}{c}\frac{\partial \mathit{\text{u}}}{\partial \mathit{\text{y}}}\\ \frac{\partial \mathit{\text{v}}}{\partial \mathit{\text{y}}}\end{array}\right]||\right)$

If, for example, ρ = 16, it means that neither the x nor the y side of the pixel spans more than 16 texels, and therefore the level to use is L = log216 = 4. If ρ = 1 there is one-to-one correspondence between pixels and texels and the level to use is L = log21 = 0. Note that ρ < 1 means that we have magnification. For example ρ = 0.5 means a texel spans over two pixels and the mipmap level would be L = log20.5 = −1, which means the original texture with twice the resolution in both sides.

In the general case ρ is not a power of two, therefore log2ρ will not be an integer. We can choose to use the nearest level or to interpolate between the two nearest levels (⌊log2ρ⌋, ⌈log2ρ⌉). See Figure 7.9 for an example of mipmap at work.

7.4 Perspective Correct Interpolation: From per-Vertex to per-Fragment Texture Coordinates

In the previous chapter we have seen how attribute values of the vertices can be linearly interpolated to assign color or normal to the fragment. Considering the UV-coordinates as yet another vertex attribute the problem seems easily solved. Unfortunately it is slightly more complicated: if we linearly interpolate texture coordinates on the checkerboard example and use a perpective projection (see Section 4.6.2.2) we will get a result like that shown in Figure 7.10. The reason is that perspective projection does not preserve ratios between distances (see Chapter 4.6.2.2), which means that, for example, the middle point of the perspective projection of a segment is not the projection of the middle point of the segment. You may wonder if this means that the linear interpolation for color and normal we used for Gourad shading and Phong shading also produces incorrect results with perspective projection. The answer is yes it does, but the resulting artifact is generally not noticeable. We will make a simple example in 2D to find out the correct texture coordinate we should look up for a given fragment. Consider the situation shown in Figure 7.11. The segment $\overline{{\mathit{\text{a}}}^{\prime }{\mathit{\text{b}}}^{\prime }}$ is the projection on the view plane of segment $\overline{\mathit{\text{a}}\mathit{\text{b}}}$ and the point ${\mathit{\text{p}}}^{\prime }={\mathit{\text{α}}}^{\prime }{\mathit{\text{a}}}^{\prime }+{\mathit{\text{β}}}^{\prime }{\mathit{\text{b}}}^{\prime },{\mathit{\text{α}}}^{\prime }+{\mathit{\text{β}}}^{\prime }=1$ is the projection of the point $\mathit{\text{p}}=\mathit{\text{α}}\mathit{\text{a}}+\mathit{\text{β}}\mathit{\text{b}},\mathit{\text{α}}+\mathit{\text{β}}=1$. The vertices (a, b) of the segment are assigned texture coordinates, respectively, 0 and 1. Our problem is, given the interpolation coordinates of point p′, to find out its correct texture coordinates, which are just the interpolation coordinates of point p with respect to the segment extremes.

Using homogeneous coordinates and substituting $\frac{\mathit{\text{a}}}{{\mathit{\text{a}}}_{\mathit{\text{z}}}/\mathit{\text{d}}}\to {\mathit{\text{a}}}^{\prime }$ and $\frac{\mathit{\text{b}}}{{\mathit{\text{b}}}_{\mathit{\text{z}}}/\mathit{\text{d}}}\to {\mathit{\text{b}}}^{\prime }$ we have:

$\begin{array}{l}{\mathit{\text{p}}}^{\prime }=\left(\begin{array}{c}{\mathit{\text{α}}}^{\prime }{\mathit{\text{a}}}^{\prime }+{\mathit{\text{β}}}^{\prime }{\mathit{\text{b}}}^{\prime }\\ 1\end{array}\right)\hfill \end{array}=\left(\begin{array}{c}{\mathit{\text{α}}}^{\prime }\frac{\mathit{\text{a}}}{{\mathit{\text{a}}}_{\mathit{\text{z}}}/\mathit{\text{d}}}+{\mathit{\text{β}}}^{\prime }\frac{\mathit{\text{b}}}{{\mathit{\text{b}}}_{\mathit{\text{z}}}/\mathit{\text{d}}}\\ 1\end{array}\right)=\left(\begin{array}{c}\frac{{\mathit{\text{α}}}^{\prime }}{{\mathit{\text{w}}}_{\mathit{\text{a}}}}\mathit{\text{a}}+\frac{{\mathit{\text{β}}}^{\prime }}{{\mathit{\text{w}}}_{\mathit{\text{b}}}}\mathit{\text{b}}\\ 1\end{array}\right)$

where we applied ${\mathit{\text{a}}}_{\mathit{\text{z}}}/\mathit{\text{d}}\to {\mathit{\text{w}}}_{\mathit{\text{a}}}$ and ${\mathit{\text{b}}}_{\mathit{\text{z}}}/\mathit{\text{d}}\to {\mathit{\text{w}}}_{\mathit{\text{b}}}$. We recall that $\left(\begin{array}{c}\mathit{\text{a}}\\ 1\end{array}\right)=\left(\begin{array}{c}\mathit{\text{a}}\mathit{\text{w}}\\ \mathit{\text{w}}\end{array}\right),\forall \mathit{\text{w}}\ne 0$. Multiplying point p′ for any non-zero value, we will always have points along the line passing through the point of view and p′, meaning all the points that project to the same point p′. We choose to multiply p′ by $\frac{1}{\frac{{\mathit{\text{α}}}^{\prime }}{{\mathit{\text{w}}}_{\mathit{\text{a}}}}+\frac{{\mathit{\text{β}}}^{\prime }}{{\mathit{\text{w}}}_{\mathit{\text{b}}}}}$:

${\mathit{\text{p}}}^{\prime }=\left(\begin{array}{c}\frac{{\mathit{\text{α}}}^{\prime }}{{\mathit{\text{W}}}_{\mathit{\text{a}}}}\mathit{\text{a}}+\frac{{\mathit{\text{β}}}^{\prime }}{{\mathit{\text{W}}}_{\mathit{\text{b}}}}\mathit{\text{b}}\\ 1\end{array}\right)\frac{1}{\frac{{\mathit{\text{α}}}^{\prime }}{{\mathit{\text{w}}}_{\mathit{\text{a}}}}+\frac{{\mathit{\text{β}}}^{\prime }}{{\mathit{\text{w}}}_{\mathit{\text{b}}}}}=\left(\begin{array}{c}\frac{\frac{{\mathit{\text{α}}}^{\prime }}{{\mathit{\text{w}}}_{\mathit{\text{a}}}}}{\frac{{\mathit{\text{α}}}^{\prime }}{{\mathit{\text{w}}}_{\mathit{\text{a}}}}+\frac{{\mathit{\text{β}}}^{\prime }}{{\mathit{\text{w}}}_{\mathit{\text{b}}}}}\mathit{\text{a}}+\frac{\frac{{\mathit{\text{β}}}^{\prime }}{{\mathit{\text{w}}}_{\mathit{\text{b}}}}}{\frac{{\mathit{\text{α}}}^{\prime }}{{\mathit{\text{w}}}_{\mathit{\text{a}}}}+\frac{{\mathit{\text{β}}}^{\prime }}{{\mathit{\text{w}}}_{\mathit{\text{b}}}}}\mathit{\text{b}}\\ \frac{1}{\frac{{\mathit{\text{α}}}^{\prime }}{{\mathit{\text{w}}}_{\mathit{\text{a}}}}+\frac{{\mathit{\text{β}}}^{\prime }}{{\mathit{\text{w}}}_{\mathit{\text{b}}}}}\end{array}\right)$

Note that the terms multiplying a and b sum to 1, meaning that the point is on the segment $\overline{\mathit{\text{a}}\mathit{\text{b}}}$. Since it is also, by construction, on the line L it means it is the point p, therefore the two terms are the barycentric coordinates of point p, alias the texture coordinates we were looking for.

${\mathit{\text{t}}}_{{\mathit{\text{p}}}^{\prime }}=\frac{{\mathit{\text{α}}}^{\prime }\frac{{\mathit{\text{t}}}_{\mathit{\text{a}}}}{{\mathit{\text{w}}}_{\mathit{\text{a}}}}+{\mathit{\text{β}}}^{\prime }\frac{{\mathit{\text{t}}}_{\mathit{\text{b}}}}{{\mathit{\text{w}}}_{\mathit{\text{b}}}}}{{\mathit{\text{α}}}^{\prime }\frac{1}{{\mathit{\text{w}}}_{\mathit{\text{a}}}}+{\mathit{\text{β}}}^{\prime }\frac{1}{{\mathit{\text{w}}}_{\mathit{\text{b}}}}}$

Note that another way to write this expression is:

${\mathit{\text{t}}}_{{\mathit{\text{p}}}^{\prime }}={\mathit{\text{α}}}^{\prime }\left[\begin{array}{c}{\mathit{\text{t}}}_{\mathit{\text{a}}}\\ 1\end{array}\right]+{\mathit{\text{β}}}^{\prime }\left[\begin{array}{c}{\mathit{\text{t}}}_{\mathit{\text{b}}}\\ 1\end{array}\right]={\mathit{\text{α}}}^{\prime }\left[\begin{array}{c}{\mathit{\text{t}}}_{\mathit{\text{a}}}/{\mathit{\text{w}}}_{\mathit{\text{a}}}\\ 1/{\mathit{\text{w}}}_{\mathit{\text{a}}}\end{array}\right]+{\mathit{\text{β}}}^{\prime }\left[\begin{array}{c}{\mathit{\text{t}}}_{\mathit{\text{b}}}/{\mathit{\text{w}}}_{\mathit{\text{b}}}\\ 1/{\mathit{\text{w}}}_{\mathit{\text{b}}}\end{array}\right]$

which is called hyperbolic interpolation. The generalization to the interpolation of n values is straigthforward, so we have the rule to compute perspectively correct attributes interpolation for the triangle. Consider the triangle (a, b, c) with texture coordinates ta, tb, tc and a fragment with position p′ = α′a + β′b + γ′c. The texture coordinates for p′ are obtained as:

$\begin{array}{ccc}{\mathit{\text{t}}}_{{\mathit{\text{p}}}^{\prime }}=\frac{{\mathit{\text{α}}}^{\prime }\frac{{\mathit{\text{t}}}_{\mathit{\text{a}}}}{{\mathit{\text{w}}}_{\mathit{\text{a}}}}+{\mathit{\text{β}}}^{\prime }\frac{{\mathit{\text{t}}}_{\mathit{\text{b}}}}{{\mathit{\text{w}}}_{\mathit{\text{b}}}}+{\mathit{\text{γ}}}^{\prime }\frac{{\mathit{\text{t}}}_{\mathit{\text{c}}}}{{\mathit{\text{w}}}_{\mathit{\text{c}}}}}{{\mathit{\text{α}}}^{\prime }\frac{1}{{\mathit{\text{w}}}_{\mathit{\text{a}}}}+{\mathit{\text{β}}}^{\prime }\frac{1}{{\mathit{\text{w}}}_{\mathit{\text{b}}}}+{\mathit{\text{γ}}}^{\prime }\frac{1}{{\mathit{\text{w}}}_{\mathit{\text{c}}}}}& & \left(7.1\right)\end{array}$

7.5 Upgrade Your Client: Add Textures to the Terrain, Street and Building

Assuming our terrain is a big rectangular square we could just find an image of a terrain to use as texture and map it on the polygon representing the terrain. Say our terrain is a square patch of 512m × 512m and we make a texture map from a 512 × 512 image. The result would be that a single texel of the texture covers one square meter of terrain. It will hardly look like a grass field and be assured that the texels will be clearly visible from the inside-the-car view. If we want more detail, say that one texel covers only a square centimeter, our image should be 512,000 × 512, 000 pixels, which would make 786 GB of video memory just for the terrain (considering simple RGB, 8 bits per channel), against the few GBs available on current GPUs.

The real solution to this problem is texture tiling, which consists of putting side by side multiple copies of the same image to fill a larger space. If the result does not show noticeable discontinuities, the image used is said to be tileable, like the one in Figure 7.12 (Left). To actually place copies of the texture side by side we could change the geometry and transform the single large quadrilateral corresponding to the whole plane to 1000 × 1000 quadrilaterals, or, more efficiently, to use texture wrapping repeat mode for both coordinates and assign texture coordinates (0, 0), (512, 0), (512, 512), (0, 512).

We start by adding a function that creates a texture, shown in Listing 7.1.

12 NVMCClient.createTexture = function (gl, data) {

13  var texture = gl.createTexture();

14  texture.image = new Image();

15 

16  var that = texture;

17  texture.image.onload = function () {

18 gl.bindTexture(gl.TEXTURE_2D, that);

19 gl.pixelStorei(gl.UNPACK_FLIP_Y_WEBGL , true);

20 gl.texImage2D(gl.TEXTURE_2D, 0, gl.RGBA, gl.RGBA, gl.UNSIGNED_BYTE, that.image);

21 gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MAG_FILTER, gl.LINEAR);

22 gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MIN_FILTER, gl.LINEAR_MIPMAP_LINEAR);

23 gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_S, gl.REPEAT);

24 gl.texParameteri(gl.TEXTURE_2D , gl.TEXTURE_WRAP_T, gl.REPEAT);

25 gl.generateMipmap(gl.TEXTURE_2D);

26 gl.bindTexture(gl.TEXTURE_2D, null);

27 };

28 

29  texture.image.src = data;

30  return texture;

31}

LISTING 7.1: Create a texture. (Code snippet from http://envymycarbook.com/chapter7/0/0.js.)

At line 13 we use the WebGL command createTexture() to create a WebGl texture object. Then we add to this object a member Image, that is, a JavaScript image object and assign the path to where the image must be loaded from. Lines 17-27 set how the texture is set up when the image will be loaded.

The first thing is to bound our texture to a texture target. A texture target specifies which will be the target of the next calls. Up to now we have only seen textures as bidimensional images but there are other kinds of textures made available in WebGL. For example, gl.TEXTURE_1D for texture made only of one line of pixels, or gl.TEXTURE_3D, which is a stack of bidimensional textures.

With the call at line 18 we say that from now on the gl.TEXTURE_2D is the texture we just created. Then at line 20 we associate the JavaScript image we loaded from disk to our texture. This is the point where the image is created in video memory. The first parameter of gl.texImage2D (and all subsequent calls) is the texture target. The second parameter is the level of the texture: 0 indicates the original image and i = 1 … log(size) the i − th level of mipmapping.

The third parameters tells how the texture must be stored in video memory. In this case we are telling each texture has four channels gl.RGBA. The fourth and fifth parameters tell how the image (the sixth parameter) must be read. In this case gl.RGBA says that the image has four channels and gl.UNSIGNED_BYTE that each channel is encoded with an unsigned byte code (0 … 255).

Line 21 indicates how the texture must be sampled when the texture is magnified (linear interpolation) and line 22 when the texture is minified. In the latter case we passed the parameter gl.LINEAR_MIMAP_LINEAR that says that the texture value must also be interpolated linearly within the same level and also linearly in between mipmapping levels. We invite the reader to change this value on gl.LINEAR_MIMAP_NEAREST and look at the differences. At lines 23-24 we specify the wrapping mode for both coordinates to be gl.REPEAT and at line 26 we make WebGL generate the mipmap pyramid. The call gl.bindTexture(gl.TEXTURE_2D,null) unbinds the texture from the texture target. This is not strictly necessary but consider that if we do not do this any call to modify a texture parameter with gl.TEXTURE_2D will affect our texture. We strongly advise to end each texture parameter setting with the unbinding of the texture.

We will create 4 textures: one for the ground, one for the street, one for the building facade and one for the roofs, by adding the lines in Listing 7.2 in function onInitialize.

398 NVMCClient.texture_street = this.createTexture(gl, 

   ../../../media/textures/street4.png) ;

399 NVMCClient.texture_ground = this.createTexture(gl, 

   ./../../media/textures/grass.png);

400 NVMCClient.texture_facade = this.createTexture(gl, 

   ../../../media/textures/facade.png) ;

LISTING 7.2: Loading images from files and creating corresponding textures. (Code snippet from http://envymycarbook.com/chapter7/0/0.js.)

7.5.1 Accessing Textures from the Shader Program

So far we have seen how to create a texture object. Now we want to use our texture in our shader program. This is done by the texture mapping units or texture units, as they are usually referred. Texture units work in conjunction with the vertex and fragment shader and are in charge to access the textures with the parameters specified in the texture object. Listing 7.3 shows the vertex and fragment shader for texturing. The vertex shader just passes along the texture coordinates just like it does for position, and everything happens in the fragment shader.

2 var vertex_shader = "

3 uniform mat4 uModelViewMatrix;    

4 uniform mat4 uProjectionMatrix;   

5 attribute vec3 aPosition;     

6 attribute vec2 aTextureCoords;    

7 varying vec2 vTextureCoords;     

8 void main(void)      

9 {        

10 vTextureCoords = aTextureCoords;    

11 gl_Position = uProjectionMatrix *    

12 uModelViewMatrix * vec4(aPosition, 1.0);   

13 }";

14 var fragment shader = “

15  precision highp float;     

16  uniform sampler2D uTexture;    

17  uniform vec4 uColor;      

18  varying vec2 vTextureCoords;     

19  void main(void)      

20  {        

21 gl_FragColor = texture2D(uTexture, vTextureCoords);   

22 } “;

LISTING 7.3: Minimal vertex and fragment shaders for texturing. (Code snippet from http://envymycarbook.com/chapter7/0/shaders.js.)

At line 16 we declare the uniform variable uTexture of type sampler2D. Doing that, we have created a reference to texture unit (uTexture) which will sample value of a bidimensional texture. Then, at line 21, we access the value contained in the texture bound to the texture target gl.TEXTURE_2D of texture unit uTexture at coordinates vTextureCoords. Texture units are identified with an index between 0 and MAX_TEXTURE_IMAGE_UNITS (a hardware dependent value that we can find with function gl.getParameter). On the Javascript/client side we need to do two things: bind our texture to a texture unit and pass to the shader the index to such texture unit. Listing 7.4 shows how these operations are done. At line 318 we set the current active texture unit to 0 and at line 319 we bind the texture this.texture_ground to the gl.texture_2D target. This means that texture unit 0 now will sample the texture this.texture_ground. Finally, at line 320, we set the value of uTexture to 0.

318 gl.activeTexture(gl.TEXTURE0);

319 gl.bindTexture(gl.TEXTURE_2D, this.texture_ground);

320 gl.uniform1i(this.textureShader.uTextureLocation, 0);

LISTING 7.4: Setting texture access. (Code snippet from http://envymycarbook.com/chapter7/0/0.js.)

Figure 7.13 shows a client with textures applied to the elements of the scene.

7.6 Upgrade Your Client: Add the Rear Mirror

In terms of texture mapping, adding a rear mirror only means changing the source of the texture from a still image loaded from somewhere (the hard disk, Internet) to one produced at every frame. The tricky part is how to produce the image to put on the mirror.

We have seen in Chapter 6 that the light arriving on a perfectly reflecting surface bounces away in the specular direction, that is in the symmetric direction with respect to the surface normal. This means that referring to Figure 7.14 the image we see in the mirror from the driver’s view frame V is the same as the one seen from view frame V′, which is V′ mirrored with respect to the plane XY containing the rear mirror. Since the mirror is a part of the car it makes sense that its four corners are specified in the frame of the car. Here we assume that these points have been already multiplied by the car’s frame and hence pi : i = 0…3 are in world coordinates. Also we assume frame V is in world coordinates. Since we want to mirror frame V with respect to the mirror plane we build the frame M as any frame whose XY plane contains the mirror. For example, we can take p0 as origin, $\frac{\left({\mathit{\text{p}}}_1-{\mathit{\text{p}}}_0\right)}{||{\mathit{\text{p}}}_1-{\mathit{\text{p}}}_0||}$ and $\frac{\left({\mathit{\text{p}}}_3-{\mathit{\text{p}}}_0\right)}{||{\mathit{\text{p}}}_3-{\mathit{\text{p}}}_0||}$ as x and y axes and x × y as z axis. The expression to find the mirrored frame is:

${\mathit{\text{V}}}^{\prime }=\mathit{\text{M}}{\mathit{\text{Z}}}_{\mathit{\text{m}}}{\mathit{\text{M}}}^{-1}\mathit{\text{V}}$

that is, express frame V in frame M coordinates, then mirror the z component, then again express it in world coordinates.

The next step is perform the rendering of the scene with V′ as viewing frame and use the result as texture to be mapped on the polygon representing the rear mirror. Note that we do not want to map the whole image on the such polygon but only the part of the frustum that passes through the mirror. This is done by assigning the texture coordinates to vertices pi using their projection on the viewing window:

${\mathit{\text{t}}}_{\mathit{\text{i}}}=\mathit{\text{T}}\mathit{\text{P}}{\mathit{\text{V}}}^{\prime }{\mathit{\text{p}}}_{\mathit{\text{i}}}$

where P is the projection matrix and T is the matrix that maps coordinates from NDC [−1, 1]3 to texture space [0, 1]3, which you can find explained in Section 7.7.5. Note that we actually need only the first two components of the result.

7.6.1 Rendering to Texture (RTT)

Rendering to texture is one of the ways we can do off-screen rendering, that is, store the result of the rendering somewhere instead of showing it on the screen.

We introduce a new function to our client, shown in Listing 7.5, that creates an alternative framebuffer. The call at line 41 is the most explicative: gl.bindFrameBuffer(gl.FRAMEBUFFER,null) tells WebGl where to redirect the result of rasterization. The default value (that is null) indicates the on-screen framebuffer but we can create new framebuffer to be used. Let us start from the begining, by showing how to create a framebuffer object. Referring to Listing 7.5, at line 13 we create an object TextureTarget, which is a simple container of a framebuffer and its texture. At line 14 we create the WebGl object framebuffer, then we set it as current output buffer and target of all subsequent operations. At lines 17-20 we set the size in pixels. At line 36 we assign to the framebuffer its color attachment, that is the buffer to which the color will be written. In this case the color attachment is the texture we create with the lines 22-30. This means that the channels and the bit planes of the new framebuffer are defined by those of the texture textureTarget.texture. This texture is used as output buffer when this framebuffer is to be the current one, but it can be normally bound to show the result in the rear mirror. Please note, and this is an important bit of information and often a source of painful bugs, that when a framebuffer is used for rendering, its color attacment cannot be bound as texture target. In other words you cannot simultaneously read and write the same texture.

At line 37 we specify the buffer to use as depth buffer. This is an object we have not seen yet, a renderbuffer, that is, just another type of buffer like the textures but more generic where data contained is not necessarily the output written by gL_FragColor, for example, stencil or depth buffer (like in this case). With the call at line 34 we declare that its physical storage will be a 16-bit depth value. Note that we may create and use a framebuffer even without depth attachment, if, for the specific purpose of rendering, depth test is not necessary. Finally we return the object TextureTarget.

6 NVMCClient.rearMirrorTextureTarget = null;

7 TextureTarget = function () {

8 this.framebuffer = null;

9 this.texture = null;

10};

11

12 NVMCClient.prepareRenderToTextureFrameBuffer = function (gl, generateMipmap, w, h) {

13  var textureTarget = new TextureTarget();

14  textureTarget.framebuffer = gl.createFramebuffer();

15  gl.bindFramebuffer(gl.FRAMEBUFFER, textureTarget.framebuffer);

16

17  if (w) textureTarget.framebuffer.width = w;

18  else textureTarget.framebuffer.width = 512;

19  if (h) textureTarget.framebuffer.height = h;

20  else textureTarget.framebuffer.height = 512;;

21

22  textureTarget.texture = gl.createTexture();

23  gl.bindTexture(gl.TEXTURE_2D, textureTarget.texture);

24  gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE MAG FILTER, gl.LINEAR);

25  gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE MIN FILTER, gl.LINEAR);

26  gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_S, gl.REPEAT);

27  gl.texParameteri(gl.TEXTURE 2D, gl.TEXTURE_WRAP_T, gl.REPEAT);

28

29  gl.texImage2D(gl.TEXTURE_2D, 0, gl.RGBA, textureTarget.framebuffer.width, textureTarget.framebuffer.height, 0, gl.RGBA, gl.UNSIGNED_BYTE, nu ll);

30  if (generateMipmap) gl.generateMipmap(gl.TEXTURE_2D);

31

32  var renderbuffer = gl.createRenderbuffer();

33  gl.bindRenderbuffer(gl.RENDERBUFFER, renderbuffer);

34  gl.renderbufferStorage(gl.RENDERBUFFER, gl.DEPTH_COMPONENT16, textureTarget . framebuffer.width, textureTarget . framebuffer .height);

35

36  gl.framebufferTexture2D(gl.FRAMEBUFFER, gl.COLOR_ATTACHMENT0, gl.TEXTURE_2D, textureTarget.texture, 0);

37  gl.framebufferRenderbuffer(gl.FRAMEBUFFER, gl.DEPTH_ATTACHMENT , gl.RENDERBUFFER, renderbuffer);

38

39  gl.bindTexture(gl.TEXTURE_2D, null);

40  gl.bindRenderbufierCgl.RENDERBUFFER, null);

41  gl.bindFramebuffer(gl.FRAMEBUFFER, null);

42

43  return textureTarget;

44}

LISTING 7.5: Creating a new framebuffer. (Code snippet from http://envymycarbook.com/chapter7/1/1.js.)

Figure 7.15 shows a view from the driver’s prespective with rearview appearing in the rear mirror.

7.7 Texture Coordinates Generation and Environment Mapping

Up to now we learned how to stick images to geometry, but the techniques at hand allow us to do it much more. For example, we could do a scrolling sign on a panel by changing the texture coordinates from frame to frame. This is just the most trivial example: once it is assumed that texture coordinates can be a function of something (of time in the example above) and having complete freedom as to the content of the texture, texturing becomes a powerful weapon to achieve more realistic renderings.

With the term environment mapping we refer to texture mapping used to show the reflection of the surrounding environment on the 3D scene to be represented. This is a very general notion that does not specify if the surrounding environment is something different from the 3D scene or if it is part of it. In some of the most common techniques we will see that the environment is represented by images and encoded into textures.

7.7.1 Sphere Mapping

If we take a picture of a perfectly reflecting sphere with a camera we will have a picture like the one shown in Figure 7.16.(a), containing all the surrounding environment except the part exactly behind the sphere. Suppose that the picture is taken at an infinite distance, so that the projection is orthogonal, as illustrated by Figure 7.16.(b). In this way we have established a one-to-one correspondence between the direction of reflection of view rays (rendered in blue) and the image. The idea of sphere mapping is to use the image as texture and use this correspondence for a generic object, computing the texture coordinate as a function of the reflection ray. Note that parallel reflection rays will always map to the same texture coordinates, no matter how far they are from each other, which is the approximation error introduced by sphere mapping. The smaller the distance between parallel rays with respect to the distance of the reflected environment, the less noticeable is the error. If the environment tends to be infinitely far or, equivalently, the sphere and the object tend to be infinitely small, the approximation error tends to 0.

7.7.1.1 Computation of Texture Coordinates

Figure 7.16(c) illustrates how reflection rays are mapped to texture space in the 2D case. Given a normalized ray R = (y, z), consider the angle α formed by R = (y, z + 1) with the z axis. The texture coordinate t is found as $\mathit{\text{t}}=\frac{\mathit{\text{s}}\mathit{\text{i}}\mathit{\text{n}}\left(\mathit{\text{α}}\right)}{2}+\frac{1}{2}$, where the division and summation simply map [−1, 1] to the conventional texture space [0, 1]. Writing sin(α) in terms of R we obtain the formula you usually find in the manuals:

$\begin{array}{lll}\mathit{\text{s}}\hfill & =\hfill & \frac{{\mathit{\text{R}}}_{\mathit{\text{x}}}}{2\sqrt{{\mathit{\text{R}}}_{\mathit{\text{x}}}^2+{\mathit{\text{R}}}_{\mathit{\text{y}}}^2+{\left({\mathit{\text{R}}}_{\mathit{\text{z}}}+1\right)}^2}}+\frac{1}{2}\hfill \\ \mathit{\text{t}}\hfill & =\hfill & \frac{{\mathit{\text{R}}}_{\mathit{\text{y}}}}{2\sqrt{{\mathit{\text{R}}}_{\mathit{\text{x}}}^2+{\mathit{\text{R}}}_{\mathit{\text{y}}}^2+{\left({\mathit{\text{R}}}_{\mathit{\text{z}}}+1\right)}^2}}+\frac{1}{2}\hfill \end{array}$

7.7.1.2 Limitations

Sphere mapping is view dependent, meaning that, besides the aforementioned approximation, a sphere map is only valid from the point of view from which it was created. Furthermore, the information contained in the texture is not a uniform sampling of the environment, but it is more dense in the center and less in the boundary. In the practical realization, the tessellation of the object’s surface may easily lead to artifacts in the boundary region of the sphere map, because neighbor points in the surface may correspond to faraway texture coordinates, as shown in Figure 7.17.

7.7.2 Cube Mapping

Like sphere mapping, cube mapping works on the assumption that the environment is infinitely far and establishing a correspondence between reflection ray and texture coordinates.

Let us suppose that we place an ideal cube in the middle of the scene and we shot 6 photos from the center of the cube along the three axes and in both directions (as shown in Figure 7.18) so that the viewing window of each photo matches exactly with the corresponding face of the cube. The result is that our cube contains all the environment. Figure 7.18.(b) shows a development of the cube so that each square is a face of the cube.

Computation of Texture Coordinates

Mapping the reflection direction to the texture map, which in this case is referred to as cube map, is fairly intuitive: we simply find the intersection between the extension of the reflection direction starting from the center of the cube. This is done by the following formulas:

$\begin{array}{lll}\mathit{\text{s}}\hfill & =\hfill & \frac{1}{2}\frac{{\mathit{\text{R}}}_1}{{\mathit{\text{R}}}_{\mathit{\text{m}}\mathit{\text{a}}\mathit{\text{x}}}+1}\hfill \\ \mathit{\text{t}}\hfill & =\hfill & \frac{1}{2}\frac{{\mathit{\text{R}}}_2}{{\mathit{\text{R}}}_{\mathit{\text{m}}\mathit{\text{a}}\mathit{\text{x}}}+1}\hfill & & & \left(7.2\right)\end{array}$

where Rmax is the biggest component in absolute value: Rmax = max(|Rx|, |Ry|, |Rz|), and R1 and R2 are the other two components.

Limitations

Cube mapping has several advantages over sphere mapping: a cube map is valid from every view direction, we do not have the artifact due to the singularity along the (0, 0, −1) direction and the mapping from the cube to the cube maps does not introduce distortion. However, like sphere mapping, the method works correctly on the same assumptions, that is: the environment is far, far away.

7.7.3 Upgrade Your Client: Add a Skybox for the Horizon

Now we are going to add a skybox that, as the word suggests, is none other than a huge box containing all our world and on the faces of this box there are images of the environment as it is seen from the scene. Which point of the scene? Any, because the box is so large that only the view direction counts. This is the simplest use of cubemapping and does not even have to do with reflection. The idea is that we just render an infinitely large cube with a cubemap on it. Now, we obviously cannot position the cube vertices to infinite, but consider that if we could, we would have two effects that we can achieve in other ways:

1. The portion of the cube rendered on the screen depends only on the view direction. This can be done by rendering the unitary cube (or a cube of any size) centered in the viewer position.
2. Everything in the scene is closer than the box and therefore every fragment of the rasterization of a polygon of the scene would pass the depth test. This can be done by rendering the cube first and telling WebGL not to write on the depth buffer while rendering the cube, so everything rendered after will pass the depth test.

Creating a cubemap is just like creating a single 2D texture and hence we are not going to comment it line by line. The important thing is that we specify a different target: instead of gl.TEXTURE_2D we have gl.TEXTURE_CUBE_MAP and then we have — gl.TEXTURE_CUBE_MAP _[POSITIVE |NEGATIVE]_[X|Y|Z] to load single 2D textures on the faces.

At line 48 in Listing 7.6 we set the shader to use (that we will see shortly), and then we pass the projection matrix we are using, as usual. Next, we copy the modelview matrix but we put that 0’s in the translation which part (that is, the last column), which means that the point of view is now set to the origin. Then, at line 58 we disable the writing on the depth buffer and render the unitary cube.

47 NVMCClient.drawSkyBox = function (gl) {

48  gl.useProgram(this.skyBoxShader);

49  gl.uniformMatrix4fv(this.skyBoxShader.uProjectionMatrixLocation, false, this.projectionMatrix);

50  var orientationOnly = this.stack.matrix ;

51  SglMat4.col$(orientationOnly,3,[0.0,0.0,0.0,1.0]);

52

53  gl. uniformMatrix4fv(this.skyBoxShader.uModelViewMatrixLocation, false, orientationOnly);

54  gl.uniform1i(this.skyBoxShader.uCubeMapLocation, 0);

55

56  gl.activeTexture(gl.TEXTURE0);

57  gl.bindTexture(gl.TEXTURE_CUBE_MAP, this.cubeMap);

58  gl.depthMask(false);

59  this.drawObject(gl, this.cube, this.skyBoxShader, [0.0, 0.0, 0.0, 1]);

60  gl.depthMask(true);

61  gl.bindTexture(gl.TEXTURE_CUBE_MAP, null);

62}

LISTING 7.6: Rendering a skybox. (Code snippet from http://envymycarbook.com/chapter7/3/3.js.)

The shader, shown in Listing 7.7, is very simple, being just a sampling of the cubemap based on the view direction. The only difference with the bidimensional texture is that we have a dedicated sampler, sampleCube, and a dedicated function, textureCube, which takes a three-dimensional vector, and accesses the cubemap by using Equation (7.2).

2 var vertexShaderSource = "

3 uniform mat4 uModelViewMatrix; 

4 uniform mat4 uProjectionMatrix; 

5 attribute vec3 aPosition;  

6 varying vec3 vpos;   

7 void main(void)    

8 {      

9  vpos = normalize(aPosition); 

10 gl_Position = uProjectionMatrix * uModelViewMatrix * vec4(aPosition , 1.0) ;

11 }";

12 var fragmentShaderSource = "

13  precision highp float;   

14  uniform samplerCube uCubeMap; 

15  varying vec3 vpos;   

16  void main(void)   

17  {     

18 gl_FragColor = textureCube (uCubeMap,normalize(vpos));

19 } ";

LISTING 7.7: Shader for rendering a skybox. (Code snippet from http://envymycarbook.com/chapter7/3/shaders.js.)

7.7.4 Upgrade Your Client: Add Reflections to the Car

At this point adding the reflections on the cars is a simple matter of using a shader similar to the one in Listing 7.7, with the difference that the texture is accessed at the position indicated by the reflection direction.

Since we need the normal and the position they are passed as attributes and interpolated as usual (see Listing 7.8). Then the reflection direction is computed with Formula (6.6) by the native function reflect, at line 35. We pass the matrix uViewToWorldMatrix to the shader, that is, the matrix converting the coordinates from the view reference frame, where the reflection computation is done, to the world reference frame, where the cubemap is expressed.

This is not the only way to do it. We could compute the refraction direction directly in world space and avoid the final transformation in the fragment shader.

5 shaderProgram.vertexShaderSource = "

6 uniform mat4 uModelViewMatrix;   

7 uniform mat4 uProjectionMatrix;  

8 uniform mat3 uViewSpaceNormalMatrix; 

9 attribute vec3 aPosition;    

10  attribute vec4 aDiffuse;   

11  attribute vec4 aSpecular;    

12  attribute vec3 aNormal;   

13  varying vec3 vPos;    

14  varying vec3 vNormal;    

15  varying vec4 vdiffuse;    

16  varying vec4 vspecular;   

17  void main(void)      

18  {       

19  vdiffuse = aDiffuse;     

20 vspecular = aSpecular;   

21 vPos = vec3(uModelViewMatrix * vec4(aPosition, 1.0)); 

22 vNormal =normalize(uViewSpaceNormalMatrix * aNormal);

23 gl_Position = uProjectionMatrix * uModelViewMatrix * vec4(aPosition, 1.0);

24 }” ;

25 shaderProgram fragmentShaderSource = “

26  precision highp float;   

27  uniform mat4 uViewToWorldMatrix; 

28  uniform samplerCube uCubeMap; 

29  varying vec3 vPos;    

30  varying vec4 vdiffuse;   

31  varying vec3 vNormal;    

32  varying vec4 vspecular;   

33  void main(void)     

34  {       

35 vec3 reflected_ray = vec3(uViewToWorldMatrix* vec4(reflect (vPos,vNormal) ,0.0)) ;

36 gl_FragColor = textureCube (uCubeMap,reflected_ray)* vspecular+vdiffuse;

37}";

LISTING 7.8: Shader for reflection mapping. (Code snippet from http://envymycarbook.com/chapter7/4/shaders.js.)

7.7.4.1 Computing the Cubemap on-the-fly for More Accurate Reflections

We recall that the cube mapping works on the assumption that the environment is infinitely far away. The way we implemented reflection mapping, the car surface will always reflect the environment, no matter in which part of the scene the car is, just like if the rest of the scene (the track, the buildings, the trees etc..) were not present. We can have a more convincing reflection computing the cubemap on-the-fly, that is, instead of using the static reflection map used for the skybox, we create a new cube map for each frame, taking 6 views from, say, the center of the car, filling the faces of the cubemaps with the results. We already know how to render to a texture since we have seen it in Section 7.6. The only difference is that in this case the texture will be a face of the cubemap.

The only sensible change to the code is the introduction of function drawOnReflectionMap. The first lines of the function are shown in Listing 7.9. First of all, at line 62, we set the projection matrix with a 90 angle and aspect ratio 1 and at line 2 we set the viewport to the size of the framebuffer we created. Then we perform a render of the whole scene except the car for each of the six axis aligned directions. This requires to: set the view frame (line 65), bind the right framebuffer (line 66) and clearing the used buffers.

61 NVMCClient.drawOnReflectionMap = function (gl, position){

62  this.projectionMatrix = SglMat4.perspective(Math.PI /2.0 ,1.0 ,1.0 ,300);

63  gl.viewport (0,0,this.cubeMapFrameBuffers [0]. width ,this.cubeMapFrameBuffers [0]. height);

64  // +x

65  this, stack, load (SglMat4.lookAt (position, SglVec3. add (position,[1.0,0.0,0.0]),[0.0,1.0,0.0]));

66  gl.bindFramebuffer(gl.FRAMEBUFFER, this.cubeMapFrameBuffers [0]) ;

67  gl.clear(gl.COLOR_BUFFER_BIT | gl DEPTH_BUFFER_BIT);

68  this.drawSkyBox(gl) ;

69  this.drawEverything(gl,true, this.cubeMapFrameBuffers[0]);

70

71  // -x

72  this.stack.load(SglMat4.lookAt(position ,SglVec3.add(position, [−1.0 ,0.0 ,0.0]) ,[0.0 , −1.0 ,0.0]));

73  gl.bindFramebuffer(gl.FRAMEBUFFER , this.cubeMapFrameBuffers [1]);

74  gl.clear(gl.COLOR_BUFFER_BIT | gl.DEPTH_BUFFER_BIT);

75  this.drawSkyBox(gl);

76  this.drawEverything(gl,true, this.cubeMapFrameBuffers[1]);

LISTING 7.9: Creating the reflection map on the y. (Code snippet from http://envymycarbook.com/chapter7/4/0.js.)

Figure 7.19 shows a snapshot from the client with skybox, normal mapping (discussed later) applied to the street, and reflection mapping applied on the car.

7.7.5 Projective Texture Mapping

In this section we will see how to project an image on the scene, like a video projector or the Batman signal does. Like for environment mapping, there are no new concepts involved, just a smart use of texture coordinates generation.

The projector is described just like the view reference frame and it is conceptually the same object, with the only difference that instead of projecting the scene on its viewing plane, an image ideally placed in its viewing plane is projected into the scene. What we need to realize this effect is simply to assign to each point of the scene the texture coordinate obtained by projecting its position to the viewing plane of the projector.

$\begin{array}{cccc}\stackrel{\mathit{\text{t}}}{\overbrace{\left[\begin{array}{c}\mathit{\text{s}}\\ \mathit{\text{t}}\\ \mathit{\text{r}}\\ \mathit{\text{q}}\end{array}\right]}}& \stackrel{\mathit{\text{T}}}{\overbrace{\left[\begin{array}{llll}\frac{1}{2}\hfill & 0\hfill & 0\hfill & \frac{1}{2}\hfill \\ 0\hfill & \frac{1}{2}\hfill & 0\hfill & \frac{1}{2}\hfill \\ 0\hfill & 0\hfill & \frac{1}{2}\hfill & \frac{1}{2}\hfill \\ 0\hfill & 0\hfill & 0\hfill & 1\hfill \end{array}\right]}}& {\mathit{\text{P}}}_{\mathit{\text{p}}\mathit{\text{r}}\mathit{\text{o}}\mathit{\text{j}}}{\mathit{\text{V}}}_{\mathit{\text{p}}\mathit{\text{r}}\mathit{\text{o}}\mathit{\text{j}}}& \stackrel{\mathit{\text{p}}}{\overbrace{\left[\begin{array}{c}\mathit{\text{x}}\\ \mathit{\text{y}}\\ \mathit{\text{z}}\\ 1\end{array}\right]}}\end{array}$

where Pproj Vproj are the projection matrix and view matrix of the projector, respectively, and T is the transformation that maps the space from the canonical viewing volume [−1, +1]3 to [0, +1]3. Note that the final values actually used to access the texture will be the normalized ones, i.e. (s/q, t/q), while coordinate r is unused in this case.

7.8 Texture Mapping for Adding Detail to Geometry

Until now we used texture mapping as a way to add color information to the scene. The textures always contained color, which could be a picture or the surrounding environment. In a more abstract way, we added detailed information about the color to a rougher description of a surface (geometry).

If we observe the objects of the physical world, we can see that almost every object can be seen as a rough, let us say a large scale, geometric description and a finer geometric detail applied to its surface. For example, the trunk of a tree may be seen as a cylinder (large scale) plus its cortex (the finer detail), the roof of a house may be seen as a rectangle plus the tiles, a street may be seen as a plane plus the rugosity of the asphalt. In CG, the advantage of this way of viewing things is that usually the geometric detail can be efficiently encoded as a texture image and used at the right time (see Figure 7.20). In the following we will see a few techniques that use texture mapping to add geometric detail without actually changing the geometry.

7.8.1 Displacement Mapping

With displacement mapping the texture contains, for each texel, a single value encoding how much the corresponding geometric point should be moved (displaced) along the normal in that point. To implement this technique we should, at some point along the pipeline, replace the original geometry with a tessellation having a vertex for each texel in order to displace it accordingly. The way this is done is by implementing a ray casting algorithm in the fragment shader and since it is not strictly texture mapping we will explain it in Section 9.3.

7.8.2 Normal Mapping

With normal mapping we use textures to encode the normals and use them during rasterization to compute per-fragment lighting. We refer to these textures with the name of normal maps.

Unlike displacement mapping, the geometry is not altered, we just change the value of the normal to be the one that we would have if the geometry was actually displaced. This means that the normal we will have on each point of the geometry is inconsistent with the geometry around the point and makes us realize how much the perception of the appearance of a surface is deeply related to its normal, that is, to the lighting which in turn depends on the normal.

Figure 7.21 shows the error committed by using normal mapping instead of the exact geometry. The view ray r1 hits the base geometry at point p and the normal np is used for shading, while if we had the actual geometry the hitting point would have been p′ and the normal np′. This is the typical parallax error and depends on two factors: how far is the real surface from the base geometry and how small is the angle formed by the view ray and the base geometry.

The view ray r2 will entirely miss the object, while it would have hit the real geometry. This is the very same parallax error with more evident consequences and it means that we will be able to spot the real shape of geometry on the object’s silhouette. In a few words, implementing normal mapping simply means redo the Phong shading we did in Section 6.9.2, perturbing the interpolated normal with a value encoded in a texture.

7.8.2.1 Object Space Normal Mapping

When using normal mapping we must choose in which reference frame we express the perturbation of the normal. If the reference frame is the one where we express the object we speak of object space normal mapping. In this case the value stored in the normal maps is simply added to the interpolated normal n:

${\mathit{\text{n}}}^{\prime }=\mathit{\text{n}}+\mathit{\text{d}}$

Figure 7.22 shows a classic example of object space normal mapping. On the left is a detailed model of the head of a statue, made of 4M triangles. In the middle is the base geometry and on the right the result of using normal mapping.

In order to implement normal mapping, we have to consider in which space the summation is actually done. We recall from Chapter 6 that light is normally specified in world space, while the values in the normal map are expressed in object space. Hence we are left with two options: either we apply to each vector of the normal map the same object-to-world space transformation that is applied to the object, or we apply the inverse transformation, world-to-object space to the light position (or direction for directional lights). With the first option, we will have to perform a matrix multiplication in the fragment shader, while in the second case we can transform the light position/direction in the vertex shader and let the interpolation provide the value for the fragment.

7.8.3 Upgrade Your Client: Add the Asphalt

We interpret the asphalt as a rugosity of the street. Since our street is laying on the horizontal plane we can simply apply object space normal mapping. The modifications to our client are straightforward. We already applied textures to the street in Section 7.5; what we need to add is to load a texture containing a normal map for the asphalt and modify the shader program to take this new texture into account. Figure 7.23 shows a normal map where the three (x, y, z) coordinates are stored as R, G, B color. Note that the color values stored in a texture are in the interval [0, M], while the object space coordinates for the normal are in the interval [0, 1]. The reason the texture is all bluish is that the up direction is stored in the blue channel and for the street it is always in the positive halfspace, while the other two may be both positive and negative, so it will have lower value on average. Listing 7.10 shows the fragment shader that uses normal maps. Note that after we access the normal map we remap the values from [0, 1] to [−1, 1] (lines 25 to 27); then we use this value as normal to compute the shading (line 28). For the sake of readability of the code we use a simple Lambertian material, which is reasonable for the asphalt, but, of course, you can plug every arbitrarily complex shading instead.

15 var fragmentShaderSource = "

16  precision highp float ;  

17  uniform sampler2D texture; 

18  uniform sampler2D normalMap; 

19  uniform vec4 uLightDirection; 

20  uniform vec4 uColor;  

21  varying vec2 vTextureCoords ; 

22  void main(void)   

23  {     

24 vec4 n=texture2D(normalMap, vTextureCoords); 

25 n.x =n.x*2.0 -1.0;      

26 n.y =n.y*2.0 -1.0;      

27 n.z =n.z*2.0 -1.0;      

28 vec3 N=normalize(vec3(n.x,n.z,n.y));   

29 float shade = dot(-uLightDirection.xyz , N); 

30 vec4 color=texture2D(texture , vTextureCoords); 

31 gl_FragColor = vec4(color.xyz* shade ,1.0) ; 

32}" ;

LISTING 7.10: Fragment shader for object space normal mapping. (Code snippet from http://envymycarbook.com/chapter7/2/shaders.js.)

7.8.4 Tangent Space Normal Mapping

Suppose that we want to define a normal map that is not specific for a predefined mapping M but that can be placed wherever we want, just like we can place a sticker on any part of our car. In this case we cannot simply encode the normal variation as an absolute value, simply because we do not know a priori the normal value at the point where each texel will be mapped. In other words, we want to do with the normal mapping what we do with the color mapping: to create the texture without taking care where it is applied.

The main difference between applying the color from the texture to the surface and applying the normal is that the color encoded in a texel does not have to be transformed in any way to be mapped on the surface: a red texel is red whatever geometric transformation we apply to the object. Not so for the normals, which need a 3D reference frame to be defined. In object space normal mapping, such a reference frame is the same reference frame as for the object and that is why we can write directly the absolute values for the normal.

Let us build a three-dimensional frame Ct centered at a point t in the texture and with axis u = [1,0,0], v = [0,1,0] and n = [0,0,1]. What we want is a way to map any value expressed in the frame Ct on the surface at a point where t is projected p = M(t). In order to do this, we build a frame Tf with origin M(t) and axis

$\begin{array}{lll}{\mathit{\text{u}}}_{\mathit{\text{o}}\mathit{\text{s}}}\hfill & =\hfill & \frac{\partial \mathit{\text{M}}}{\partial \mathit{\text{u}}}\left(\mathit{\text{p}}\right)\hfill \\ {\mathit{\text{v}}}_{\mathit{\text{o}}\mathit{\text{s}}}\hfill & =\hfill & \frac{\partial \mathit{\text{M}}}{\partial \mathit{\text{v}}}\left(\mathit{\text{p}}\right)\hfill \\ {\mathit{\text{n}}}_{\mathit{\text{o}}\mathit{\text{s}}}\hfill & =\hfill & {\mathit{\text{u}}}_{\mathit{\text{o}}\mathit{\text{s}}}\times {\mathit{\text{v}}}_{\mathit{\text{o}}\mathit{\text{s}}}\hfill \end{array}$

therefore we can map a vector expressed in Ct as [x, y, z]T to Tf as Mp(x,y,z) = x uos + y vos + z nos. Note that, by construction, vectors uos and vos are tangent to the surface. This is why the set of all vectors generated by them is called tangent space and the frame Tf tangent frame.

Also note that, since we will only be transforming vectors, the origin of the frame will not come into play in the transformation between Ct and Tf.

It is worth mentioning that historically uos, vos and nos are referred to as T (tangent), B (bitangent of binormal) and N (normal). We prefer to use another notation to be more consistent with the rest of this book.

Once we have a way to compute the tangent frame we may apply normal mapping simply by tranforming the light position/direction from object space to tangent space, by multiplying it by the inverse of Tf. As for object space normal mapping, this transformation is done per-vertex and the result interpolated.

7.8.4.1 Computing the Tangent Frame for Triangulated Meshes

If we had an analytic form for M, then computing uos, vos would only be a matter of calculating a partial derivative. Instead, we usually have only the texture coordinates of a triangulated mesh like that shown in Figure 7.24. We will now use what we learned about frames in Chapter 4 to compute the tangential vectors uos and vos for the vertex v0.

Let us consider the frame Vf, with origin in v0 and axes v1 − v0 and v2 − v0, and frame If with origin in t0 and axes t1 − t0 and t2 − t0. These frames are known because we know both vertex position and texture coordinates of the vertices of the triangle. The other important thing we know is that uos and u have the same coordinates in the respective frames, therefore we can find uos by finding the coordinates uI of u = [1, 0, 0]T in the frame If and then uos = uIu v10 + uIv v20.

The end point of vector u is t0 + u. We recall from Chapter 4 that we can express the coordinates of t0 + u in the frame If as:

$\begin{array}{lll}{\mathit{\text{u}}}_{{\mathit{\text{I}}}_{\mathit{\text{u}}}}\hfill & =\hfill & \frac{\left(\left({\mathit{\text{t}}}_0+\mathit{\text{u}}\right)-{\mathit{\text{t}}}_0\right)\times {\mathit{\text{t}}}_{20}}{{\mathit{\text{t}}}_{10}\times {\mathit{\text{t}}}_{20}}=\frac{\mathit{\text{u}}\times {\mathit{\text{t}}}_{20}}{{\mathit{\text{t}}}_{10}\times {\mathit{\text{t}}}_{20}}=\frac{{\left[1,0\right]}^{\mathit{\text{T}}}\times {\mathit{\text{t}}}_{20}}{{\mathit{\text{t}}}_{10}\times {\mathit{\text{t}}}_{20}}=\frac{{\mathit{\text{t}}}_{{20}_{\mathit{\text{v}}}}}{{\mathit{\text{t}}}_{10}\times {\mathit{\text{t}}}_{20}}\hfill \\ {\mathit{\text{u}}}_{{\mathit{\text{I}}}_{\mathit{\text{v}}}}\hfill & =\hfill & \frac{{\mathit{\text{t}}}_{10}\times \left(\left({\mathit{\text{t}}}_0+\mathit{\text{u}}\right)-{\mathit{\text{t}}}_0\right)}{{\mathit{\text{t}}}_{10}\times {\mathit{\text{t}}}_{20}}=\frac{{\mathit{\text{t}}}_{20}\times \mathit{\text{u}}}{{\mathit{\text{t}}}_{10}\times {\mathit{\text{t}}}_{20}}=\frac{{\mathit{\text{t}}}_{10}\times {\left[1,0\right]}^{\mathit{\text{T}}}}{{\mathit{\text{t}}}_{10}\times {\mathit{\text{t}}}_{20}}=\frac{-{\mathit{\text{t}}}_{{10}_{\mathit{\text{u}}}}}{{\mathit{\text{t}}}_{10}\times {\mathit{\text{t}}}_{20}}\hfill \end{array}$

therefore

${\mathit{\text{u}}}_{\mathit{\text{o}}\mathit{\text{s}}}=\frac{{\mathit{\text{t}}}_{20\mathit{\text{v}}}}{{\mathit{\text{t}}}_{10}\times {\mathit{\text{t}}}_{20}}{\mathit{\text{v}}}_{10}+\frac{-{\mathit{\text{t}}}_{10\mathit{\text{u}}}}{{\mathit{\text{t}}}_{10}\times {\mathit{\text{t}}}_{20}}{\mathit{\text{v}}}_{20}$

The same derivation for vI leads to:

${\mathit{\text{v}}}_{\mathit{\text{o}}\mathit{\text{s}}}=\frac{-{\mathit{\text{t}}}_{20\mathit{\text{u}}}}{{\mathit{\text{t}}}_{10}\times {\mathit{\text{t}}}_{20}}{\mathit{\text{v}}}_{10}+\frac{-{\mathit{\text{t}}}_{10\mathit{\text{v}}}}{{\mathit{\text{t}}}_{10}\times {\mathit{\text{t}}}_{20}}{\mathit{\text{v}}}_{20}$

Note that doing this derivation we used the fact that the surface at point v0 is described by the triangle (v0,v1,v2). Specifically, the tangent plane at v0 contains the triangle. However this is not true because there are other triangles, lying in different planes, which share the same vertex. If we use one of those, we will obtain a different tangent frame. This is the very same consideration we did for vertex normal computation in Chapter 6 and shares the same conclusion: we can find the tangent frame for a vertex by averaging over all the frames computed using the triangles sharing that vertex.

7.9 Notes on Mesh Parametrization

We have seen in Chapter 3 that a 3D parametric surface is a function f that maps the points from a bidimensional domain Ω ∈ ℝ2 to a surface S ∈ ℝ3. For example, we may express a plane (see Figure 7.25) as:

$\mathit{\text{f}}\left(\mathit{\text{u}},\mathit{\text{v}}\right)=\left(\mathit{\text{u}},\mathit{\text{v}},\mathit{\text{u}}\right)$

If we had to find the texture coordinates for this plane for the vertex at position (x, y, z) we could simply use the inverse of f:

$\mathit{\text{g}}\left(\mathit{\text{x}},\mathit{\text{y}},\mathit{\text{z}}\right)={\mathit{\text{f}}}^{-1}\left(\mathit{\text{x}},\mathit{\text{y}},\mathit{\text{z}}\right)$

However, most of the time our surfaces have been modelled manually and are not parametric, so we do not have f. The term mesh parameterization indicates the definition of a bijective and continous mapping from a simple domain to a mesh S. In other words, we want to turn S into a parametric surface in order to use the inverse function g to define the texture coordinates.

When the use of mesh parametrization is limited to uv-mapping we just need an injective function g from S to Ω (the difference is that we do not need every point of Ω to be mapped on S). Note that when we specify the texture coordinate we are actually describing g by giving samples of its value, that is, its value on the vertices.

We already did some parametrization in the clients of this chapter by defining the texture coordinates for the buildings, the street and the ground. In those cases, however, finding g is straightforward because Ω and S are both rectangles, but consider finding g for the case shown in Figure 7.26. A few questions arise: Does g always exists? There maybe be more than one, and, if so, which is the best?

7.9.1 Seams

The answer to the first question is yes. Consider just taking each individual triangle of S and placing it onto the plane Ω in such a way that no triangles overlap, as shown in Figure 7.26. In this simple manner we will have our inijective function g that maps each point of S onto Ω. Note that the adjacent triangles on the mesh will not, in general, be mapped onto adjacent triangles on Ω, that is, g is not continuous.

Do we care that g is continuous? Yes, we do. Texture sampling is performed in the assumption that close points on S map to close points on Ω. Just consider texture access with bilinear interpolation, where the result of a texture fetch is given by the four texels around the sample point. When the sample is near the border of a triangle (see Bottom of Figure 7.26), the color is influenced by the adjacent triangle on Ω. If g is not continuous, the adjacent texels can correspond to any point on S or just to undefined values, that is, to values that do not belong to g(S) .

In parametrization the discontinuities on g are called seams. The method we used to build g is just the worst case for seams because it generates one seam for each edge of each triangle.

How do we know if we can avoid seams? Let us assume the mesh S is made of rubber and so we can deform it as we please. If we manage to “flatten” S onto Ω we have a seamless parametrization g. For example if S is a hemisphere and Ω is a plane (see Figure 7.27) we can flatten S onto Ω and define g(x,y,z) = (x, z). Conversely, if S is a sphere, there is no way we can deform it into a plane. What we need is a cut to “open” the sphere, that is, we create a seam, and now we can stretch the cut sphere onto the plane. A parametrization g that has no seams, like for the hemisphere above, is called global parameterization.

In practice you would rarely encounter global parameterizations on a planar domain for non-trivial shapes; you will always see cases like the one in Figure 7.28. On the other hand, you may find some seamless parameterizations on more complex domains than the plane.

7.9.2 Quality of a Parametrization

So we know there can be several parameterizations for the same S and Ω, but which is the best one? How can we measure how good a given g is?

Let us consider S and Ω again, and this time assume that S is made of paper, which can bend but cannot be stretched. If we are able to unfold S on the plane we have parameterization g with no distortion. Otherwise we would need to locally stretch the tissue so that it can be flattened. The amount of stretch we do is called distortion and it is a characteristic of the parameterization.

Let us consider an (infinitely) small square on the surface S placed at point p and its image on Ω. We have three cases for the image of the square on Ω:

• a square with the same area and the same angles, in this case the parametrization is called isometric
• a parallelogram with the same area as the square, in this case the parameterization is called equiareal
• a square with a different area, and we say the parameterization is called conformal or angle preserving

Note that every isometric parametrization is conformal and equiareal.

Do we care about distortion? Yes we do. Distortion on parameterization causes uneven sampling of textures. We may have large areas of S corresponding to just a few texels in texture space. A typical way to visually assess the quality of a parameterization is to set a regular pattern as texture image and look how regular it is when mapped on the mesh. The more the parameterization is isometric, the more the areas described by the patterns will be similar over the mesh and the right angles made by the line will stay right on the mesh. Looking again at Figure 7.27, we can see how the parameterization obtained by orthogonal projection (Top row) is okay in the center but degrades towards the borders, while a more sophisticated algorithm (described in [38]) appears more regular everywhere.

As aforementioned, parametrization is a wide topic with a vast literature and there are many proposals on how to achieve a good parameterization. We want to point out that distortion and seams are two related characteristics of a parameterization. For example, the trivial parametrization by triangle has 0 distortion but a large number of seams. On the other hand, the flattened hemisphere has no seams but quite a large distortion that we could reduce by making cuts. Figure 7.29 shows an example of a bad parameterization obtained without allowing any seam and a good one obtained by allowing seams. In fact the seams actually partition the mesh in three separate parts that are then arranged in texture space.

7.10 3D Textures and Their Use

A 3D texture, or volumetric texture, is none other than a stacked series of sz textures of size sx × sy that are accessed by specifying three coordinates UVT. Note that filtering (both magnification and minification/mipmapping) is extended to 3D by considering the z dimension, that is, the 2 × 2 × 2 = 8 texels around the access coordinates. So the ith mipmap level is created by taking the average of a group of eight texels. 3D textures are supported in Opengl and OpenGL ES but not (yet) included in the WebGL API so we will not see them at work in our client. Here we briefly mention some use for it.

So far we have always represented volumetric objects by their external surface, but we know from Section 3.5 that the scene may be encoded with voxels and it is clear that a voxel-based representation maps naturally on a 3D texture. If we wanted to visualize a “slice” of volume data we would only need to render a plane and assign the 3D textures’ coordinates to the vertices, letting texture sampling do the rest.

A more intriguing matter is what is called volume rendering, that is, how to render a volume of data on a 2D device, for which we showed the sample in Figure 3.17. This is a vast topic on its own for which we refer the reader to [45].

With the term participating media we indicate those elements that fill a volume without a precise boundary, such as fog and smoke, where light-matter interaction does not stop at the surface. In these cases 3D textures are often updated dynamically by a simulation algorithm that animates the media.

7.11 Self-Exercises

7.11.1 General

1. Suppose we do not use perspective correct interpolation and we see the same scene with two identical viewer settings except that they have different near plane distances. For which setting is the error less noticeable? How much would the error be with an orthogonal projection?
2. If we want to obtain a rendering of a checkerboard with n × n b/w tiles, how small can the texture be? Do we need an n × n pixels image or can we do it with a smaller one? Hint: Play with texture wrapping
3. Consider the sphere mapping and the cube mapping in term of mesh parameterization. What is S and what is Ω? Which one is the less distorted parametrization? Which one has seams?
4. May a torus be parametrized on a planar domain without seams? And with one seam? How many seams do we need?
5. What is the reason for 3D textures? Is a 3D texture of size sx × sy × sz not the same as sz 2D textures?

7.11.2 Client

1. Make a running text on a floating polygon over the car. Hint: Make the u texture coordinate increase with time and use “repeat” as wrapping mode.
2. Implement a client where the headlights project a texture. Hint: See Section 7.7.5