i
i
i
i
i
i
i
i
10.9. Image Processing 469
0.0545
0.0545
0.2442
0.2442
0.4026
0.0545
0.0545
0.2442
0.2442
0.4026
0.0545
0.0545
0.2442
0.2442
0.4026
0.0545
0.0545
0.2442
0.2442
0.4026
0
.054
5
0
.054
5
0.2
44
2
0.2
44
2
0
.4026
0.054
5
0.054
5
0.2
44
2
0.2
44
2
0.402
6
0
.054
5
0
.054
5
0.2
44
2
0.2
44
2
0
.402
6
0.0545
0.0545
0.2
44
2
0.2
44
2
0.4026
0.05450.0545 0.24420.2442 0.4026
0.05450.0545 0.24420.2442 0.4026
0.05450.0545 0.24420.2442 0.4026
0.05450.0545 0.24420.2442 0.4026
0.05
4
5
0.05
4
5
0.2
44
2
0.2
44
2
0.
4
026
0
.054
5
0
.054
5
0.244
2
0.2442
0
.402
6
0.05
4
5
0.05
4
5
0.2
44
2
0.2
44
2
0.
4
0
2
6
0
.
0
54
5
0
.054
5
0.244
2
0.2442
0
.4
026
0.0030
0.0030
0.0030
0.0030
0.0133
0.0133
0.0133
0.0133
0.0133
0.0133
0.0133
0.0133
0.0219
0.0219 0.0219
0.0219
0.0596 0.0596
0.0596 0.0596
0.0983
0.0983 0.0983
0.0983
0.1621
0.0545
0.0545
0.2442
0.2442
0.4026
0.05450.0545 0.24420.2442 0.4026
(a)
(b) (c)
Figure 10.21. One way to use a Gaussian blur kernel is to sample a 5 × 5 area, weighting
each contribution and summing them. Part (a) of the figure shows these weights for
a blur kernel with σ = 1. Using separable filters, two one-dimensional Gaussian blurs,
(b) and (c), are performed, with the same net result. The first pass, shown in (b) for
5 separate rows, blurs each pixel horizontally using 5 samples in its row. The second
pass, (c), applies a 5-sample vertical blur filter to the resulting image from (b) to give
the final result. Multiplying the weights in (b) by those in (c) gives the same weights as
in (a), showing that this filter is equivalent and therefore separable. Instead of needing
25 samples, as in (a), each of (b) and (c) effectively each use 5 per pixel, for a total of
10 samples.
i
i
i
i
i
i
i
i
470 10. Image-Based Effects
A problem with the sinc and Gaussian filters is that the functions go
on forever. One expedient is to clamp such filters to a specific diameter or
square area and simply treat anything beyond as having a value of zero.
A sharp dropoff can sometimes cause problems. Cook [195], in his classic
paper on stochastic sampling, presents a truncated Gaussian filter:
Cook(x)=
⎧
⎪
⎨
⎪
⎩
e
−
x
2
2σ
2
− e
−
w
2
2σ
2
,x≤ w,
0,x>w,
(10.7)
where w is the radius of interest.
7
Beyond this radius, the evaluated number
would be negative, so it is instead set to zero. This filter kernel has a
smoother dropoff than just clamping the Gaussian. Other filtering kernels,
such as Mitchell-Netravali [872], are designed for various properties (such
as blurring or edge sharpening) while avoiding expensive (on the CPU, less
so on a modern GPU) power functions. Bjorke [95], Mitchell et al. [877],
and Rost [1084] each provide some common rotation-invariant filters and
much other information on image processing on the GPU.
The trend for GPUs is that reading texels is the most expensive shader
operation (due to bandwidth costs) and that computations drop in relative
cost as time progresses. One advantage of using the GPU is that built-in
interpolation and mipmapping hardware can be used to help minimize the
number of texels accessed. For example, say the goal is to use a box filter,
i.e., to take the average of the nine texels forming a 3 × 3 grid around
a given texel and display this blurred result. These nine texture samples
would then be weighted and summed together by the pixel shader, which
would then output the blurred result to the pixel.
Nine samples are unnecessary, however. By using bilinear filtering of
a texture, a single texture access can retrieve the weighted sum of up to
four neighboring texels. So the 3 × 3 grid could be sampled with just four
texture accesses. See Figure 10.22. For a box filter, where the weights are
equal, a single sample could be placed midway among four texels, obtaining
the average of the four. For a filter such as the Gaussian, where the weights
differ in such a way that bilinear interpolation between four samples can be
inaccurate, each sample can still be placed between two texels, but offset
closer to one than the other. For instance, imagine one texel’s weight was
0.01 and its neighbor was 0.04. The sample could be put so that it was a
distance of 0.8 from the first, and so 0.2 from the neighbor, and the sample
weight would be 0.05. Alternatively, the Gaussian could be approximated
by using a bilinear interpolated sample for every four texels, finding the
offset that gives the closest approximation to the ideal weights.
7
As discussed in the previous paragraph, the unused constant scale factor is not
shown.
i
i
i
i
i
i
i
i
10.9. Image Processing 471
Figure 10.22. On the left, a box filter is applied by performing nine texture samples
and averaging the contributions together. In the middle, a symmetric pattern of five
samples is used, with the outside samples each representing two texels, and so each is
given twice the weight of the center sample. This type of pattern can be useful for other
filter kernels, where by moving the outside samples between their two texels, the relative
contribution of each texel can be changed. On the right, a more efficient four sample
pattern is used instead. The sample on the upper left interpolates between four texels’
values. The ones on the upper right and lower left each interpolate the values of two
texels. Each sample is given a weight proportional to the number of texels it represents.
Some filtering kernels are separable. Two examples are the Gaussian
and box filters. This means that they can be applied in two separate one-
dimensional blurs. Doing so results in considerably less texel access being
needed overall. The cost goes from d
2
to 2d,whered is the kernel diameter
or support [602, 877, 941]. For example, say the box filter is to be applied
in a 5 × 5 area at each pixel in an image. First the image could be filtered
horizontally: The two neighboring texels to the left and two to the right
of each pixel, along with the pixel’s value itself, are equally weighted by
0.2 and summed together. The resulting image is then blurred vertically,
with the two neighboring texels above and below averaged with the central
pixel. Instead of having to access 25 texels in a single pass, a total of 10
texels are accessed in two passes. See Figure 10.21. Using the bilinear
filtering technique for sampling, this can be reduced to three texel accesses
per pixel per pass, for a total of six accesses, versus nine for a single-pass
area interpolated approach.
Downsampling is another GPU-related technique commonly used for
blurring. The idea is to make a smaller version of the image to be ma-
nipulated, for example halving the resolution along both axes to make a
quarter-screen image. Depending on the input data and the algorithm’s
requirements, the original image may be filtered down in size or simply
sampled or created at this lower resolution. When this image is accessed
to blend into the final, full resolution image, magnification of the texture
will use bilinear interpolation to blend between samples. This gives a fur-
ther blurring effect. Performing manipulations on a smaller version of the
original image considerably decreases the overall number of texels accessed.
Also, any filters applied to this smaller image have the net effect of increas-
i
i
i
i
i
i
i
i
472 10. Image-Based Effects
ing the relative size of the filter kernel. For example, applying a kernel
with a width of five (i.e., two texels to each side of the central pixel) to the
smaller image is similar in effect to applying a kernel with a width of nine
to the original image. Quality will be lower, but for blurring large areas
of similar color, a common case for many glare effects and other phenom-
ena, most artifacts will be minimal. The main problem is flickering during
animation [602].
One last process is worth mentioning: ping-pong buffers [951]. This
term is commonly used in GPU image processing. It is simply the idea
of applying operations between two offscreen buffers, each used to hold
intermediate or final results. For the first pass, the first buffer is the input
texture and the second buffer is where output is sent. In the next pass the
roles are reversed, with the second now acting as the input texture and the
first getting reused for output. In this second pass the first buffer’s original
contents are overwritten—it is just being reused as temporary storage for
a processing pass.
The field of animation is beyond the scope of this volume, but it should
be noted that the GPU can also assist in performing texture animation
via image processing. For example, James [600] first showed how a variety
of animated effects can be done by modifying textures on the GPU from
frametoframe. Thatis,atextureimage can be modified by a pixel shader
and the result rendered on a surface. Then in the next frame, the result
is modified again by the same pixel shader and rendered. Using one- and
two-dimensional cellular automata rules, effects such as fire, smoke, and
interactive rippling water can be created on a surface. Textures created
can be used to modify the surface color, shading normal, heightfield, or
any other attributes. Many articles have been written about performing
fluid flow animation on the GPU; see Sanders et al. [1108] for a summary of
past work in this area, along with a presentation of their own optimization
technique. Tatarchuk’s article on simulating rain [1246] is noteworthy for
its innovative use of a wide range of techniques. Crane et al. [205] discuss
three-dimensional fluid flow, a topic touched upon later in Section 10.16
on volume rendering.
A type of filter that has seen much recent use in graphics and related
fieldsisthebilateral filter. This filter uses not only the distance between
the texel and central pixel for weighting, but the difference in their values
as well. This results in a “smart blur” that preserves sharp edges. The
recent SIGGRAPH course on the subject [988] is a good overview of the
bilateral filter and its applications.
Pixel shaders can be used in post processing to imitate thermal imag-
ing [549], reproduce film grain [928], perform edge detection [95, 370, 877],
generate heat shimmer [928] and ripples [28], posterize an image [28], render
clouds [55], simulate VCR playback and rewind [254], and for a huge num-
i
i
i
i
i
i
i
i
10.9. Image Processing 473
Figure 10.23. Image processing using pixel shaders. The original image in the upper left
is processed in various ways. The upper right shows a Gaussian difference operation,
the lower left edge shows detection, and lower right a composite of the edge detection
blended with the original image. (Images courtesy of NVIDIA Corporation.)
ber of other operations [95, 600, 875, 876, 941]. See Figure 10.23. These
examples use a color image as the only input. Synthesized images also
come with a Z-buffer, and this data can also be used in post processing,
as well as any other buffers the user cares to generate. For example, the
stencil buffer can be used to perform screen dissolves and other video-like
effects [693].
Rather than continue with an exhaustive (and exhausting) litany of all
possible effects, the sections that follow cover some of the common uses
for image processing on the GPU, along with related areas such as high
dynamic range images.
..................Content has been hidden....................
You can't read the all page of ebook, please click here login for view all page.