9.2.1 Single-Frame Super-Resolution

Single-frame SR approaches are widely applied solutions to the resolution enhancement problem, especially in cases where only one degraded low-resolution (LR) observation is available [17]. Single-frame SR approaches can also be referred to as image interpolation or reconstruction techniques and thus can be used interchangeably. Recovering a high-resolution image from an under-sampled (according to Nyquist limits [18]) and noisy observation is a highly ill-posed inverse problem. Thus, prior image models relating neighboring pixels or prior models learned through similar image patches are needed extra information that can aid in solving the inverse problem. A common approach among these SR techniques is that they take advantage of the relation between neighboring pixels of the same image to estimate the values of missing pixels. Figure 9.1 illustrates a general single-frame SR process where fand f⁻ denote the forward and backward degradation process of the imaging system, respectively.

A well-known problem with common kernel-based super-resolution, such as bilinear and bicubic interpolation [19], is the blurring and blockiness effects of sharp edges. The blurring of sharp edges results from the inaccurate kernel resizing to adapt to the edge sharpness. The edge blockiness, also known as staircase effects, is mainly due to the failure of the filter to adapt to various edge orientations. Reference [20] proposed an edge-directed interpolation method that uses local covariance estimates of the input LR frame to adapt interpolation coefficients to arbitrarily oriented edges of the reconstructed HR image. This method is motivated by the geometric regularity property [21] of an ideal step edge. Recently, Reference [22] presented a new edge-directed interpolation approach that uses multiple low resolution training windows to reduce the covariance mismatch between the LR and HR pixels. Reference [23] proposed a smart interpolation method that uses anisotropic diffusion [24], a technique aiming at reducing noise and enhancing contrast without significantly degrading edges and objects in an image. References [25], [26], [27], [28], and [29] proposed SR methods based on the optimal recovery principle, which model the local image regions using ellipsoidal signal classes and adapts the interpolating kernels accordingly. Another model-based SR approach [30] is based on multi-resolution analysis in the wavelet domain, where the statistical relationships between coefficients at coarser scales are modeled using hidden Markov trees to predict the coefficients on the finest scale. Finally, learning-based SR approaches recover missing high-frequency information in an image through matching using a large image database [31], [32], [33], [34].

Many single-frame SR methods are computationally efficient. However, super-resolving from a single low-resolution image is known to be a highly ill-posed inverse problem due to the low number of observations relative to the large number of missing pixels/unknowns. Thus, the gain in quality in the single-frame SR approach is limited by the minimal number of information provided to recover missing details in the reconstructed HR signal.

9.2.2 Multi-Frame Super-Resolution

Multi-frame SR techniques offer a better solution to the resolution enhancement problem by exploiting extra information from several neighboring frames in a video sequence. In a multi-frame acquisition system, the subpixel motion between the camera and object allows the frames of the video sequence to contain oversampled similar information that make the reconstruction of an HR image possible.

This chapter focuses on multi-frame SR techniques that enhance the resolution of images by combining information from multiple low-resolution (LR) frames of the same scene to estimate a high-resolution (HR) unaliased and sharp/deblurred image under realistic bandwidth requirements and cost-effective hardware [6]. Interest in multi-frame super-resolution re-emerged in the recent years mainly due to the use of multi-frame image sequences, which can take advantage of additional spatio-temporal information available in the video content, and the increase of hardware computational power and advancement of display technologies, which make SR applications possible. Figure 9.2 presents a block diagram showing a multi-frame SR estimation process using multiple degraded LR frames, with subpixel shifts, to estimate one HR reference frame. The figure also shows the LR pixels registered relative to a common HR reference grid to further visualize the number of pixels used in the process of estimating the missing pixels in the SR image.

Since Reference [35] first introduced the multi-frame image restoration and registration problem/solution, several multi-frame SR approaches have been proposed in the past two decades. These multi-frame SR approaches are described and broadly categorized according to their methods of solution into Bayesian, regularized norm minimization, fusion-restoration (FR), and non-iterative fusion-interpolation (FI) approaches. Bayesian maximum a posteriori (MAP) solutions have gained great attention and proved to be effective due to the inclusion of a priori knowledge about the HR estimate and inherent probabilistic formulation of the relation between the LR observations and the HR image [36], [37], [38], [39]. Regularized norm minimization solutions aim at minimizing an error term with some smoothness regularization assumption on the HR estimate [40], [41], [42]. It was shown in Reference [42] that the l₁-norm minimization approach is the most robust to errors in the system and that the bilateral total variation regularization function gave the best performance in terms of robustness and edge preservation. The two-step fusion-restoration (FR) SR estimation methods [40], [42] were devised in order to decrease the computational requirements and to increase the robustness to outliers. The FR methods employ a non-iterative fusion step, which includes a one-step registration process, and a restoration step, which simultaneously deblurs and denoises the fused image by minimizing an error function with or without a specific regularization term. The non-iterative fusion-interpolation (FI) SR approaches [43], [44] are composed of a fusion step and a non-iterative reconstruction step using kernel-based interpolation, which is inherently efficient in nature; however, these non-iterative methods can result in lower visual quality than the iterative methods as discussed later in this chapter. All the previous methods consider the subpixel motion between the observations to recover missing information by solving the SR problem.

9.4.1 Bayesian Super-Resolution Formulation

Bayesian MAP-based estimators are common solutions for the SR problem since they offer fast convergence and high-quality performance [36], [37], [38], [39]. In Bayesian SR solutions all parameters or unknowns (that is, HR image, motion parameters, and noise) and observable variables (that is, the LR observations) are assumed to be unknown stochastic quantities with assumed probability distributions based on subjective beliefs. In the following formulation of the MAP solution, the motion parameters are assumed to be known for simplicity. For example, in cases of compressed video content the motion vectors can be retrieved from the headers of the bitstream (note that motion estimation is a mature field of research with many proposed accurate methods). In order to estimate the HR image z, a Bayesian MAP estimator is formed given the low-resolution frames y_k, for k = 1,2,...,K, and appropriate prior. The HR estimate z^ can be computed by maximizing the a posteriori probability Pr(z|, or by minimizing the log-likelihood function as follows:

z^=argminzlog[ Pr(z|{ yk }) ].⁢(9.3)

Using Bayes rule and assuming that the LR observations y_k are statistically independent of z, the problem reduces to

z^=argminz{ −log[ Pr({ yk }|z) ]−log[ Pr(z) ] }.⁢(9.4)

Now solving for an accurate HR estimate in Equation 9.4 is highly dependent on the prior HR image density Pr(z) and the conditional LR density Pr({y_k}|z) models. Note that when dropping the prior HR probability model in Equation 9.4, the MAP optimization problem reduces to an ML estimation problem that is highly unstable under small errors in the parameters of the acquisition model and noisy conditions [37]. It has been widely used in Bayesian SR formulation literature [36], [37], [38], [39], [45], [46], [47], [48], [49], [50], [51], [52] that the noise model is assumed to be a zero mean Gaussian. From Equation 9.4, and given that the elements of n_k are i.i.d Gaussian random variables, the conditional probability distribution can be modeled as follows:

Pr(yk|z)=1(2π)N2σηNexp{ −12ση2‖yk−Wkz‖2 },⁢(9.5)

where ση2 is the noise variance. The problem of determining which HR image prior model is the best for a particular HR reconstruction is still an open problem widely targeted by various existing literature [6], [53]. However, a common approach followed by existing MAP-based SR solutions is the assumption of smoothness constraints on the HR priors within homogeneous regions [36], [37], [54], [55]. These priors can generally be modeled as

Pr(z)∝exp[ −λ2‖Qz‖2 ],⁢(9.6)

where Q represents a linear high-pass operator that penalizes the estimates that are not smooth and λ controls the variance of the prior distribution. In References [36] and [37], these piecewise smoothness priors in Equation 9.6 take the form of Huber-Markov random fields that are modeled as Gibbs prior functional according to Reference [56]. Then, the prior model can be written as follows:

Pr(z)=12zTCz−1z,⁢(9.7)

where C_z is the covariance of the HR image prior model imposing piecewise smoothness constraints between neighboring pixels. Thus, with the smoothness prior model (Equation 9.7) and mutually independent additive Gaussian noise on the prior error model (Equation 9.5), the MAP SR estimation problem can be formulated by minimizing the following convex cost function with a unique global minimum:

f(z)=12ση2∑k=1K(yk−WkzT)(yk−Wkz)+12zTCz−1z,⁢(9.8)

where W_k is the degradation matrix for frame k, the term ση2 denotes the noise variance, z is the HR frame in lexicographical form, and y_k are the observed LR frames in vector form. Thus, the MAP estimator can be reformulated as a least-squares error minimization problem, which in matrix notation is the l₂-norm square of the error vector, with a smoothness regularization constraint, given by

z^=argminz{ Cη−1∑k=1K‖yk−Wkz‖22+λΓ(z) },⁢(9.9)

where ‖ • ‖22 is the square of the l₂-norm and C_η is the i.i.d. Gaussian noise covariance matrix equal to ση2I. The λ in the second term is a regularization weighting factor, and Г(z) is a smoothness regularization constraint in function of the SR image prior.

9.4.2 Regularized-Norm Minimization Formulation

Here, an SR image is estimated by following a regularized-norm minimization paradigm [40], [41], [42], [57]. Then, in an underdetermined system of equations (Equation 9.1), estimating the HR image z given a sequence of LR observations y_k, for k = 1,2,...,K, is commonly formulated as an optimization problem minimizing an error criteria and a regularization term. Thus, the SR optimization problem can be written as

z^=argminz{ f(z) },⁢(9.10)

where the cost function f(z) has the following form:

f(z)=1γ∑k=1KE(yk,Wkz)+λΓ(z).⁢(9.11)

In the above equation, E(·) is the error term in function of y_k and z. The weighting factors γ and λ are constant tuning parameters. The regularization term Г(·), which is in function of the HR image only, is designed to preserve important image content or structures, such as edges and objects, and also to increase the robustness of the solution to outliers and errors in the system. Unlike MAP-based algorithms, such as the one presented in Section 9.4.1, there is no a priori assumption made about the distribution function of the reconstructed HR image. Reference [42] proved that an l₁-norm imposed on the error residual is the most robust solution against outliers. Also in Reference [42], different regularization terms are considered for best performance in terms of robustness and edge preservation. Therefore, a cost function formulation for the SR problem can be expressed as follows:

f(z)=1γ∑k=1K‖ yk−Wkz ‖pp+λΓ(z),⁢(9.12)

where ‖ ⋅ ‖pp is the l_p-norm raised to the power p = {1,2}. The weighting factor γ is set to { 1,ση2 } for p = {1,2}, respectively. Previous SR solutions are based on the warp-blur observation model following Equation 9.1. Assuming a circularly symmetric blurring matrix and a translation or rotation type of motion, then the motion and blur matrices in Equation 9.1 can be swapped. Consequently, the observation model, referred to by the blur-warp model, can be deduced from Equation 9.1 by defining the degradation matrix as W_k = DF_kH, where F_k is the warping matrix of size N × N, the term H denotes the blurring matrix of size N × N representing the common PSF function, and D is the decimation matrix of size N₁N₂× N. The question as to which of the two models (blur-warp or warp-blur) should be used in SR solutions is addressed in Reference [58]. Following this blur-warp observation model, a fast implementation of the regularized-norm minimization solution, referred to as the fusion-restoration (FR) approach, can be achieved by solving for a blurred estimate zb=Hz^ of the HR image followed by an interpolation and deblurring iterative step. Often, the blurred HR estimate is a non-iterative approach composed of registering all the LR observations relative to the HR grid and estimating the HR pixel by using an average or median operator of the LR pixels at each HR location [40], [41], [42]. The formulation of this FR approach will be discussed in Section 9.7.2.

9.4.3 Non-Iterative Kernel-Based Formulation

In the category of non-iterative kernel-based SR solutions, following the blur-warp observation model, all the LR observations are registered and merged on a common HR grid and non-iterative kernel-based solutions solve for the best estimate through adaptive kernel design techniques. The blur-warp acquisition model can be visualized as a non-uniform sampler (U = DF_k) applied on the blurred HR estimate z_b as shown in Figure 9.4.

A locally adaptive approach using such SR estimators is described in References [43] and [44], where the fused LR samples are processed using a moving observation window to estimate the interpolation kernel and an estimation window to apply the designed kernel on the spanned LR observed samples to estimate the missing HR pixels. Figure 9.5 illustrates the non-iterative estimation approach using locally adaptive convolution kernel processing.

From Figure 9.5, consider that the observation window is of size W_x × W_y pixels on the HR grid and spans P = KW_xW_y/L² LR pixels denoted by vector G_i, and the estimation window is of size D_x × D_y pixels on the HR grid and spans D_xD_y estimated SR pixels denoted by vector D_i, where i is the location of the respective window index. Estimating a local set of SR pixels can then be achieved by simply filtering the vector G_i by its locally designed kernel coefficients W_i, following D_i = W_i.G_i. In Reference [43], the interpolation kernel coefficients are designed by minimizing the mean square error of D_i− W_iG_i. Solving for the optimal weights of the matrix W_i for each partition reduces to an adaptive Wiener filter solution for the considered observed LR pixels in the window. In References [44] and [59], the weights of the Wiener filter are defined as

Wi=Ri−1Pi,⁢(9.13)

where Ri=E{ GiGiT } and Pi=E{ GiDiT }. Thus, the determination of the weighting coefficients in Equation 9.13 requires the unknown HR image that can be either modeled parametrically or by training data. To avoid training, a parametric modeling approach can be adopted as described in Reference [43]. This category of SR estimation is inherently computationally efficient since it is non-iterative in nature.

9.4.4 Super-Resolution Solutions in Practice

Existing SR approaches suffer from extremely high computational requirements due to the high dimensionality in terms of large number of unknowns to be estimated in the solution of the SR inverse problem. Generally, MAP-based SR methods are computationally expensive, but can converge to a high-quality solution. Even for solutions with fast convergence rates, commonly used Bayesian approaches are conditioned on the number of different LR observations and the HR image prior statistical model that can lead to very high computational requirements even for small image estimates. To reduce the computations required for the regularized norm minimization SR solutions, the fusion-restoration (FR) methods register and merge all the LR observations on one HR grid before using an iterative regularized minimization reconstruction process. However, these solutions are still computationally intensive due to the high dimensionality of the problem with good reconstruction quality. Moreover, the non-iterative fusion-interpolation (FI) SR approaches, also referred to as kernel-based SR, are inherently less computationally intensive but suffer from limited reconstruction quality depending on their assumed statistical model. Learning-based SR approaches recover missing high-frequency information in an image through matching using a large image database of training sets. The drawbacks of these methods is the requirement of large representative training sets that are targeted toward example-based and specific applications. For example, the MAP-based SR approach in Reference [37] requires a total of approximately 395 ×10⁶ multiplication and addition operations to estimate an HR image with 256 × 256 pixels from 16 LR frames of size 64 × 64 pixels with translational motion and noise. Also, for the same problem, the FR-based SR method in Reference [42] requires approximately a total of 200 × 10⁶ multiplication and addition operations. Additionally, the inherently faster non-iterative FI-based SR approach in Reference [43], with parameters set as W_x = W_y = 12, D_x = D_y = 4, and ρ = 0.75, requires 72 × 10⁶ multiplication and addition operations but suffers from a limited reconstruction quality.

Figure 9.6 compares the visual quality of the HR estimates generated using various SR solutions described in References [37], [42], and [43]. As it can be seen, the MAP-based [37] and the FR-based [42] result in a noticeably sharper image and approximately 2 dB increase in PSNR compared to the FI-based SR approach [43]. Given these results, the following sections present a new trend of efficient selective quality-preserving SR techniques that are driven by perceptual modeling of low-level features.

9.5.1 Selective Perceptual Super-Resolution Framework

A block diagram of the SELP-SR estimation framework is shown in Figure 9.9. Initially, a rough estimate z₀ of the high-resolution image is obtained by either interpolating one of the LR images in the observed sequence or by fusing all the LR frames on one HR grid (also referred to as the shift-and-add technique). Other techniques, such as learning-based approaches [31], [32], [33], can also be used to produce initial estimates. At each iteration and for each estimated HR image, active pixels are detected based on the human visual detection model described in Section 9.5.2. Then, only these perceptually active pixels are updated by the SR estimation algorithm, which is based generally on minimizing an SR cost function. As a result, at each iteration, only a subset of pixels (that is, the active pixels) is selected for the SR processing phase. The algorithm stops whenever the change between the current HR estimate and the previous one is less than a small number e, or whenever the total number of specified iterations is reached. Only the selected perceptually active pixels need to be included in computing the change in the estimated SR frames between iterations; the other pixels remain unchanged. The SELP SR framework is very flexible in that any iterative SR estimation algorithm can be easily integrated in the SELP system.

9.5.2 Perceptual Contrast Sensitivity Threshold Model

Neurons in the primary visual cortex of the human visual system (HVS) are sensitive to various stimuli of low-level features in a scene, such as color, orientation, contrast, and luminance intensity [60]. The luminance sensitivity also referred to as light adaptation is the discrimination of luminance variations at every light level. Moreover, the contrast sensitivity is the response to local variations of luminance to the surrounding luminance [60]. Limits on the human visual sensitivity to low-level stimuli, such as light and contrast, are converted to masking thresholds that are used in perceptual modeling. Masking thresholds are levels above which a human can start distinguishing a significant stimulus or distortion [61]. Thus, the human visual detection model discriminates between image components based on contrast sensitivity of local information to their surroundings. In References [12] and [13], the SELP SR scheme attempts to exclude less significant information from SR processing by exploiting the masking properties of the human visual system through generating contrast sensitivity detection thresholds. The contrast sensitivity threshold is the measure of the smallest contrast, or the so-called just noticeable difference (JND), that yields a visible signal over a uniform background.

Digital natural images can be represented using linear weighted combinations of cosine functions through the discrete cosine transform (DCT). This is exploited in the lossy JPEG standard and many other image compression algorithms, including lossless ones. The models presented in References [62] and [63] derive contrast sensitivity thresholds and contrast masking thresholds for natural images in the DCT domain. The contrast sensitivity model considers the screen resolution, the viewing distance, the minimum display luminance L_min, and the maximum display luminance L_max. The contrast sensitivity thresholds are computed locally in the spatial domain using a sliding window of size N_blk × N_blk. The obtained thresholds per block will be used to select the pixels to be super-resolved for each HR estimate. The model, described in this section, involves first computing the contrast sensitivity threshold t₁₂₈ for a uniform block having a mean grayscale value equals to 128, and then obtaining the threshold for any block having arbitrary mean intensity using the approximation model presented in Reference [63].

The contrast sensitivity threshold t₁₂₈ of a block in the spatial domain is computed as

t128=TMgLmax−Lmin,⁢(9.14)

where M_g is the total number of grayscale levels (that is, M_g = 255 for eight-bit images), and L_min and L_max are the minimum and maximum display luminances, respectively. The threshold luminance T is evaluated based on the parametric model derived in Reference [62] using a parabolic approximation, where T = min(10^g0,1, 10^g1,0) and the terms g_0,1 and g_1,0 are defined as follows:

g0,1=log10Tmin+K(log10[ 1/(2Nblkwy) ]−log10[ fmin ])2,⁢(9.15)

g1,0=log10Tmin+K(log10[ 1/(2Nblkwy) ]−log10[ fmin ])2,⁢(9.16)

where w_x and w_y denote the horizontal width and vertical height of a pixel in degrees of visual angle, respectively. The term T_min denotes the luminance threshold at the frequency f_min, where the threshold is minimum, and K determines the steepness of the parabola. The parameters T_min, f_min, and K are the luminance-dependent parameters of the parabolic model and are computed as follows [62]:

Tmin={ (LLT)αTLTS0forL≤LT,L/S0forL>LT,⁢(9.17)

fmin={ f0(LLf)αfforL≤Lf,f0forL>Lf,⁢(9.18)

K={ K0(LLk)αKforL≤LK,K0forL>LK.⁢(9.19)

In Reference [62], the values of the constants in Equations 9.17 to 9.19 are set as L_T = 13.45cd/m², S₀ = 94.7, α_T = 0.649, α_f = 0.182, f₀ = 6.78 cycles/deg, L_f = 300cd/m², K₀ = 3.125, α_K = 0.0706, and L_K = 300cd/m². For a background value of 128, the local background luminance is computed as

L=Lmin+128Lmax−LminMg,⁢(9.20)

where L_min and L_max denote the minumum and maximum luminance of the display, respectively. Once the threshold t₁₂₈ at a grayscale value 128 is calculated using Equation 9.14, the just noticeable difference (JND) thresholds for the other grayscale values are approximated using a power function [63] as follows:

tJND=t128(∑n1=0Nblk−1∑n2=0Nblk−1In1,n2Nblk2(128))aT,⁢(9.21)

where I_{n1, n2} is the intensity level at pixel location (n₁, n₂) and a_T is a correction exponent that controls the degree to which luminance masking occurs and is set to α_T = 0.649, as given in Reference [63]. Note that if the block has a mean of 128, then t_JND of Equation 9.21 reduces to t₁₂₈ as expected.

A DELL UltraSharp 1905 FP liquid crystal display is used to show the images. For a screen resolution of 1280 × 1024, and for a measured luminance of L_min = 0cd/m² and L_max = 175cd/m², the threshold t₁₂₈ is computed to be equal to 3.3092 for N_blk = 8.

9.5.3 Perceptually Significant Pixels Detection

The perceptual mask determines the significant pixels that need to be processed at each iteration of the SR algorithm. For each estimated HR image, the perceptual mask is generated based on comparisons with the computed JND thresholds t_JND (Section 9.5.2) over a local block of size 8 × 8 pixels. Note that t₁₂₈ is computed only once according to Equation 9.14 since it is a constant. Also, for all eight-bit images, the remaining t_JND values, corresponding to the 255 possible mean intensity values (other than 128), can be precomputed and stored in a lookup table (LUT) in memory. For each image block, the mean of the block is computed and the corresponding t_JND is simply retrieved from the LUT.

As shown in Figure 9.10, after t_JND is obtained, the center pixel of a sliding 3 ×3 window is compared to its four cardinal neighbors. If any absolute difference between the current center pixel and any of its four cardinal neighbors is greater than t_JND, then the corresponding pixel mask is set to one indicating an active pixel. The remaining pixel mask locations are set to zero corresponding to non-active pixels. The sliding window will scan all the pixels of the HR image estimate to generate the perceptual mask. In general, the active locations consist of pixels in regions where edges are visible to the HVS. As a result, the perceptual mask signals a subset of pixels that are perceptually relevant for SR processing, resulting in significant computational savings. The SELP framework saves a significant number of computations in the SR processing stage with minimal overhead of operations per pixel for generating the perceptual mask. Figure 9.11 shows an example of the perceptually active pixels that are obtained by applying the SELP mask to a blurred and noisy version of the 512 × 512 Monarch image.

9.6.1 Attentive-Selective Perceptual Super-Resolution Framework

A block diagram of the AT-SELP SR estimation framework is shown in Figure 9.12. Similarly as in Figure 9.9, at each iteration and for each estimated HR image, active pixels are detected based on the human visual detection model described in Sections 9.5.2 and 9.6.2. Then, only these perceptually active pixels are updated by the SR estimation algorithm which is generally based on minimizing an SR cost function as in Equation 9.10. As a result, at each iteration, only a subset of pixels (that is, the active pixels) is selected for the SR processing phase.

The first phase of the SR algorithm processes the perceptually active pixels determined by a contrast sensitivity mask M_p, as explained in Section 9.5.3. Then, in the second phase of the SR estimation, only the subset of active pixels that is determined to be salient by the selective attention mask M_a is further iterated upon. The process of updating the HR estimates of the perceptual/attentive active pixels continues until a maximum number of iterations is reached or the system stabilizes; that is, until |M_a·(z_n+1 − z_n)|/|M_a·z_n| < ε in the attentive active region and |M_p·(z_n+1 − z_n)|/|M_p·z_n| < s·ε in the perceptual non-attentive active region. Here, s is a scaling factor greater than 1 and ε is a predetermined threshold that represents the desired accuracy or quality of the SR algorithm. Only the selected perceptually active and salient attentive pixels need to be included in computing the change in the estimated SR frames between iterations; the other pixels do not change their values. It is necessary in the first phase of the algorithm to super-resolve the perceptually significant information of the non-attended regions to a certain acceptable quality level (s.ε) that will not attract and bias the HVS perception of the background quality, thus, leading to an acceptable homogeneous quality of the entire image. As a result, the salient regions are reconstructed with a higher visual acuity while maintaining a trade-off between smoothing the flat regions dominated by noise and sharpening the perceptually relevant edges.

9.6.2 Saliency-Based Mask Detection

Any saliency-based visual attention model, such as those in References [68], [69], and [70], can be adopted to detect the attentive mask M_a in the AT-SELP SR framework to further reduce the set of active pixels selected by the contrast sensitivity detector, thus reducing the computational complexity of the SR algorithms. Saliency maps S combine several low-level image features that compete to attract the human attention, providing measures of the level of attention at every point in the visual scene. After computing the saliency information at each point in the scene, the attentive mask M_a is generated by choosing the pixel locations corresponding to the highest τ% of the saliency map values. Existing saliency map generation techniques inspired by a hierarchical visual attention model (IT) [68], a foveated gaze attentive model (GAF) [69], and a frequency-tuned attention model (FT) [70] are detailed in this section for further comparisons and analysis in the AT-SELP SR framework.

9.7.1 Perceptual MAP-Based SR Estimation

Existing Bayesian MAP-based SR estimators present high-quality estimation but suffer from high computational requirements [36], [37], [38], [39]. In order to illustrate the significant reduction in computations for MAP SR techniques, the popular gradient-descent optimization-based algorithm from Reference [37] is integrated into the attentive-perceptual selective SR framework (SELP, AT-SELP).

In Reference [37], a uniform detector sensitivity is assumed over the span of the detector degradation model, then the point spread function (PSF) H in the observation model (Section 9.3) is represented using an averaging filter. Following the Bayesian MAP-based SR formulation in Section 9.4.1, the regularization smoothness constraint term (in the cost function of Equation 9.9) is represented in Reference [37] by Γ(z)=12zTCz−1z, where C_z is the covariance of the HR image prior model imposing a piecewise smoothness relationship between neighboring pixels in z as follows [37]:

Cz−1=1λ∑i=1Ndi,k(∑j=1Ndi,jzj),⁢(9.32)

where λ is scaling factor controlling the effect of rapidly changing features in z, and z_j is the pixel at the jth location of the lexicographically ordered vector z. The coefficients d_i,j, which express a priori assumption about the relationship between neighboring pixels in z, are defined as follows [37]:

di,j={ 1fori=j,−1/4fori≠j:zj∈cardinalneighborsofzi⁢(9.33)

In the AT-SELP-MAP SR scheme, the gradient descent minimization procedure is applied selectively and only to pixels that are determined to be perceptually significant by the attentive-perceptual detectors (Sections 9.5.3 and 9.6.2). Following the steepest descent solution in the perceptually driven framework, the HR image is estimated using Equation 9.8 as follows [37]:

z^n+1=​z^n−βnM{ 1ση2∑k=1KWT(Wzn−y)​+12Cz−1zn },⁢(9.34)

where M represents the attentive-perceptual mask for selecting active pixels at every iteration of the SR process. Note that the initial HR image estimate is an interpolated version of one of the LR frames. The elements in M take binary values, where ones indicate active pixels that should be included in the SR update at the current iteration, whereas zero mask values indicate that the corresponding pixels are non-active and will not be processed in the current iteration.

9.7.2 Perceptual Fusion-Restoration SR Estimation

The efficient perceptually driven framework can be used in conjunction with the fusion-restoration (FR) SR algorithms. Reference [42] proposed a two-step algorithm by using first a non-iterative data fusion step followed by an iterative gradient-descent deblurring-interpolation step. This algorithm models the relative motion between low-resolution frames as translational and the point spread function (PSF) as an L₁ × L₂ Gaussian lowpass filter with a standard deviation equals to 1. Following the regularized-norm SR formulation in Section 9.4.2, a fast fusion-restoration implementation of the minimization solution (Equation 9.12) can be achieved by solving for a blurred estimate zb=Hz^ of the HR image followed by an interpolation and deblurring iterative step. An initial blurred version of the HR estimate z_b is produced in the data fusion step by registration followed by a median operator of the LR frames on the HR grid, referred to as the “median shift and add” operator. As for the regularization term in Equation 9.12, a bilateral total variation regularization that preserves edges is adopted as follows [42]:

ΓBTV(z)=∑l=−RR∑m=0Rα| m |+| l |‖ z−SxlSymz ‖1,⁢(9.35)

where Sxl and Sym shift the HR image z by l and m pixels in the horizontal and vertical directions, respectively, and R ≥ 1 represents several scales of shifting values. The weight α is applied as a decaying factor for convergence purposes, and is chosen between 0 ≤ α ≤ 1. The SR problem in Equation 9.12 reduces to deblurring and interpolating for the missing pixels in the initial HR estimate z_b that is formulated as a regularized l₁-norm minimization problem [42]. In the AT-SELP-FR SR scheme, the steepest descent minimization is applied selectively only to the active pixels identified by the perceptual detection schemes:

z^n+1=z^n−βnM[ HTATsign(AHz^n−Azb)+ ]λ∑l=−RR∑m=0Rα| m |+| l |[ I−Sy−mSx−1 ]sign(z^n−SxlSymz^n)],

where β_n is the step size in the direction of the gradient and λ is a regularization weighting factor. Matrix A is an N × N diagonal matrix with diagonal values equal to the square root of the number of measurements that contribute to make each element of z_b. Also, Sx−l and Sy−m define a shifting effect in the opposite directions of Sxl and Sym. The term M represents the perceptually attentive masking that selects the active pixels that are processed at each iteration, thus reducing the computations required in Reference [42].

9.7.3 Perceptually Driven SR Results

The performance of the efficient perceptually driven SR framework is assessed using a set of test images shown in Figure 9.14. The images used in this experiment include Cameraman, Lena, and Clock from the USC image database [71], and Monarch from the LIVE image database [72], as shown in Figure 9.14. These images, all of the size of 256 × 256 pixels, differ in their characteristics. For example, the Clock and Monarch images contain many smooth regions, while the Cameraman and Lena images have more edges and texture variations. Moreover, the Lena image can be used to demonstrate the application of SR in face recognition applications.

A sequence of LR images is generated from a single HR image by passing the HR image through the SR degradation model described in Section 9.3 to generate a sequence of blurred, shifted, and noisy LR images. Then, for a resolution factor of four, the degradation process is applied by randomly shifting the reference 256 × 256 HR image in the horizontal and vertical directions, blurring the shifted images with a lowpass filter of size 4 × 4, and subsampling the result by a factor of four in each direction to generate sixteen 64 × 64 LR frames. The blur filters are modeled as an average filter and as a Gaussian filter with standard deviation of one for the MAP-based [37] and the FR-based [42] SR observation models, respectively. Then, an additive Gaussian noise of variance ση2 = 16 is added to the resulting LR sequence. Figure 9.3 depicts the simulated sequence generation process.

The attentive-selective perceptual MAP-based super-resolution and attentive-selective perceptual fusion-restoration super-resolution schemes referred to as AT-SELP-MAP [15] and AT-SELP-FR [14], respectively, are compared with their existing non-selective counterparts MAP-SR [37] and FR-SR [42], as well as the selective SR counterparts SELP-MAP and SELP-FR [12], [13]. The simulation parameters of the compared MAP-based SR methods are set to λ = 100, ε = 0.0001, and s = 100, and a maximum of twenty iterations are performed. On the other hand, the simulation parameters of the compared FR-based SR methods are set to R = 2, α = 0.6, λ = 0.08, and β = 8, ε = 0.0001, s = 120, and a maximum of thirty iterations are performed. The parameter τ of the saliency map detection is set to 20% to identify the attentive active pixels. Experimentally, percentage values between 20 and 40% offered promising results in terms of quality and computational efficiency trade-off. These numbers are plausible, since shifts of attention based on competing saliency results needs 30 to 70 ms [73]. Then, in a 30.33 ms time interval given to view one frame, assuming 30 fps video sequence, one can only perceive 20 to 40% of the salient regions. For the saliency mask detection schemes, different saliency maps generated from existing visual attention models presented in References [68], [69], [70] are also integrated and tested in the proposed AT-SELP SR framework.

Figure 9.15 gives a quantitative comparison among the baseline MAP, SELP-MAP, and proposed AT-SELP-MAP SR methods using different saliency detection techniques applied to super-resolve the ninth frame of the simulated Cameraman LR sequence. Similarly, Figure 9.16 shows a quantitative comparison among the baseline FR, SELP-FR, and proposed AT-SELP-FR SR methods. Error measures, represented by the SNR gain per iteration, and the computational complexity, signified by the number of processed pixels per iteration, are presented in Figures 9.15, 9.16a, and 9.16b, respectively. It can be seen that in the case of baseline MAP and FR SR methods all the pixels are processed at each iteration for all the images (that is, for a 256 × 256 image, the total number of processed pixels is 65536 at each iteration). As shown in Figures 9.15b and 9.16b for the SELP-MAP and SELPFR SR methods, respectively, the number of processed pixels per iteration varies from one image to the other depending on the visual content. In Figures 9.15b and 9.16b, the visual attention processing takes effect around the fifth iteration, thus further reducing the detected active pixels processed. Due to the attentive selectivity, the AT-SELP SR framework presents considerable savings in terms of the number of processed pixels, leading to a significant reduction in computational complexity. Comparing the SNR measures in Figures 9.15a and 9.16a, it can be easily seen that the M_a(IT) mask integrated into the ATSELP SR framework has the best error performance among the other implemented saliency mask detectors. Also, the efficient and simple saliency mask generator M_a(FT) adopted from Reference [70] does not enhance the overall quality of the SR estimate due to selecting objects and features in the image that are not relevant to SR processing applications. Furthermore, the overall quantitative assessment is also shown in terms of PSNR measures in Tables 9.1 and 9.2.

Since the employed PSNR and SNR gain measures do not necessarily reflect the resulting visual quality [74], Figures 9.17 and 9.18 show the SR outputs obtained for the ninth frame of the simulated Cameraman sequence for all compared MAP-based and FR-based SR methods, respectively. Despite the fact that the AT-SELP-SR scheme processes significantly less pixels per iteration, it produces a comparable visual quality to the existing selective and non-selective [37], [42] SR schemes. Furthermore, Tables 9.3 and 9.4 show similar pixel savings achieved by the different visual attention methods when integrated in the AT-SELP SR framework. It is shown in Table 9.3 that up to 66% in pixel savings can be achieved by the AT-SELP-MAP SR over the baseline MAP SR, while around 31% in pixel savings is achieved over the efficient SELP-MAP SR. Table 9.4 shows that the proposed AT-SELP-FR framework saves around 71% in SR processing over the non-selective baseline-FR and around 44% in SR processing over the efficient SELP-FR SR.

TABLE 9.1
PSNR values in dBs of MAP-based SR methods for all test sequences using magnification factor L = 4 and noise variance ση2 = 16

TABLE 9.2
PSNR values in dBs of FR-based SR methods for all test sequences using magnification factor L = 4 and noise variance ση2 = 16.