1.4.1.1 Super-resolution Approach

Super-resolution approaches infer a high-resolution image from low-resolution data using a pre-learnt model that relates both representations [17]. Even though the majority of the works focus on improving data quality, some approaches have also tried to boost biometric recognition performance [18–20]. Ben-Ezra et al. [21] presented a Jitter Camera that exploits a micro actuator for enhancing the resolution provided by a low-resolution camera. Fig. 1.4 depicts the results attained for a low-resolution frame acquired in a surveillance scenario. Despite these improvements, it is commonly accepted that these approaches have to be extended to more realistic scenarios as described in [22]. This becomes particularly evident when trying to use such approaches to biometric recognition at-a-distance.

1.4.1.2 Gait Recognition

The way humans walk can also be used for identification purposes and is usually known as gait recognition [23,24]. The advantages of gait can be summarized in the following: (i) it can be easily measured at-a-distance, (ii) it is difficult to disguise or occlude, and (iii) it is robust to low-resolution images. Moreover, a recent study about the covariate factors affecting recognition performance has found that gait is time-invariant in the short and medium term, thus gaining special attention among reliable biometric traits. On the other hand, gait strongly depends on the control over clothing and footwear [25], which negatively impacts its feasibility in surveillance scenarios.

Notwithstanding, many methods have been introduced in the literature to optimize gait recognition systems. Ran et al. [26] used human walking to segment and label body parts that can be helpful to perform real-time recognition. Gait patterns were captured and stacked in a 3D data cube containing all possible deformations. The symmetries between the patterns were analyzed in order to measure all possible changes and correctly label different parts of the human body.

Venkat and De Wilde [27] faced the problem of low-resolution in video data focusing on potential information from sub-gaits that can contribute to the recognition system. Moustakas et al. [28] exploited the height and stride length as soft biometric traits. These features were combined in a probabilistic framework to accurately perform gait recognition system.

Conversely, Jung et al. [29] exploited gait to estimate the head pose in surveillance scenarios. In this approach, a 3D face model was also inferred to improve recognition performance.

Choudhury and Tjahjadi [30] exploited human silhouettes extracted from a gait system to perform recognition. By analyzing the shape of the contours, they were able to overcome some typical side effects introduced by the presence of noise in the gait recognition system. Considering that this strategy is highly dependent on clothing, the authors introduced an extended version of the previous work [31] handling occlusion factors related to variations of view, subject's clothing and the presence of a carried item. Nevertheless, the system requires the availability of a matching template for any possible view of interest.

Kusakunniran [32] proposed a recognition model that directly extracts gait features by from raw video sequences on a spatio-temporal feature domain. They introduced the space-time interest points (STIPs) that represent a point of interest of a dominant walking pattern, which is used to represent the characteristics of each individual gait. The advantage of this method is that it does not require any pre-processing of the video stream (e.g., background subtraction, edge detection, human silhouettes, and so on). This makes the proposed method robust to partial occlusion caused by, among others, carrying items or hair/clothes/footwear variations over time.

1.4.3.1 Typical Design of Master–Slave Systems

In Fig. 1.5, an overview of a typical video surveillance system aimed at biometric recognition is depicted. Such a system is a generalization of the face recognition system proposed by Wheeler et al. [52], in which two cameras cooperate in a master–slave architecture for the tracking of an individual and for the cropping of the face to achieve biometric recognition. In master–slave architectures, the hardware usually consists of:

• A Wide Field of View camera (WFOV) that acts as a master. By providing a wide view of the scene, such cameras allow actions like the tracking of objects/persons and detection of events of interest.

• A Narrow Field of View camera (NFOV) that acts as a slave. This camera provides a narrowed view of the scene and allows focusing on a single element of the scene. If such cameras provide good resolution images, the acquisition of several biometric traits will be possible (face, ear, periocular area, in descending size order).

The WFOV is responsible of providing the view of the whole scene in which the system will operate. Since it is a stationary device, a background/foreground segmentation approach is applicable and thus detects moving objects in the scene. Intrinsic and extrinsic parameters of the camera have to be determined by means of a calibration procedure, so that a mapping with the real world coordinates is provided. As well as for the wide camera, a calibration procedure is also required for the NFOV camera. Firstly, it needs to be calibrated with the WFOV camera:

• The pan, tilt, and zoom values of the NFOV camera are set such that it is in its home position;

• A homography matrix is then estimated, by creating correspondences among the points in the wide scene and those in the narrow scene.

A further calibration is usually applied to the NFOV camera, in order to determine how the pan, tilt, and zoom values affect the field of view of the camera. The zoom point is calibrated in order to reduce the offset between several levels of zoom. The concept of a zoom point is introduced in [52] and indicates the pixel location that points at the same real world coordinates, even if the zoom factor is changing. Once the full calibration is accomplished, it is simple to determine the pan, tilt, and zoom values of the NFOV from the region of interest in the WFOV.

Since multiple subjects/objects may be detected and tracked in the WFOV video, a Target Scheduler module is needed in order to keep trace of the position of the targets (Target Records) in the video and their current state. The scheduler passes the information regarding the position of the target to a PTZ controller that calculates the PTZ values and zooms-in on the detected target (the zoom value may vary depending on the resolution of the video and on the size of the biometric trait). Once the biometric trait is cropped, the Recognition Module handles the recognition activities (segmentation of the trait, feature extraction, matching). If a trait matches one present in the gallery, an ID number is associated, and the Target Records dataset is updated.

1.4.3.2 Systems Based on Logical Alignment of the Cameras

In Section 1.4.2, we described multiple surveillance systems designed to face the issues of the calibration between pairs of cameras. Ideally, if two identical cameras were mounted at the same point so that they could collect the same view of the scene, a calibration between them would not be necessary. Even though this configuration is not possible, the use of beam splitter² can mimic this process and ease the calibration between the static and PTZ cameras. Solutions that use a beam splitter [64] are perfect examples of how to approach the problem of low-calibration constraints in multi-camera systems. To better understand how the beam splitter works and how a multi-camera system can be configured, see the schematic view in Fig. 1.6.

A particularly interesting approach that relaxes the constraints of calibration was presented by Park et al. [9]. They proposed different multi-camera systems that indirectly solve the problem of sharing the same view between two cameras. In this approach, designated as coaxial–concentric configuration, the cameras are mounted in a way that they are all logically aligned along a shared view-axis. Therefore, it overcomes the problems related to the epipolar geometry (for details, refer to Section 1.3.4). A picture of the system proposed by Park et al. [9] is shown in Fig. 1.7.

In the system of Park et al. [9] (Fig. 1.7), the multi-camera consisted of a hexahedral dark box with one of its sides tilted by 45 degrees and attached to a beam splitter. PTZ camera was configured inside the dark box and the static camera was placed outside the box. The incident beam was split at the beam splitter and captured by PTZ and static cameras to provide almost the same image to both of the cameras. All the camera axes were effectively parallel in this configuration (it enables the use of a single static camera to estimate the pan and tilt parameters of the PTZ camera). It is worth noting that such a system ensures a high level of matching between the two camera streams. However, the field of view does not completely overlap due to different camera lens and optics. As such, the authors introduced a calibration method for estimating with minimum error the pan-tilt parameters of the PTZ camera after a user-assisted one-time parametrization.

A similar solution was presented by Yoo et al. [65], where the wide-view and narrow-view cameras were combined with a beam splitter to simultaneously acquire facial and iris images. The authors combined two sensors (an image sensor for face and infra-red sensor for irises) with a beam splitter. The integrated dual-sensor system was therefore able to map rays to same position in both camera sensors, thus avoiding excessive calibration and the need of depth information.

Compared to other camera systems proposed in the literature, the approaches based on a logical alignment of the cameras feature interesting advantages:

1. World coordinates and their matching between pairs of camera streams are not involved in the calibration process;

2. Just a simple calibration, which mainly consists in a visual alignment between camera streams, is required;

3. The calibrated system can be easily deployed at a different location with no need of re-calibration.

Moreover, these approaches were already demonstrated to be feasible for human recognition at-a-distance (rank-1 face recognition accuracy of 91.5% in case of single person tracking with a probe set of 50 subjects against a notably larger gallery set of 10.050 subjects). However, the strict configuration required having the camera focal points aligned which might represent a limitation of the proposed approach in some video surveillance scenarios since the dimensions of the system inhibit its deployment in outdoor scenarios.

1.4.3.3 QUIS–CAMPI System

Recently, Neves et al. [10,66] have introduced an alternative solution to extend PTZ-assisted facial recognition to surveillance scenarios. The authors proposed a novel calibration algorithm [10,67] capable of accurately estimating pan-tilt parameters without resorting to intermediate zoom states, multiple optical devices or highly stringent configurations. This approach exploits geometric cues, i.e., the vanishing points available in the scene, to automatically estimate subjects height (h) and thus determine their 3D position. Furthermore, the authors have built on the work of Lv et al. [68] to ensure robustness against human shape variability during walking.

The proposed surveillance system is divided into five major modules, broadly grouped into three main phases: (i) human motion analysis, (ii) inter-camera calibration, and (iii) camera scheduling. The workflow chart of the surveillance system used for acquiring the QUIS–CAMPI dataset is given in Fig. 1.8 and described in detail afterwards.

The master camera is responsible for covering the whole surveillance area (about 650 m²) and for detecting and tracking subjects in the scene, so that it can provide to the PTZ camera a set of facial regions. In the calibration phase, the coordinates $(x_i(t),y_i(t))$ of the ith subject in the scene need to be converted to the correspondent pan-tilt angle. However, 3D positioning is required, which involves solving the following underdetermined equation:

$\lambda \left(\begin{array}{c}\hfill x_i\hfill \\ \hfill y_i\hfill \\ \hfill 1\hfill \end{array}\right)=\underset{:={\mathbf{P}}_{\mathbf{m}}}{\underbrace{{\mathbf{K}}_{\mathbf{m}}[{\mathbf{R}}_{\mathbf{m}}|{\mathbf{T}}_{\mathbf{m}}]}}\left(\begin{array}{c}\hfill X\hfill \\ \hfill Y\hfill \\ \hfill Z\hfill \\ \hfill 1\hfill \end{array}\right),$

where ${\mathbf{K}}_{\mathbf{m}}$ and $[{\mathbf{R}}_{\mathbf{m}}|{\mathbf{T}}_{\mathbf{m}}]$ denote the intrinsic and extrinsic matrices of the master camera, whereas ${\mathbf{P}}_{\mathbf{m}}$ represents the camera matrix.

To address this ambiguity, the existing systems either relied on highly stringent camera disposals [52,54] or on multiple optical devices [9]. In contrast, the authors introduced a novel calibration algorithm [10] that exploited geometric cues, i.e., the vanishing points available in the scene, to automatically estimate subjects' height and thus determine their 3D position. By considering the ground as the XY plane, the Z coordinate equals the subject height, and therefore, Eq. (1.2) can be rearranged as

$\lambda \left(\begin{array}{c}\hfill x_i\hfill \\ \hfill y_i\hfill \\ \hfill 1\hfill \end{array}\right)=[{\mathbf{p}}_1{\mathbf{p}}_{2}h{\mathbf{p}}_3+{\mathbf{p}}_4]\left(\begin{array}{c}\hfill X\hfill \\ \hfill Y\hfill \\ \hfill 1\hfill \end{array}\right),$

where ${\mathbf{p}}_i$ is the set of column vectors of the projection matrix ${\mathbf{P}}_{\mathbf{m}}$.

The corresponding 3D position in the PTZ referential can then be calculated using the extrinsic parameters of the camera as

$\left(\begin{array}{c}\hfill X_p\hfill \\ \hfill Y_p\hfill \\ \hfill Z_p\hfill \end{array}\right)=[{\mathbf{R}}_{\mathbf{p}}|{\mathbf{T}}_{\mathbf{p}}]\left(\begin{array}{c}\hfill X\hfill \\ \hfill Y\hfill \\ \hfill Z\hfill \\ \hfill 1\hfill \end{array}\right).$

The corresponding pan and tilt angles are given by

${\theta }_p=\arctan \left(\frac{X_p^{\prime }}{Z_p^{\prime }}\right)$

and

${\theta }_t=\arcsin \left(\frac{Y_p^{\prime }}{\sqrt{{(X_p^{\prime })}^2+{(Y_p^{\prime })}^2+{(Z_p^{\prime })}^2}}\right).$

When multiple targets are available in the scene, a camera scheduling approach determines the sequence of observations that minimizes the cumulative transition time, in order to start the acquisition process as soon as possible and maximize the number of samples taken from the subjects in the scene. Considering that this problem has no known solution that runs in polynomial time, the authors have introduced a method capable of inferring an approximate solution in real-time [69].

1.4.3.4 Other Biometrics: Iris, Periocular, and Ear

Commercial iris recognition systems can identify subjects with extremely low error rates. However, they rely on highly restrictive capture volumes, reducing their workability in less constrained scenarios. In the last years, different works have attempted to relax the constraints of iris recognition systems by exploiting innovative strategies to increase both the capture volume and the stand-off distance, i.e., the distance between the front of the lens and the subject. Successful identification of humans using iris is greatly dependent on the quality of the iris image. To be considered of acceptable quality, the standards recommend a resolution of 200 pixels across the iris (ISO/IEC 2004), and an in-focus image. Also, sufficient near infra-red (IR) illumination should be ensured (more than 2 mW/cm²) without harming human health (less than 10 mW/cm² according to the international safety standard IEC-60852-1). The volume of space in front of the acquisition system where all these constraints are satisfied is denoted as the capture volume of the system. Considering all these constraints, the design of an acquisition framework capable of acquiring good quality iris images in unconstrained scenarios is extremely hard, particularly at large stand-off distances. This section reviews the most relevant works and acquisition protocols for iris and periocular recognition at-a-distance.

In general, two strategies can be used to image iris in less constrained scenarios: (i) the use of typical cameras and (ii) the use of magnification devices. In the former, the Iris-on-the-Move system is notorious for having significantly decreased the cooperation in image acquisition. Iris images are acquired on-the-move while subjects walk through a portal equipped with NIR illuminators. Another example of a widely used commercial device is the LG IrisAccess4000. Image acquisition is performed at-a-distance; however, the user has to be directed to an optimal position so that the system can acquire an in-focus iris image. The need for fine adjustment of the user position arises from the limited capture volume of the system.

Considering the reduced size of periocular region and iris, several approaches have exploited magnification devices, such as PTZ cameras, which permit extending the system stand-off distance while maintaining the necessary resolution for reliable iris recognition. Wheeler et al. [58] introduced a system to acquire iris at a resolution of 200 pixels from cooperative subjects at 1.5 m using a PTZ camera assisted by two wide view cameras. Dong et al. [70] also proposed a PTZ-based system, and due to a higher resolution of the camera they were capable of imaging iris at a distance of 3 m with more than 150 pixels. As an alternative, Yoon et al. [59] relied on a light stripe to determine 3D position, avoiding the use of an extra-wide camera. The eagle eye system [60] uses one wide-view camera and three close-view cameras for capturing the facial region and the two irises. This system uses multiple cameras with hierarchically-ordered field of view, a highly precise pan-tilt unit, and a long focal length zoom lens. It is one of a few example systems that can perform iris recognition at a large stand-off distance (3–6 m). Experimental tests show good acquisition quality for single stationary subjects of both face and irises. On the other hand, the average acquisition time is 6.1 s which does not match with the requirements of real-time processing in non-cooperative scenarios.

Regarding periocular recognition at-a-distance, few works have been developed. In general, the periocular region is significantly less dependent on face distortions (i.e., neutral expression, smiling expression, closed eyes, and facial occlusions) than the whole face for recognition across all kinds of unconstrained scenarios. The work by Juefei-Xu and Savvides [62] is considered the only notable proposal to perform periocular recognition in highly unconstrained environments. The authors utilized the 3D generic elastic models (GEMs) [71] to correct the off-angle pose to recognize non-cooperative subjects. To deal with illumination changes, they exploited a parallelized implementation of the anisotropic diffusion image preprocessing algorithm running on GPUs to achieve real-time processing time. In their experimental analysis, they reported a verification rate of 60.7% (in the presence of facial expression and occlusions) but, more notably, they attained a 16.9% performance boost over the full face approach. Notwithstanding the encouraging results achieved, the periocular region at-a-distance still represents an unexplored field of research. The same holds for ear recognition. Ear is another interesting small biometric that has been proved relatively stable and has drawn researchers' attention recently [72]. However, like other similar biometrics (e.g., iris and periocular), it is particularly hard to be managed in uncontrolled and non-cooperative environments. Currently, the recognition of human ears, with particular regard to challenges of at-a-distance scenarios, has not been faced yet, thus representing a promising and uncharted field of research which could reserve interesting opportunities and achievements in the recent future.