Visual perception is a complex process requiring interaction between the receptors in the eye that sense the stimulus and the neural system and the brain that are responsible for communicating and interpreting the sensed visual information. This process involves several physical, neural, and cognitive phenomena whose understanding is essential to design effective and computationally efficient imaging solutions. Building on the research advances in computer vision, image and video processing, neuroscience, and information engineering, perceptual digital imaging has become an important and rapidly developing research field. It greatly enhances the capabilities of traditional imaging methods, and numerous commercial products capitalizing on its principles have already appeared in divergent market applications, including emerging digital photography, visual communication, multimedia, and digital entertainment applications.

The purpose of this book is to fill the existing gap in the literature and comprehensively cover the system design, implementation, and application aspects of perceptual digital imaging. Because of the rapid developments in specialized imaging areas, the book is a contributed volume where well-known experts are dealing with specific research and application problems. It presents the state-of-the-art as well as the most recent trends in image acquisition, processing, storage, display, and visual quality evaluation. The book serves the needs of different readers at different levels; it can be used as textbook in support of graduate courses in computer vision, digital imaging, visual data processing, computer graphics, and visual communication, or as stand-alone reference for graduate students, researchers, and practitioners.

This book provides a strong, fundamental understanding of theory and methods, and a foundation on which solutions for many of today’s most interesting and challenging imaging problems can be built. It details recent advances in the field and explores human visual system-driven approaches across a broad spectrum of applications, including image quality and aesthetics assessment, digital camera imaging, white balancing and color enhancement, thumbnail generation, image restoration, super-resolution imaging, digital halftoning and dithering, color feature extraction, semantic image analysis and multimedia, video shot characterization, image and video encryption, display quality enhancement, and more.

The book begins by focusing on human visual perception. The human visual system can be subdivided into two major components, that is, the eyes, which capture light and convert it into signals that can be understood by the nervous system, and the visual pathways in the brain, along which these signals are transmitted and processed. Chapter 1 discusses characteristics of human vision, focusing on the anatomy and physiology of the above components as well as a number of phenomena of visual perception that are of particular relevance to digital imaging.

As motion is ubiquitous in normal viewing conditions, it is essential to analyze the effects of various sources of movement on the retinotopic representation of the environment. Chapter 2 deals with an analysis of human visual perception based on real-time constraints of ecological vision, considering two inter-related problems of motion blur and moving ghosts. A model of retino-cortical dynamics is described in order to provide a mathematical framework for dealing with motion blur in human vision.

Chapter 3 addresses important issues of perceptual image and video quality assessment. Built on the knowledge on perception of images and videos by humans and refined computational models of visual processing, a number of assessment methods capable of producing the quality scores can be designed. Although human qualitative opinion represents the palatability of visual signals, subjective quality assessment is usually time consuming and impractical. Thus, a more efficient approach is to design the algorithms that can objectively evaluate visual quality by automatically generating the quality scores that correlate well with subjective opinion.

Chapter 4 focuses on visual aesthetic quality assessment of digital images. Computational aesthetics is concerned with exploring techniques to predict an emotional response to a visual stimulus and with developing methods to create and enhance pleasing impressions. Among various modules in the aesthetic algorithm design, such as data collection and human study, feature extraction, and machine learning, constructing and extracting the features using the knowledge and experience in visual psychology, photography, and art is essential to overcome the gap between low-level image properties and high-level human perception of aesthetics.

The human visual system characteristics are also widely considered in the digital imaging technology design. As discussed in Chapter 5, digital camera designers largely rely on perceptually based image processing to ensure that a captured image mimics the scene and is visually pleasing. Perceptual considerations affect the decisions made by the automatic camera control algorithms that adjust the exposure, focus, and white balance settings of the camera. Various camera image processing steps, such as demosaicking, noise reduction, color rendering, edge enhancement, and compression, are similarly influenced in order to execute quickly without sacrificing perceptual quality.

Chapter 6 presents the framework that addresses the problem of joint white balancing and color enhancement. The framework takes advantage of pixel-adaptive processing that combines the local and global spectral characteristics of the captured visual data in order to produce the image with the desired color appearance. Various example solutions can be constructed within this framework by following simple but yet powerful spectral modeling and combinatorial principles. The presented design methodology is efficient, highly flexible, and leads to visually pleasing color images.

Taking advantage of their small size, thumbnails are commonly used in preview, organization, and retrieval of digital images. Perceptual thumbnail generation, explored in Chapter 7, aims to provide a faithful impression about the image content and quality. Unlike the conventional thumbnail, its perceptual counterpart displays both global composition and important visual features, such as noise and blur, of the original image. This allows the user to efficiently judge the image quality by viewing the low-resolution thumbnail instead of inspecting the original full-resolution image.

Chapter 8 reviews the principles of patch-based image models and explores their possible scientific connections with human vision models. The evolution from first-generation patch models, which relate to dictionary construction and learning, to second-generation patch models, which include structural clustering and sparsity optimization, offers insights on how locality and convexity have served in mathematical modeling of photographic images. The potential of patch-based image models is demonstrated in various image processing applications, such as denoising, compression artifact removal, and inverse halftoning.

Super-resolution imaging aims at producing a high-resolution image or a sequence of high-resolution images from a set of low-resolution images. The process requires an image acquisition model that relates a high-resolution image to multiple low-resolution images and involves solving the resulting inverse problem. Chapter 9 surveys existing relevant methods, with a focus on efficient perceptually driven super-resolution techniques. Such techniques utilize various models of the human visual system and can automatically adapt to local characteristics that are perceptually most relevant, thus producing the desired image quality and simultaneously reducing the computational complexity of processing.

Digital halftoning refers to the process of converting a continuous-tone image or photograph into a binary pattern of black and white pixels for display on binary devices, such as ink-jet printers. Similar to dithering used in computer graphics, this process creates the illusion of depth when outputting an image on a device with a limited palette. Chapter 10 discusses the methods of dither array construction employing models of visual perception, including the extension of the stochastic dither arrays to nonzero screen angles and the challenging problem of lenticular printing.

Color features are widely used in content analysis and retrieval. However, most of them show severe limitations due to their poor connection to the color perception mechanism of the human visual system and their inability to characterize all the properties of the color composition in a visual scenery. To overcome these drawbacks, Chapter 11 focuses on perceptual color descriptors that reflect all major properties of prominent colors. Extracted global and spatial properties using these refined descriptors can be combined further to form the final descriptor that is unbiased and robust to non-perceivable color elements in both spatial and color domains.

Exploiting information in the sense of visual semantics, context, and implicit or explicit knowledge not only allows for better scene understanding by bridging the semantic and conceptual gap that exists between humans and computers but also enhances content-based multimedia analysis and retrieval performance. To address this problem, Chapter 12 deals with concept-based multimedia processing using semantic and contextual knowledge. Such high-level concepts can be efficiently detected when an image is represented by a model vector with the aid of a visual thesaurus and visual context, where the latter can be interpreted by utilizing an ontology-based fuzzy representation of knowledge.

Chapter 13 presents perceptually driven video shot characterization, employing an un-supervised approach to identify meaningful components that influence the semantics of the scene through their behavioral and perceptual attributes. This is done by using the perceptual grouping and prominence principles. Namely, the former takes advantage of an organizational model that encapsulates the grouping criteria based on spatiotemporal consistency exhibited by emergent clusters of grouping primitives. The latter models the cognitive saliency of the subjects based on attributes that commonly influence human judgment. The video shot is categorized based on the observations that direct visual attention of a human observer across the visualization space.

With the proliferation of digital imaging devices, protecting sensitive visual information from unauthorized access and misuse becomes crucial. Given the extensive size of visual data, full encryption of digital images and videos may not be necessary or economical in some applications. Chapter 14 discusses perceptual encryption of digital images and videos that can be implemented by selectively encrypting part of the bitstream representing the visual data. Of particular interest are attacks on perceptual encryption schemes for popular image and video formats based on the discrete cosine transform.

Finally, Chapter 15 explores perceptual effects to exceed physical limitations of display devices. By considering various characteristics of human visual perception, display qualities can be significantly enhanced, for instance, in terms of perceived contrast and disparity, brightness, motion smoothness, color, and resolution. Similar enhancement could often be achieved only by improving physical parameters of displays, which might be impossible without fundamental design changes in the existing display technology and clearly may lead to overall higher display costs.

As the above overview suggests, this book is a unique up-to-date reference that should be found useful in the design and implementation of various digital imaging-related tasks. Moreover, each chapter offers a broad survey of the relevant literature, thus providing a good basis for further exploration of the presented topics. The book includes numerous examples and illustrations of perceptual digital imaging results, as well as tables summarizing the results of quantitative analysis studies. Complementary material for further reading is available online at http://www.colorimageprocessing.org.

I would like to thank the contributors for their effort, valuable time, and motivation to enhance the profession by providing material for a wide audience while still offering their individual research insights and opinions. I am very grateful for their enthusiastic support, timely response, and willingness to incorporate suggestions from me to improve the quality of contributions. Finally, a word of appreciation for CRC Press / Taylor & Francis for giving me the opportunity to edit a book on perceptual digital imaging. In particular, I would like to thank Nora Konopka for supporting this project, Jessica Vakili for coordinating the manuscript preparation, Jim McGovern for handling the final production, Andre Barnett for proofreading the book, and John Gandour for designing the book cover.

Rastislav Lukac
Foveon, Inc. / Sigma Corp., San Jose, CA, USA
E-mail: [email protected]
Web: www.colorimageprocessing.com