Chapter 1 Introduction

Multimedia is probably one of the most overused terms of the 90s (for example, see [Sch97b]). The field is at the crossroads of several major industries: computing, telecommunications, publishing, consumer audio-video electronics, and television/movie/broadcasting. Multimedia not only brings new industrial players to the game, but adds a new dimension to the potential market. For example, while computer networking was essentially targeting a professional market, multimedia embraces both the commercial and the consumer segments. Thus, the telecommunications market involved is not only that of professional or industrial networks—such as medium- or high-speed leased circuits or corporate data networks—but also includes standard telephony or low-speed ISDN. Similarly, not only the segment of professional audio-video is concerned, but also the consumer audio-video market, and the associated TV, movie, and broadcasting sectors.

As a result, it is no surprise when discussing and establishing multimedia as a discipline to find difficulties in avoiding fuzziness in scope, multiplicity of definitions, and non-stabilized terminology. When most people refer to multimedia, they generally mean the combination of two or more continuous media, that is, media that have to be played during some well-defined time interval, usually with some user interaction. In practice, the two media are normally audio and video, that is, sound plus moving pictures.

One of the first and best known institutes that studied multimedia was the Massachusetts Institute of Technology (MIT) Media Lab in Boston, Massachusetts. MIT has been conducting research work in a wide variety of innovative applications, including personalized newspapers, life-sized holograms, or telephones that chat with callers [Bra87]. Today, many universities, large-scale research institutes, and industrial organizations work on multimedia projects.

From the user's perspective, “multimedia” means that information can be represented in the form of audio signals or moving pictures. For example, movement sequences in sports events [Per97] or an ornithological lexicon can be illustrated much better with multimedia compared to text and still images only, because it can represent the topics in a more natural way.

Integrating all of these media in a computer allows the use of existing computing power to represent information interactively. Then this data can be transmitted over computer networks. The results have implications in the areas of information distribution and cooperative work. Multimedia enables a wide range of new applications, many of which are still in the experimental phase. Think for a moment that the World Wide Web (WWW) took its current form only at the beginning of the 90s. On the other hand, social implications inherent in global communication should not be overlooked. When analyzing such a broad field as multimedia from a scientific angle, it is difficult to avoid reflections on the effects of these new technologies on society as a whole. However, the sociological implications of multimedia are not the subject of this book. We are essentially interested in the technical aspects of multimedia.

1.1 Interdisciplinary Aspects of Multimedia

If we look at applications and technologies, there is a strong interest in existing multimedia systems and their constant enhancement. The process of change that takes place in the background in various industrial sectors should not be underestimated:

• The telecommunications industry used to be interested primarily in telephony. Today, telephone networks evolve increasingly into digital networks that are very similar to computer networks. Switching systems used to be made up of mechanical rotary switches. Today, they are computers. Conventional telephones have been evolving into computers, or they even exist as pure software in the form of “IP telephony.”

• The consumer electronics industry—with its “brown ware”—contributed considerably to bringing down the price of video technology that is used in computers. Optical storage technology, for example, emerged from the success of CD players. Today, many manufacturers produce CD drives for computers and hi-fi equipment or television sets and computer screens.

• The TV and radio broadcasting sector has been a pioneer in professional audio-video technology. Professional systems for digital cutting of TV movies are commercially available today. Some of these systems are simple standard computers equipped with special add-on boards. Broadcasters now transmit their information over cables so it is only natural that they will continue to become information vendors over computer networks in the future.

• Most publishing companies offer publications in electronic form. In addition, many are closely related to movie companies. These two industries have become increasingly active as vendors of multimedia information.

This short list shows that various industries merge to form interdisciplinary vendors of multimedia information.

Many hardware and software components in computers have to be properly modified, expanded, or replaced to support multimedia applications. Considering that the performance of processors increases constantly, storage media have sufficient capacities, and communication systems offer increasingly better quality, the overall functionality shifts more and more from hardware to software. From a technical viewpoint, the time restrictions in data processing imposed on all components represent one of the most important challenges. Real-time systems are expected to work within well-defined time limits to form fault-tolerant systems, while conventional data processing attempts to do its job as fast as possible.

For multimedia applications, fault tolerance and speed are not the most critical aspects because they use both conventional media and audio-video media. The data of both media classes needs to get from the source to the destination as fast as possible, i.e., within a well-defined time limit. However, in contrast to real-time systems and conventional data processing, the elements of a multimedia application are not independent from one another. In other words they do not only have to be integrated, they also have to be synchronized. This means that in addition to being an integrated system, composed of various components from both data types, there has to be some form of synchronization between these media.

Our goal is to present the multimedia systems from an integrated and global perspective. However, as outlined above, multimedia systems include many areas, hence we have decided to split the content about multimedia system fundamentals into three volumes. The first volume deals with media coding and content processing. The second volume describes media processing and communication. The third volume presents topics such as multimedia documents, security, and various applications.

1.2 Contents of This Book

If the word multimedia can have several meanings, there is a risk that the reader might not find what he or she is looking for. As mentioned above, this book is an integral part of a three-volume work on “Multimedia Fundamentals.” Let us start by defining the scope of this first volume.

The primary objective of the book is to provide a comprehensive panorama of topics in the area of multimedia coding and content processing. It is structured as a reference book to provide fast familiarization with all the issues concerned. However, this book can also be used as a textbook for introductory multimedia courses. Many sections of this book explain the close relationships of the wide range of components that make up multimedia coding, compression, optical storage, and content processing in a multimedia system.

1.3 Organization of This Book

The overall goal is to present a comprehensive and practical view of multimedia technologies. Multimedia fundamentals can be divided as shown in Figure 1-1. We will present material about the most important multimedia fields in these volumes. The overall organization attempts to explain the largest dependencies between the components involved in terms of space and time. We distinguish between:

• Basics: In addition to the computer architecture for multimedia systems, one of the most important aspects is a media-specific consideration.

• Systems: This section covers system aspects relating to processing, storage, and communication and the relevant interfaces.

• Services: This section details single functions, which are normally implemented through individual system components.

• Usage: This section studies the type and design of applications and the interface between users and computer systems.

In this volume, we will present the basics of multimedia, concentrating on media-specific considerations such as individual media characteristics and media compression, and their dependencies on optical storage, content analysis, and processing.

Techniques like the sampling theorem or Pulse Code Modulation (PCM), discussed in a later chapter, with their respective mathematical background and practical implementations form the basis for digital audio-video data processing. Several techniques have evolved from these basics, each specialized for a specific medium. Audio technology includes music and voice processing. The understanding of video technology is essentially based on the development of digital TV technology, involving single pictures and animation. As demand on quality and availability of technologies increases, these media have high data rates, so that appropriate compression methods are necessary. Such methods can be implemented both by hardware and software. Furthermore, the high demand on quality and availability of multimedia technology has also placed heavy demands on optical storage systems to satisfy their requirements. As storage capacity and availability of other resources increase, content analysis is becoming an integral part of our multimedia systems and applications.

Figure 1-1 The most important multimedia fields, as discussed in this book.

1.3.1 Media Characteristics and Coding

The section on media characteristics and coding will cover areas such as sound characteristics with discussion of music and the MIDI standard, speech recognition and transmission, graphics and image coding characteristics. It will also include presentation of image processing methods and a video technology overview with particular emphasis on new TV formats such as HDTV. In addition to the basic multimedia data such as audio, graphics, images, and video, we present basic concepts for animation data and its handling within multimedia systems.

1.3.2 Media Compression

As the demand for high-quality multimedia systems increases, the amount of media to be processed, stored, and communicated increases. To reduce the amount of data, compression techniques for multimedia data are necessary. We present basic concepts of entropy and source compression techniques such as Huffman Coding or Delta Modulation, as well as hybrid video compression techniques such as MPEG-4 or H.263. In addition to the basic concepts of video compression, we discuss image and audio compression algorithms, which are of great importance to multimedia systems.

1.3.3 Optical Storage

Optical storage media offer much higher storage density at lower cost than traditional secondary storage media. We will describe various successful technologies that are the successors of long-playing records, such as audio compact discs and digital audio tapes. Understanding the basic concepts such as pits and lands in the substrate layers, modulation and error handling on CD-DA, and modes on CD-ROM, is necessary in order to understand the needs of media servers, disk management, and other components in multimedia systems.

1.3.4 Content Processing

Coding and storage directly or indirectly influence the processing and analysis of multimedia content in various documents. In recent years, due to the World Wide Web, we are experiencing wide distribution of multimedia documents and requests for multimedia information filtering tools using text recognition, image recognition, speech recognition, and other multimedia analysis algorithms. This section presents basic concepts of content analysis, such as similarity-based search algorithms, algorithms based on motion vectors and cut detection, and others. Hopefully this will clarify the effects of content analysis in applications such as television, movies, newscasts, or sports broadcasts.

1.4 Further Reading About Multimedia

Several fields discussed in this book are covered by other books in more detail. For example, multimedia databases are covered in [Mey91], video coding in [Gha99] and in the Handbook of Multimedia Computing [Fur98]. Moreover, the basics of audio technology, video technology, image processing, and various network systems are discussed in specialized papers and books, while this book describes all coding components involved in the context of integrated multimedia systems.

There is extensive literature on all aspects of multimedia. Some journals that frequently publish papers in this area are IEEE Multimedia, IEEE Transaction on Multimedia, Multimedia Systems (ACM Springer), and Multimedia Tools and Applications. Many other journals also publish papers on the subject.

In addition to a large number of national and international workshops in this field, there are several interdisciplinary, international conferences on multimedia systems, in particular: the ACM Multimedia Conference (the first conference took place in Anaheim, California, in August 1993), the IEEE Multimedia Conference (first conference held in May 1994), and the European Workshop on Interactive Distributed Multimedia Systems and Telecommunication Services (IDMS).