The JMF in Conjunction with Other APIs

The JMF is a powerful API, supporting a high-level and uniform approach to controlling, playing, and processing time-based media. An equally important JMF strength is that it is part of the larger Java platform. The implication of this is that the JMF approach to time-based media can be combined with other features of Java; either from the core platform, or one or more of the other specialist APIs that extend the functionality of Java. This leads to programs that can combine time-based media with other media (for example, 3D graphics) in either traditional multimedia paradigms, or in new and innovative approaches that are only possible because of the common platform-independent glue that is Java.

Consider an educational application dealing with Australian Aboriginal culture and civilization in the region of the Olgas (central Australian geographic formation) across the last 50,000 years. The application might combine a library of still images of the region, maps, hypertext about the languages and customs of the region, 3D interactive models of reconstructed camp sites, and audio and video interviews with tribal elders of today, perhaps telling some of the Dreamtime legends unique to the area. Such an application could be written entirely in Java, using features of the core platform, the JMF, and Java 3D (perhaps with other APIs also), and would run on any platform.

What about an application of tomorrow? A virtual or immersive audio-video conference with colleagues overseas who are speaking a different language unknown to the recipient. Such an application would use the JMF in conjunction with features of Java 3D, the JAI, and other APIs such as Java Speech. Not only would the audio and video be streamed between participants and projected into a virtual environment (for example, a model of a new building that is being jointly designed), but the audio stream would be extracted, processed by a recognizer (for the speaker's language); then translated into the hearer's language and reintegrated with the video as subtitles or as a separate synthesized audio track. Again, Java provides the framework to enable such a future application.

Applications that involve the JMF in conjunction with other APIs fall into two broad categories. The simpler form, such as the preceding archaeological/cultural multimedia application, uses the JMF as a plug-in component of the entire application. The JMF portion (playing the interviews) is a logical, self-contained component of the entire application. Such applications, although often large, are relatively modular and don't hinge on low-level interfacing of different APIs. The more difficult applications, such as the virtual AV conference involving language translation, aren't so modular in their composition. They tend to rely on the fusing of APIs at a lower level of detail. In the multi-language AV conference application, the media streams must be demultiplexed and passed off to other APIs (the speech recognition and translation) that don't directly support the JMF data models. This requires a knowledge of the deeper data structures in the communicating APIs and the means of converting between them. For the JMF side, that might involve strong familiarity with the Buffer, Format, Stream, Clock, and related classes and interfaces.

Chapter 14, “Integrating Across the Java Media API,” is explicitly concerned with combining the media APIs discussed in this book. As a prelude to that chapter, the following subsection discusses the key JMF classes that allow a video stream to be treated as a sequence of images by other APIs.

ImageToBuffer and BufferToImage

Two classes, ImageToBuffer and BufferToImage, play central roles as the lynchpins between the video component of the JMF and 2D or 3D graphics. For instance, with these classes, it is possible to grab individual images from a video, insert frames into a video, or even extract, modify, and reinsert them. Indeed, it is possible to write a video Renderer by using BufferToImage—simply pull out each frame and render that image (using Java's Graphics class). BufferToImage is one mechanism by which JMF originated video can be imported into other contexts. For instance, a JMF video could be applied as a texture in a Java 3D world by texturing the individual images that compose the video. Conversely, a video can be constructed as a sequence of (AWT) images.

ImageToBuffer and BufferToImage are the sole members of the javax.media.util package. As shown in Figure 9.13, the BufferToImage class provides the ability to convert from a JMF (video) Buffer to an AWT Image (BufferedImage), while ImageToBuffer provides the reverse functionality.

The BufferToImage class possesses a single constructor, accepting a Format object. That Format object specifies the format of the video buffer that must be converted. Hence, BufferToImage objects are specific to a particular Format, but can convert any Buffer in that Format into its equivalent Image. The class has a single method createImage() that accepts a Buffer (of the Format specified in the constructor) and returns an AWT Image. Null is returned if the conversion cannot be done.

Obtaining an Image from a video sequence devolves into obtaining a Buffer object that corresponds to the video frame in question. BufferToImage's createImage() can then be applied to the Buffer to generate the Image. Obtaining a Buffer for a particular video frame can be achieved through use the FrameGrabbingControl interface. A FrameGrabbingControl can be exported by a Player or Renderer through the getControl() method. The interface possesses a single method grabFrame() that returns a Buffer object that corresponds to the current frame from the video stream. In addition to FrameGrabbingControl, there is also the useful FramePositioningControl that can be employed to precisely position a stream at a particular frame number. As shown in Listing 9.7, these can be used in conjunction so that a particular (known) frame number can be grabbed and turned into an image. In this case, the 500th frame is used.

Listing 9.7 A Particular Video Frame Is Grabbed and Transformed into an AWT Image

int    desiredFrame = 500;
Player    player = Manager.createRealizedPlayer(new MediaLocator(...));
FramePositioningControl positioner =
           player.getControl("javax.media.control.FramePositioningControl");
if (positioner==null) return;
FrameGrabbingControl grabber =
           Player.getControl("javax.media.control.FrameGrabbingControl");
if (grabber==null)  return;
player.prefetch();
// Some sort of pause/polling/event listening to ensure its prefetched

positioner.seek(desiredFrame);

// Assumption here is that Player object doesn't "regress" into an earlier
// "less prepared" state. More generally should wait till Player returns to // // 
prefetched.
Buffer inTheBuff = grabber.grabFrame();
Format videoFormat = inTheBuff.getFormat();
BufferToImage converter = new BufferToImage(videoFormat);
Image captured = converter.createImage(inTheBuff);

// Now do something useful with captured, the image that corresponds to the
// desired (500th) frame.

Going in the opposite direction—from an AWT Image to a JMF Buffer—employs the ImageToBuffer class. The class possesses a single static method that accepts an AWT Image and desired frame rate and returns a JMF Buffer with an RGB Format. Although that operation is simple, construction of a stream from the Buffers is somewhat more difficult. Sun's example Screen Grabber [DataSource] linked from their JMF solutions page, http://java.sun.com/products/java-media/jmf/2.1.1/solutions/index.html, is a good example of building a stream from individual images.