RTP with the JMF

Three packages within the JMF are concerned with RTP. They are

javax.media.rtp— The top-level of the three packages dealing with RTP. It comprises 26 classes (most interfaces) dealing with streaming content with RTP.

javax.media.rtp.event— A package of 23 events that might result when using RTP.

javax.media.rtp.rtcp— A package of five classes (four of which are interfaces) defining usage of RTCP within the JMF.

Those applications employing the RTP directly will likely need to import classes from all three packages.

It is worth mentioning that as for PlugIns (discussed in Chapter 8), it isn't required that a JMF implementation support or provide the classes found in the preceding three packages. All current implementations of 2.1.1 (reference, Windows, Solaris, Linux) do so. However, it is possible that some future implementation of the JMF—possibly intended for a low-powered embedded system—won't support the RTP related classes.

One key aspect to understand about the RTP related classes of the JMF are that they are extensions to the JMF. They don't replace or supplant the core functionality of the JMF as found in the Player, Processor, DataSource, DataSink, and Manager (among others) classes. These remain unchanged and still lie at the heart of media handling. They fulfil the same role regardless of whether the media is streamed over the Internet or from the local filesystem.

In other words, a JMF program that plays RTP media will still employ a Player object obtained through the Manager class, as discussed in Chapter 8. Similarly, a JMF program transcoding RTP data it received (from a remote participant in an RTP session) to another format for saving to a local file will still use a Processor, DataSource, and DataSink object. A video-conferencing application will still use CaptureDeviceInfo, Processor, and DataSource objects (amongst others).

Indeed, as discussed in a following subsection, it is possible to play, process, and in general handle media originating from or destined for transport via RTP without employing a single class from the above three packages.

RTP Content Types and Formats

The full gamut of content types (media containers) and formats offered by the JMF aren't available for RTP. In the case of RTP, the choices are far more limited. Those users wanting to use RTP (in particular to transmit over RTP) must be aware of their choices and use only an RTP supported format and content type.

In the area of content type, although there are more than a dozen different ContentDescriptors (such as Wave, AVI, GSM, and QuickTime) within the JMF, there is only one for RTP media: ContentDescriptor.RAW_RTP. Thus the creation of a ContentDescriptor object for RTP always has the following form:

ContentDescriptor rtpContainer = new
          ContentDescriptor(ContentDescriptor.RAW_RTP);

The JMF support for the format of RTP data is also limited. Although it is possible for the user to extend the JMF by implementing the appropriate interfaces and thus adding further RTP-conversant codecs, the JMF currently provides four standard RTP-specific audio formats and three standard RTP-specific video formats.

The audio formats are known as

• ULAW_RTP

• GSM_RTP

• DVI_RTP

• G723_RTP

Whereas the video formats are known as

• JPEG_RTP

• H261_RTP

• H263_RTP

As their names imply, these formats use exactly the same compression schemes as their non-RTP versions. Thus a JPEG_RTP stream has been compressed with a JPEG codec. Construction of RTP-specific Format objects follows the same form as that for non-streaming media. For instance, use the following code to construct a Format object for RTP video data compressed with the H263 codec:

Format streamedVideoFormat = new Format(Format.H263_RTP);

Handling RTP Data Without RTP Classes

It is completely possible to play or process RTP originating or destined data without employing any of the classes found in javax.media.rtp or its two sub-packages. This is attributed to the versatility of the Manager and MediaLocator classes. Certain restrictions are inherent in this approach: In particular, only the first media stream in a session is available for processing or playing, and there is no means of monitoring the session itself.

Undoubtedly the simplest means of handling RTP data is through Sun's demonstration JMStudio. Although it doesn't provide a means for monitoring an RTP session (that requires coding as discussed in the following subsections), it is very simple to both play streaming media and to transmit it. The Open RTP Session option of the File menu allows the play of RTP transmitted data. The user simply enters the IP address and port to which the data is being sent. Similarly, the Transmit option of the File menu provides the user with a mechanism for transmitting either captured (from devices attached to the machine) audio, video, or media in a file over RTP. Thus, it is possible to carry out a video conference using the JMF, but without writing a line of code. Each user would run several instances of JMStudio simultaneously on his machine: one instance to capture and transmit his audio and video and another two instances for playing the other participant's media—one for audio and one for video.

Alternatively, it is possible to write code for handling RTP data, but without using the RTP-related classes of the JMF. It is quite possible to create a MediaLocator object for an RTP stream. This can then be used with Manager's various create methods, namely createProcessor(), createPlayer(), createDataSource(), and createDataSink() in order to obtain the appropriate object for handling the RTP data.

In these cases, the MediaLocator constructor is passed a String of the form:

"rtp://address:port[:ssrc]/content-type/[ttl]"

address is an IP address, port is an integer port number, and content-type is a string such as video or audio. The SSRC (Synchronizing Source) and TTL (Time to Live) fields are optional. By default, SSRC is the originator of the media, and TTL, being the maximum number of router hops the packets can experience before they are not propagated, is 1.

The resulting MediaLocator object (assuming that it is non-null) can then be used in the appropriate create() method of Manager. For instance, the following code fragment is part of the creation of a Player to handle video data being broadcast to a multicast session with the address 224.123.111.101 and using port 4044:

try {
  MediaLocator rtpLocation = new
            MediaLocator("rtp://224.123.111.101:4044/video/");
  Player player = Manager.createPlayer(rtpLocation);
  Player.realize();
  :

Indeed, several of the example utility classes from the previous chapter can be used without alteration to transmit or play RTP data. The PlayerOfMedia GUI application is capable of playing streaming media just as it is capable of playing media from the local file system. In the dialog box provided, the user simply enters a suitable RTP locator string, such as the one found in the preceding example, and a player will be created for the media.

The MediaStatistics utility that reports on the format of a specified media can equally report on an RTP stream.

The Location2Location utility that takes media from one specified location, performs any prescribed transcoding, and then sinks the media to another specified location can be used to handle RTP data in a number of ways. The input location might specify an RTP stream; in which case, that received stream could be transcoded and then saved to a file, for instance. If the output location specifies an RTP stream, Location2Location acts as a transmitter, streaming media out using the specified address. If both input and output locations describe RTP streams, Location2Location acts as a kind of re-transmitter: receiving a stream, possibly performing some transcoding, and retransmitting that transcoded stream.

Even the SimpleRecorder application, which captures audio or video from devices (that is, microphones, Webcams, and so on) attached to the machine, could be modified in a quite straightforward manner so that it transmits the captured data as an RTP stream. Half of the capability already exists in that SimpleRecorder allows the user to specify the destination of the captured media with the -f flag. However, the program is currently hard-coded as to content type (Wave and AVI) and formats that it uses for audio (Linear) and video (Cinepak). If this was altered so that it supported the RTP content type and formats, SimpleRecorder could transmit its captured media in a format that it could be played by another application (for example, PlayerOfMedia).

Using the RTP Classes of JMF

Given the previous section's discussion of handling streaming RTP media without the RTP classes of the JMF, it might appear that the RTP classes are superfluous at best. As with other matters concerning the JMF, it is a matter of the level of control and sophistication of the required application. Playing, sending, and transcoding a stream are all possible without recourse to the RTP-related classes of the JMF. However, user control is limited to functionality within those spheres.

Among the abilities provided by the RTP-specific classes are the following:

• Managing sessions, participants, and media streams (for example, starting, finishing, adding, and so on)

• Monitoring (through listener interfaces) of RTP events (for example, new participants joining the session)

• Gathering and generating statistics (for example, quality of connection)

In particular, the first and second set of capabilities are highly desirable for applications such as a multiway video-conferencing application in which participants can arrive and leave at different times, employ different coding schemes, and have different qualities of connection to the other participants. Automating control for these scenarios almost requires an automated system that can monitor and respond to the dynamic events across the lifetime of the session.

The central class in an implementation handling RTP data, and explicitly acknowledging that fact in order to maximize control is the RTPManager. As shown in Figures 9.6 (playback of media stream) and 9.7 (transmission of captured video), an RTPManager object effectively acts as an intermediary or shield between the JMF objects handling the media (for example, Player, or Processor) and the protocol and session details.

RTPManager is a new class as of JMF 2.1.1 and supersedes the depreciated SessionManager interface. Older examples can still be found that use SessionManager rather than RTPManager. RTPManager provides a uniform interface and method set regardless of whether a unicast, multi-unicast, or multicast session is being managed—something that SessionManager didn't provide.

However, RTPManager isn't the only class of relevance to handle RTP sessions. The following list summarizes the most important:

InetAddress— Java's representation of an internet address. Part of the java.net package. Needed to create session addresses.

LocalParticipant— The participant (listed next) that is also local to the machine. There is only one LocalParticipant: The rest are all remote.

Participant— An application sending or receiving streams within the session.

ReceiveStream— An interface representing a received stream of data within an RTP session. Each received stream in a session is represented by a separate ReceiveStream object.

ReceiveStreamListener— An interface that a class can implement in order to receive events associated with a ReceiveStream (such as timeouts, byes, a new stream, and so on).

RemoteListener— An interface that a class can implement in order to be informed about events generated by remote participants in a session (for example, the availability of sender or receiver reports or a name space collision).

RemoteParticipant— A participant in an RTP session that isn't on the same host (not local) as the RTPManager.

RTPControl— An interface allowing the control of an RTP DataSource object as well as a means of obtaining statistics about said DataSource.

RTPManager— A central class in managing an RTP session: an RTPManager object exists for each RTP session. Capable of creating sessions, streams, adding new participants, and so on.

RTPStream— Superclass of SendStream and ReceiveStream.

SendStream— An interface representing a sent (sending) stream of data within an RTP session. Each sending stream in a session is represented by a separate SendStream object.

SendStreamListener— An interface that a class can implement in order to receive events associated with a SendStream (such as timeouts, payload changes, byes, a new stream, and so on).

SessionAddress— The encapsulation of an RTP session's address as an InetAddress and associated port(s). These objects represent the unicast or multicast addresses associated with participants in the session.

SessionListener— An interface that a class can implement in order to be informed of session wide (that is, not specific to a particular stream) events such as name collisions or the addition of a new participant.

RTPManager

As previously stated, the RTPManager class plays the central role in the management of an RTP session. Unlike the key management classes of the core JMF—such as Manager or CaptureDeviceManager, which are static classes—RTPManager is an abstract class. Instances of the class are created with the static newInstance() method, and each RTP session has its own associated RTPManager object. The following line of code shows the creation of an RTPManager object:

RTPManager sessionController = RTPManager.newInstance();

Managing an RTP session in which transmission is involved via an RTPManager object typically proceeds as follows:

Listing 9.1 and Listing 9.2 show the use of RTPManager to control an RTP session. In Listing 9.1, the video image captured from an appropriate device is multicast to two targets with the specified IP and port addresses. In Listing 9.2, a movie, stored as a file, has its video track multicast to a specified address. Neither are complete applications or classes, but they show the major steps in configuring the manager and starting the session.

Both listings share a number of common features that illustrate the basics of RTP session management. The listings differ chiefly in the configuration of the processor—one transmits H263 formatted video captured from a device, whereas the other transmits JPEG video transcoded from a file—and the setting of target addresses for the RTPManager object. For the unicast session, the RTPManager is initialized with the address of the host machine and has targets specified as the addresses of the machines (plus ports) to be transmitted to. On the other hand, for a multicast session, the RTPManager is both initialized with the multicast address and has its single target specified as that address.

Listing 9.1 An RTPManager Object Handles a Multi-Unicast Session—Broadcasting Video Using H263 Format

////////////////////////////////////////////////////////////////////////////
// Need to capture video and output it as H263 RTP stream. Thus need a
// Processor, which needs a ProcessorModel that specifies ContentDescriptor,
// formats and datasource (in this case implicit to be a capture device that
// can supply the format).
/////////////////////////////////////////////////////////////////////////////
ContentDescriptor rtpContainer = new
                      ContentDescriptor(ContentDescriptor.RAW_RTP);
VideoFormat rtpH263 = new VideoFormat(VideoFormat.H263_RTP);
Format[] formats = {rtpH263};
ProcessorModel captureNTranscodeModel = new
                             ProcessorModel(formats, rtpContainer);
Processor captureNTranscodeProcessor =
                      Manager.createRealizedProcessor(captureNTranscodeModel);

//////////////////////////////////////////////////////////////////////////
// Listen to the Processor and also obtain its output DataSource so it
// can be used to create a SendStream.
/////////////////////////////////////////////////////////////////////////
captureNTranscodeProcessor.addControllerListener(this);
DataSource source = captureNTranscodeProcessor.getDataOutput();

/////////////////////////////////////////////////////////////////////////
// Create the RTPManager to handle the session, then initialise it by
// providing the address of the local machine (with whatever port
// is available.
////////////////////////////////////////////////////////////////////////
RTPManager managerOfSession = RTPManager.newInstance();
SessionAddress hostAddress = new SessionAddress();
managerOfSession.initialise(hostAddress);

/////////////////////////////////////////////////////////////////////////
// Create addresses for the two recipients and add them as targets for
//  the RTPManager. Note that these are arbitrary addresses and a user
// would substitute the known (IP) address of their recipient(s).
////////////////////////////////////////////////////////////////////////
InetAddress firstTargetIP = InetAddress.getByName("175.216.12.3");
SessionAddress firstTargetAddress = new SessionAddress(firstTargetIP,3000);
managerOfSession.addTarget(firstTargetAddress);

InetAddress secondTargetIP = InetAddress.getByName("131.236.21.177");
SessionAddress secondTargetAddress = new SessionAddress(secondTargetIP,3220);
managerOfSession.addTarget(secondTargetAddress);

////////////////////////////////////////////////////////////////////////
// Listen for all types of events that might occur in relation to this
// session. This would require that the class possess the
// appropriate listener methods (not found in this code fragment).
////////////////////////////////////////////////////////////////////////
managerOfSession.addSessionListener(this);
managerOfSession.addRemoteListener(this);
managerOfSession.addSendStreamListener(this);

/////////////////////////////////////////////////////////////////////////////
// Create and start the stream of video data. The stream is created from the
// Processor's output DataSource, with the 1 argument to createSendStream
// specifying that the first track (there should only be 1 for the DataSource
// anyway) be used as the stream.
////////////////////////////////////////////////////////////////////////////
SendStream videoStream2Send = managerOfSession.createSendStream(source,1);
videoStream2Send.start();

//////////////////////////////////////////////////////////////////////////
// When the transmission is over the targets should be removed (informed)
// and the resources acquired by the RTPManager released (via the
// dispose() call. Hence this fragment of code would be found in another
// portion of the class, such as in response to the user
// pressing a "Stop Transmission" button.
///////////////////////////////////////////////////////////////////////
managerOfSession.removeSessionListener(this);
managerOfSession.removeRemoteListener(this);
managerOfSession.removeSendStreamListener(this);
managerOfSession.removeTargets("Transmission Finished");
managerOfSession.dispose();
captureNTranscodeProcessor.stop();
captureNTranscodeProcessor.close();

Listing 9.2 An RTPmanager Object Handles a Multicast Session—Multicasting a Movie Track to a Particular Address

////////////////////////////////////////////////////////////////////////
// Need to transcode a file and transmit as JPEG RTP stream. Thus need a
//  Processor, which needs a ProcessorModel that specifies ContentDescriptor,
// formats and also the location of the media (in a file) to transmit.
////////////////////////////////////////////////////////////////////////
MediaLocator fileLocation = new
                     MediaLocator(file://D:\jmf\book\media\ex1.mov");
ContentDescriptor rtpContainer = new
                     ContentDescriptor(ContentDescriptor.RAW_RTP);
VideoFormat rtpJPEG = new VideoFormat(VideoFormat.JPEG_RTP);
Format[] formats = {rtpJPEG};
ProcessorModel transcodeModel =  new
                            ProcessorModel(fileLocation, formats, rtpContainer);
Processor transcodeProcessor =  Manager.createRealizedProcessor(transcodeModel);

///////////////////////////////////////////////////////////////////////
// Listen to the Processor and also obtain its output DataSource so it
// can be used to create a SendStream.
///////////////////////////////////////////////////////////////////////
transcodeProcessor.addControllerListener(this);
DataSource source = transcodeProcessor.getDataOutput();

////////////////////////////////////////////////////////////////////////////////
// Create the RTPManager to handle the session. As it is a multicast session the
// multicast session address is used both as the target and to initialise the
// manager. In this case the IP address 224.123.109.101 with ports 4056 and 4057
// has arbitrarily been selected as the multicast session address.
//////////////////////////////////////////////////////////////////////////////
RTPManager managerOfSession = RTPManager.newInstance();

InetAddress multicastIP = InetAddress.getByName("224.123.109.101");
SessionAddress multicastAddress = new SessionAddress(multicastIP,4056);
managerOfSession.initialize(multicastAddress);
managerOfSession.addTarget(multicastAddress);

/////////////////////////////////////////////////////////////////////////////
// Listen for all types of events that might occur in relation to this
// session. This would require that the class possess the appropriate listener
// methods (not found in this code fragment).
////////////////////////////////////////////////////////////////////////////
managerOfSession.addSessionListener(this);
managerOfSession.addRemoteListener(this);
managerOfSession.addSendStreamListener(this);

////////////////////////////////////////////////////////////////////////////
// Create and start the stream of video data. The stream is created from the
// Processor's output DataSource, with the 1 argument to createSendStream
// specifying that the first track (there should only be 1 for the DataSource
// anyway) be used as the stream.
///////////////////////////////////////////////////////////////////////////
SendStream videoStream2Send = managerOfSession.createSendStream(source,1);
videoStream2Send.start();

////////////////////////////////////////////////////////////////////////////////
// When the transmission is over the target should be removed (informed) and the
// resources acquired by the RTPManager released (via the dispose() call. Hence
// this fragment of code would be found in another portion of the class, such as
// in response to a StopEvent from the Processor
///////////////////////////////////////////////////////////////////////////////
managerOfSession.removeSessionListener(this);
managerOfSession.removeRemoteListener(this);
managerOfSession.removeSendStreamListener(this);
managerOfSession.removeTarget(multicastAddress, "File Finished");
managerOfSession.dispose();
transcodeProcessor.stop();
transcodeProcessor.close();

Figure 9.8 shows the methods of RTPManager. As has already been stated and shown in Listings 9.1 and 9.2, RTPManager objects are created via the static newInstance() method.

After an RTPManager object has been obtained and the appropriate pre-configuration performed (such as obtaining a multicast or unicast address as well as a DataSource object), the RTPManager should be initialized with the initialize() method. As its name implies, the method initializes the session. It can only be called once. It can throw either an IOException or an InvalidSessionAddressException. There are three versions of the method. The most commonly used version accepts a SessionAddress, which is the address of the local host and the associated data and control ports for the session. If a null SessionAddress is passed, a default local address will be chosen. If the RTPManager subsequently specifies a multicast session address as a target, the local address specified with initialize() is ignored. The multi-argument version of initialize() allows finer control in terms of the percentage of bandwidth consumed by RTCP traffic and even the encryption (if any) employed. The third version accepts an RTPConnector object, which is used when RTP isn't travelling over UDP.

The addTarget() method is used to specify the target of an RTP session. For transmission, this target is an IP address and port pair to be transmitted to. For receipt, this target is an IP address port pair to be listened to. The method is passed a SessionAddress object that specifies the IP address and port. The method can throw either an InvalidSessionAddressException or an IOException.

The addTarget() method effectively opens a session, causing RTCP reports to be generated as well as appropriate SessionEvents. The method should only be called after the associated RTPManager object has been initialized, and before the creation of any streams on a session.

Multi-unicast sessions— one host transmitting the same media to more than one recipient, where that transmission is specifically directed to each recipient—are supported by the mechanism of making multiple addTarget() calls. For instance, if there were four recipients, four addTarget() calls would be made; each one using a SessionAddress object that specified the receiving application's address.

Just as targets can be added to a session, they can be removed either individually with removeTarget() or en masse with removeTargets(). Typically, these methods are employed as an RTP session is being terminated. Although removeTarget() might also be used mid-session in a multi-unicast scenario to stop transmission to an address that is no longer participating or perhaps reachable. Both methods accept a String argument, which is the reason that the local participant has quit the session. This is transported via RTCP. The removeTarget() method has as its first argument a SessionAddress object that matches a current target of the session. The method might throw an InvalidSessionAddressException.

The dispose() method should be called at the end of all RTP sessions. It releases all resources that the RTPManager object has acquired during its existence and prepares the object for garbage collection.

RTPManager objects manage streams of data that fall into two categories: SendStreams for media transmission and ReceiveStreams for media receipt. SendStream objects are created with the createSendStream() method. ReceiveStream objects are created automatically as a new stream is received. They can be obtained through the NewReceiveStreamEvent (see the next subsection), or all current ReceiveStream objects can be obtained with the getReceiveStreams() method of RTPManager.

The createSendStream() method creates a new SendStream from an existing DataSource object (such as the output of a Processor). This is a necessary and vital step if data is to be sent in an RTP session. The method accepts two arguments—the DataSource and a track (or stream) index. The track index parameter specifies which track (stream) of the DataSource to use in creating the SendStream. The first track has an index of 1, the second track has an index of 2, and so on. Although an index of 0 that specifies an RTP mixer operation is desired, all tracks of the DataSource should be mixed as a single stream. The method can throw an UnsupportedFormatException or an IOException.

The getReceiveStreams() method returns a Vector, where each element of the Vector is a ReceiveStream that the RTPManager has created as the result of detecting a new source of RTP data. There is generally less call to use this method because the newly created ReceiveStream objects can be obtained through methods of the event that informs of their creation (see next subsection). Obtaining a ReceiveStream object allows its associated DataSource to be obtained and hence a Processor, Player, or DataSink created for that received media.

Listeners— ReceiveStreamListener, RemoteListener, SendStreamListener, and SessionListener—associated with the RTP session managed by the RTPManager object are added and removed through a set of add and remove methods of the RTPManager object. Listeners are vital in providing the monitoring and control capabilities of the RTP session. There are four methods for adding a listener: addReceiveStreamListener(), addRemoteListener(), addSendStreamListener(), and addSessionListener(). Listeners are usually added once an RTPManager object has been initialized. Correspondingly, there are four methods for removing listeners from an RTP session: emoveReceiveStreamListener(), removeRemoteListener(), removeSendStreamListener(), and removeSessionListener(). Listeners are usually removed at the end of an RTP session. RTP events and their associated listeners are discussed in greater detail in the following subsection.

The JMF represents participants in an RTP session by Participant objects—LocalParticipant and RemoteParticipant. An RTPManager object has several methods for determining the participants in the session it is managing. Those methods are

getActiveParticipants()— Those transmitting in the session

getAllParticipants()— All participants

getLocalParticipant()— The host participant who is also managing the session

getPassiveParticipants()— Those participating but not transmitting data

getRemoteParticipants()— All participants other than the local one

All methods except getLocalParticipant(), which returns a LocalParticipant object, return a Vector of Participant objects.

The final group of methods belonging to RTPManager pertain to session statistics. As their names indicate, the methods getGlobalReceptionStats() and getGlobalTransmissionStats() provide a means of obtaining transmission and reception statistics for the session. Session statistics are discussed further in a subsequent subsection.

RTP Events and Listeners

Four super classes of events, and their corresponding listeners, are defined with the JMF. They are

SessionEvent/SessionListener— Events that pertain to the session as a whole, such as a new Participant joining

SendStreamEvent/SendStreamListener— Changes in the streams that are being transmitted including a new stream, or a stream stopping

ReceiveStreamEvent/ReceiveStreamListener— Changes in the streams that are being received including a new stream, or a stream timing out

RemoteEvent/RemoteListener— Events that pertain to RTCP messages such as a new receiver or sender report being received

Each of the four events have subclasses that specialize in the information provided. For instance SessionEvent has two subclasses: NewParticipantEvent, and LocalCollisionEvent. All four events share the same parent class, RTPEvent, which is a subclass of MediaEvent. All events are found in the javax.media.rtp.event package.

RTP listeners— SessionListener, SendStreamListener, ReceiveStreamListener, and RemoteListener—are associated with an RTP session by means of the RTPManager object that is managing the session. The RTPManager class possesses four methods for adding and four methods for removing listeners—one for each type of listener. For instance, to add a ReceiveStreamListener for the session that the RTPManager object is managing, that object's addReceiveStreamListener() method is called.

All four listener interfaces— SessionListener, SendStreamListener, ReceiveStreamListener, and RemoteListener—define a single method update() that accepts an event of the type associated with the listener. That is, the SessionListener interface defines a single method update(SessionEvent e), and so on for the other three with their events.

SessionListener objects receive two classes of events through their update() methods. The first is a NewParticipantEvent, indicating that a new participant has joined the session. The second one is a LocalCollisionEvent, indicating that the local host's SSRC has collided (is the same as) with that of another participant.

SendStreamListener objects receive five classes of events through their update() methods. A NewSendStreamEvent indicates that the local participant has just created a new SendStream. An ActiveSendStreamEvent indicates that data transfer has begun from the DataSource used to create the SendStream. An InactiveSendStreamEvent indicates that data transfer from the DataSource used to create the SendStream has stopped. A LocalPayloadChangeEvent indicates that the format of the SendStream has changed. A StreamClosedEvent indicates that the SendStream has closed. Listing 9.3 illustrates how an anonymous SendStreamListener class might terminate an RTP session when it detects that the stream being transmitted in the session is exhausted.

Listing 9.3 An Anonymous SendStreamListener Terminates an RTP Session After Transmitted Data Is Exhausted

DataSource source = processor.getDataOutput();
RTPManager managerOfSession = RTPManager.newInstance();
SessionAddress hostAddress = new SessionAddress();
managerOfSession.initialise(hostAddress);

managerOfSession.addTarget(target1);
SendStream stream2Send =  managerOfSession.createSendStream(source,1);
managerOfSession.addSendStreamListener(new SendStreamListener() {
    public void update(SendStreamEvent e) {
        if (e instanceof InactiveSendStreamEvent) {
           managerOfSession.removeSendStreamListener(this);
           managerOfSession.removeTarget(target1,"Data Source exhausted");
           processor.close();
           managerOfSession.dispose();
      }
    }});
stream2Send.start();

ReceiveStreamListener objects receive seven classes of events through their update() methods. A NewReceiveStreamEvent indicates that the RTPManager object has just created a new ReceiveStream object for a new data source. An ActiveReceiveStreamEvent indicates that data transfer has begun. An InactiveReceiveStreamEvent indicates that data transfer has stopped. A TimeoutEvent indicates that data transfer has timed out. A RemotePlayloadChangeEvent indicates that the format of a stream has changed. In a StreamMappedEvent, the originating participant is discovered for an existing stream with a previously unknown origin. An ApplicationEvent indicates that a special RTCP application specific packet has been received.

The NewReceiveStreamEvent is particularly important because this is the means by which new streams, transmitted by a remote participant, are discovered. The newly received SendStream can then be obtained with the getReceiveStream() method (inherited from ReceiveStreamEvent). RTPStream's (the superclass of ReceiveStream) getDataSource() could then be used to obtain a DataSource object for the stream. With that DataSource, a Player, Processor, or DataSink object could then be created to handle the stream as desired. Listing 9.4 shows one way in which a newly received media stream might be handled through the creation of a Player object. The listing shows the use of an anonymous ReceiveStreamListener class that reacts to NewReceiveStreamsEvents by creating a realized Player object.

Listing 9.4 A ReceiveStreamListener Creates a New Player Object in Response to a New Stream Being Received

Player player;
SessionAddress destination = new SessionAddress(...);
RTPManager managerOfSession = RTPManager.newInstance();
managerOfSession.addReceiveStreamListener(new ReceiveStreamListener() {
  public void update(ReceiveStreamEvent e) {
    if (e instanceof NewReceiveStreamEvent) {
      ReceiveStream received = e.getReceiveStream();
      if (received==null)
        return;
      DataSource source = received.getDataSource();
      if (source==null)
        return;
      player = Manager.createRealizedPlayer(source);
     // etc. such as listening to the Player, getting itws Components, etc.
    }
}});
SessionAddress hostAddress = new SessionAddress();
managerOfSession.initialise(hostAddress);
managerOfSeesion.setTarget(destination);

RemoteListener objects receive three classes of events through their update() method. A ReceiverReportEvent indicates that a ReceiverReport RTCP packet has been received. A SenderReportEvent indicates that a SenderReport RTCP packet has been received. A RemoteCollisionEvent indicates that two particpants' SSRC have collided (are the same).

RTP Streams

The JMF represents streams of data within an RTP session as RTPStream objects. RTPStream is an interface extended by two further interfaces—ReceiveStream for a stream of data being transmitted by and received from another participant, and SendStream for a stream that the current application (participant) is sending. Both types of RTPStream objects are associated with an RTP session and managed by an RTPManager object.

SendStream objects are created by an RTPManager object's createSendStream() method from a DataSource object. On the other hand, ReceiveStream objects are created automatically by an RTPManager object when a new stream of data is received by the manager.

SendStream objects are created by a broadcasting participant in an RTP session. After a SendStream has been created, it can be started with the start() method of the object. Starting s SendStream means that data will be transmitted over the network. Typically, transmitting programs then ignore the SendStream object until the transmission is completed—at which time, the object's close() method is called. Closing a SendStream frees all resources associated with that stream; hence, the method should always be called after the stream is no longer required. It is also possible to temporarily pause transmission of a stream by calling stop() on the SendStream object associated with that stream. This will also result in the DataSource that feeds the stream being stopped (via its stop() method). Hence data isn't lost simply because stop() was called.

ReceiveStream objects are created automatically by the RTPManager object managing a particular RTP session when a new stream of data is detected. User programs typically obtain ReceiveStream by calling the getReceiveStream() method of the NewReceiveStreamEvent that was posted to all ReceiveStreamListeners for the current session. Alternatively, RTPManager provides a method getReceiveStreams() that supplies all ReceiveStream objects being managed.

However, the ReceiveStream object is really an intermediary step in handling the received media. JMF media handlers such as Processors and Players, as well as other important objects such as DataSinks, all require a DataSource for their creation via the key Manager class. RTPStream has a method getDataSource() for obtaining the DataSource associated with a stream (whether ReceiveStream or SendStream). Hence, the standard approach when a new ReceiveStream is detected is to first obtain the ReceiveStream itself from the event and then use the stream to obtain the associated DataSource. That DataSource can then be employed to create the appropriate media handler or class. For instance, if the media was to be recorded, a DataSink would be created for that DataSource.

SessionAddress and InetAddress

RTP sessions are associated with one or, in the case of a multi-unicast session, multiple addresses. Those address(es) represent the participants within a session or the multicast address used for the session. For an RTP session, an address must consist of both an IP address and a port number. In fact, two are needed—one for data and one for control—but the control port defaults to be one greater than the data port if it isn't supplied. The JMF employs the SessionAddress class to represent an RTP address, whereas the InetAddress class of java.net (core platform) represents an IP address.

An InetAddress object is Java's standard means of representing an IP address (that is, usually a machine on the Internet). Hence an InetAddress object is used as a stepping stone in the construction of a SessionAddress object.

The InetAddress class doesn't possess a constructor but rather three static methods from which an InetAddress object can be obtained. Those are getLocalHost(), which returns the InetAddress object for the machine on which the program is running; getByName(), which passes a String representing a machine, (for instance "131.236.20.1" returns an InetAddress object for that named); and getAllByName(), which also accepts a String as an address name and returns an array of valid InetAddress objects for that name. Because RTP sessions are initialized with the local machine's address via RTPManager's initialize(), InetAddress.getLocalHost() is found commonly in JMF programs using RTP. Similarly, RTP broadcasting requires a specification of the address to broadcast to, so InetAddress.getByName() is also found commonly in JMF programs using RTP. Listing 9.5 shows both these methods being used.

SessionAddress objects are the JMF's means of specifying the addresses within an RTP session. In particular, SessionAddress objects are required to initialize an RTPManager, as well as set the (transmission) targets of that manager.

SessionAddress objects represent both an IP address and ports for data and control transmission. There are a number of constructors including no arguments (no functionality): an InetAddress and int data port (the most commonly used), two InetAddress and int port pairs (one for data and one for control), and one constructor that accepts an InetAddress, int data port, and int Time to Live. The most commonly used constructor for a SessionAddress object accepts an InetAddress, the data address, and an int port number—the port on which data is transmitted. The control port number defaults to one higher than the data port.

Listing 9.5 shows the initialization phase of an RTP session in which a stream of data will be sent from the local machine to that with the IP address "145.201.33.9" on port 3000. It shows the construction of two SessionAddress objects—one to initialize the session and one as the target of the session.

Listing 9.5 Initialization Phase of an RTP Session

try {
  RTPManager managerOfSession = RTPManager.newInstance();
  managerOfSession.initialize(new
             SessionAddress(InetAddress.getLocalHost(),3000));
  managerOfSession.setTarget(new
          SessionAddress(InetAddress.getByName("145.201.33.9"),3000));
}

Participants

Participants within an RTP session—those receiving or transmitting streams of data—are represented by Participant objects within the JMF. The RTPManager object for a session keeps track of the participants within a session and provides methods for obtaining them:

The Participant interface is extended by both LocalParticipant and RemoteParticipant. These subinterfaces are really placeholders though. They don't add any significant methods and, by their names, simply identify the participant types.

Knowing a participant in a session having a Participant object, it is possible to obtain RTPStream objects that represent all streams that the participant is transmitting. It is also possible to obtain all the most recent RTCP reports for that participant. These are the two most common uses of Participant objects—as a means to obtain the streams they are transmitting or as a means to obtain their most recent reports.

Participant's getStreams() method returns a Vector of all streams that the participant is sending. If none are being sent by that participant, an empty Vector is returned. Participant's getReports() method returns a Vector of RTCPReport objects. Those objects represent the most recent report for each stream the participant is sending or receiving.

Statistics

An important task when managing an RTP session is keeping track of the quality of the connection being experienced by all participants. Particularly clever management might, for instance, adjust the payload in response to changes in the network. If it becomes burdened, the frame rate or resolution might be dropped temporarily.

The RTCP provides a basic mechanism for this kind of monitoring through participants issuing periodic reports. In the JMF, RTCP reports are represented by the Report interface and its associated Feedback interface. Report objects are associated with a single participant and a stream they are transmitting or receiving. Session wide statistics are available as GlobalReceptionStats and GlobalTransmissionStats objects from the RTPManager object responsible for a session.

Report objects can be obtained for a Participant with the getReports() method, as well as from an RTPStream object with getSenderReport(). Report objects possess a getFeedbackReports() method that returns a Vector of Feedback objects. Feedback objects provide methods for determining the number of packets lost, inter-arrival jitter, and other properties of the stream (see the JMF API for full details). The Report interface is subclassed into SenderReport and ReceiverReport. ReceiverReport is an empty interface, but SenderReport provides a number of methods for determining sender specific properties such as timestamps and byte counts.

Less specific but generally more useful are the global statistics provided by an RTPManager object for the session it is managing. These global statistics come in the form of GlobalReceptionStats and GlobalTransmissionStats objects and are obtained with the getGlobalReceptionStats() and getGlobalTransmissionStats() methods, respectively.

A GlobalReceptionStats object provides a number of methods for determining the reception quality for the entire session. These include the number of packets that failed to be transmitted, the number of bad packets received, and the number of local collisions. The JMF API provides a complete listing. A GlobalTransmissionStats object provides six methods for determining the transmission quality of an entire session. These include knowing the total number of bytes sent, number of failed transmissions, and number of local and remote collisions.

The manager of an RTP session can use these objects to maintain a profile of the session it is managing. Report objects are generated at regular intervals and have associated events so that it is easy to keep track of their arrival. However, the choice of what to do with the available statistics is still in the jurisdiction of the user's code. Simply present them to the local participant (for example, as part of their GUI) so that they are aware of the session state, or carry out dynamic adjustment of the session in response to the statistics.

Receiving and Transmitting Streams with RTPManager

As detailed in an earlier subsection, RTPManager plays the key central role in controlling an RTP session. It oversees the session in a number of ways: specifying the session address(es), creating streams for transmission, and providing a means for specifying listeners to the various events the session generates.

The earlier subsection on RTPManager provides the rough outlines of an algorithm when RTPManager is being used to transmit data. The corresponding generic and minimum algorithm for the reception (only) case is as follows:

For the often used example of an audio-video conference in which two or more parties participate using a single multicast session address, the approach is a combination of both the receive and transmit cases:

For both these algorithms, the while loops are a linear approximation of an event-driven program. The code doesn't poll for receive-stream or send-stream events, but simply adds itself as a listener for those events.

It is also feasible for a single program to employ several RTP sessions, and hence RTPManager objects, simultaneously. This isn't an uncommon technique used for managing multiple transmissions. For instance, video could be on one session, and audio could be on a separate session.

Sun's excellent AVTransmit2 and AVReceive2, found on the JMF solutions site currently at http://java.sun.com/products/java-media/jmf/2.1.1/solutions follow such an approach. AVTransmit2 is intended for transmitting on one or more RTP sessions, whereas AVReceive2 is intended for receiving (where receipt entails the playing) of one or more streams on one or more sessions. Potentially, both employ multiple RTP sessions and hence multiple RTPManager objects. They are a good next step for those wanting to further understand RTP-based streaming. Indeed, not only can they be used out of the bag for AV conference, but they also serve as a good starting point for the reader wanting to implement his own specialized RTP-based application.

Cloning and Merging for Transmission

In two broadcast RTP scenarios, it is necessary to manipulate the DataSource that forms the basis of a SendStream before the SendStream object is created with the RTPManager object responsible for the RTP session.

In some multi-RTP session scenarios, the same media is being sent directly to different recipients as separate streams. For instance, a three-way AV transmission might be structured so that each pair of participants uses a different session—each participant is then transmitting the same data in two different sessions. In the JMF, a DataSource is single use. For instance, the same DataSource cannot be processed with a Processor and then played with a Player—although the output DataSource of the Processor could be played. This is true for SendStream creation also: A single DataSource can only be used to create a single SendStream. If multiple sessions are to employ the same DataSource to create SendStream objects, the original DataSource must be transformed into a cloneable DataSource and clones created for each SendStream to be created. The following listing is an example of when a DataSource is being used in two different sessions that are managed by two RTPManagers: manager1 and manager2.

RTPManager manager1 = RTPManager.newInstance();
RTPManager manager2 = RTPManager.newInstance();
DataSource source = Manager.createDataSource("file://example.wav");
:        :        :
DataSource cloneable = Manager.createCloneableDataSource(source);
DataSource firstClone = cloneable.createClone();
SendStream firstStream = manager1.createSendStream(firstClone,1);
firstStream.start();
DataSource secondClone = cloneable.createClone();
SendStream secondStream = manager2.createSendStream(secondClone,1);
secondStream.start();

On the other hand, if data from separate sources (DataSource objects) is to be transmitted as a single stream, it is necessary to merge those DataSources into a single source before the associated SendStream is created. Also, when the SendStream object is created from the merged DataSource, the second parameter to createSendStream() should be the value of 0, indicating that all tracks (streams) of the DataSource should be mixed.

Buffering, Packet Size, and Jitter

Jitter is the phenomenon sometimes experienced when playing streaming media that isn't consistent in transmission rate (network delays) and reliability (packet loss). Jitter manifests as momentary pauses, drop outs, or jumps in the received (rendered) media.

The most useful technique in a receiver's arsenal is to use a buffer—a pool of data into which arriving media is added and from which media is taken to be rendered. The buffer acts to improve transmission inconsistencies, smoothing out differences in transmission rate. The larger a buffer, the greater the jitter that can be smoothed but also the longer the lag between receipt and rendering. For instance, a buffer size of 5 seconds implies that data currently being rendered was actually received 5 seconds ago. Large buffer sizes tend to adversely affect interactivity (for example, carrying out a conversation in which everything is delayed by a few extra seconds). Clearly there is a payoff or balance between smoothing jitter and loss of timely rendering of the data.

Another factor in the equation is the size of the packets transmitted. Larger packets use bandwidth more efficiently because less bandwidth is dedicated to packet headers. However the loss or damage of a large packet is more costly (and perhaps more likely) than that of a small packet in terms of the perceived quality of the media delivered. Losing a few milliseconds of speech might not even be noticed; on the other hand, one-half second lost could even affect comprehension of what was being said.

Both buffer size and packet size can be altered by JMF programs using RTP. There are few solid guidelines as to ideal values for each—they are both highly reliant on the particular scenario (bandwidth availability, network reliability, type of media being sent, and so on).

Receiving programs can alter the size of their buffer through the use of a BufferControl object. BufferControl is a Control interface and can be obtained for a ReceiveStream by first obtaining that ReceiveStream object's DataSource (using the getDataSource() method). The DataSource, which implements the Controls interface, can then be used to obtain the BufferControl object (using the getControl() method). BufferControl consists of six methods—the most important of which is setBufferLength(). This method accepts a single parameter of type long, being the length of the buffer in milliseconds.

Transmitting programs can alter the size of the packets they are transmitting if they can obtain a PacketSizeControl object for the transmission. PacketSizeControl is a Control object, but not all Codecs (Processors) expose PacketSizeControl objects through their getControl()/getControls() methods. The PacketSizeControl interface consists of two methods: getPacketSize() and setPacketSize(). The packet size is expressed as an int and is the maximum packet size output by the encoder.