14

Synthetic Characters

After all of our technology, the pseudo intelligence algorithms, the fast exception matrices, the portent and content monitors and everything else, we still can’t come close to generating a human voice that sounds as good as what a real live actor can give us. (Neal Stephenson in his futuristic novel The Diamond Age)

We have talked about the value of characters in presenting obstacles in story simulation and in guiding the progress of the story. We described the Final Flurry exercise created for the Department of Defense and implemented at the Industrial College of the Armed Forces. In that simulation the students were divided into groups, which represented advisors to the National Security Advisor (NSA). After a day of deliberation, the group would make recommendations to the NSA who would then pass those recommendations onto the President of the United States. The president in turn would then present those recommendations to the nation in a speech whose highlights were shown to the class as feedback for their deliberations.

We talked about the value of having predicted what the class would say and having the president’s remarks restate the class’s recommendations. We noted how the instructors of Final Flurry selected video clips from prerecorded segments that could be arranged into a tight presentation. We also pointed out that, in those cases where the class missed the point, the instructor could send a message to the class (usually a text e-mail) that explained exactly why their specific recommendation was not followed. Immediately after the first presentation of the StoryDrive version of Final Flurry, the development team from Paramount and the DoD agreed that it would be far better if the president could always respond to the specific recommendations of the class.

In ALTSIM, we said that one way of keeping the story on track was to use the Battle Captain’s commanding officer (CO), who as a matter of procedure would always review important orders that the BC was issuing. The CO could be in videophone-contact with the BC and could direct, guide, and if necessary, override the orders. This may have been the very best way of keeping a military simulation on track. In this case too, while the simulation author anticipated many reactions from the CO and prerecorded them, we again thought how effective it would be to have a real-time CO who could discuss the operation with the Battle Captain.

In both cases what we were imagining was a synthetic character, not just a nonplayer character with prescripted lines, but also a simulated character who reacted in real time to the specific input of the participants, a character who could address their recommendations to the letter.

LIVE ACTOR PARTICIPATION

The low-tech solution to the problem is not to use a synthetic character at all but to have a live person playing the role of the CO. The actor’s image could be captured and presented on the participants’ computer screens in real time. Imagine an actor playing the role of the president in the Final Flurry simulation. The instructor for each session selects elements for the president’s speech that are prescripted and adds comments and remarks that tailor the speech to the exact recommendations of the class. The actor then reads the tailored speech on camera and the presentation is either fed live to the participants in real time or is recorded for delayed use.

In ALTSIM the simulation instructor could play the CO and his or her voice could be forwarded in real time to the participants in the simulation. This process is a true extension of the man-in-the-loop philosophy of instruction and leads us directly to the question of the role of the instructor or game manager in learning simulations. We will deal with that issue in the next section of this book. But for the purposes of this chapter, we have to say that this approach may solve a lot of problems, but it does not scale up to large instructional settings, does not make use of a synthetic character and it was not part of our research objectives.

TEXT-TO-SPEECH AND SYNTHETIC AUDIO

Currently, synthetic speech is so limited in its presentation of the human voice that characters’ remarks sound robotic. While this works very well in simulations where you can actually portray the character as a robot, it seems very much out of place in simulations where a speaker is supposed to be a human being and is featured alongside video and audio clips which present the speech of actual actors. The jarring sense that the participants get when they hear the synthetic audio totally destroys the reality of the simulation. Yet as limiting as the technology is, we did experiment with it in the ALTSIM project.

We used a standard text-to-speech engine, which the simulation instructor could use to send specific messages to the participants. The instructor merely typed in the required recommendation and the system presented it through its text-to-speech engine. To provide a face to go with the speech, we used photos of an actor that captured his face as he expressed different phonemes (mouth expressions). We then created a program that matched the facial expressions with the related audio syllables as they were spoken by the text-to-speech system. The video image was far more believable than the sound of the audio. It did allow us to experiment with the concept of virtual characters and also forced us to realize early on that, at least given the current state of voice synthesis technology, the loss of believability from prerecorded audio comments that did not exactly match the participant’s responses was not nearly as great as the loss of believability created by the use of synthetic speech.

This finding was disappointing, but well in keeping with other studies that looked at the educational value of production values within simulation systems. These studies show that the closer the simulation gets to reality, the higher the need for accuracy. Participants accept a virtual world if the world is internally consistent. So when a highly realistic world is created, single elements that are not true to the realism of that world break the believability of the system. (See Chapter Twenty-One for further discussion of this concept.) The best things that could be hoped for in such a case would be creative solutions that at least explain the inconsistency.

A CREATIVE SOLUTION FOR LOW PRODUCTION VALUE TEXT-TO-SPEECH

In the case of our synthetic commanding officer, if the officer is said to be using a radio system whose properties include a good deal of radio static and which also digitizes the human voice, it may be possible to inject a synthetic character who can deliver completely original lines that are totally in keeping with the exact input of the participants and yet are somewhat believable in the context of the sound of the rest of the media in the simulation. This seemed to be the case in the few tests that we ran with ALTSIM, before we decided to drop the use of synthetic characters in favor of greater reliance on prerecorded audio and text messaging from the instructor. (We’ll discuss this and other audio techniques further in Chapter Twenty-Five.)

SUMMARY

Text-to-speech and synthetic audio can be combined with images of real people to create synthetic characters. These characters allow simulations to directly match input from the instructor and can be a valuable tool in experience management (keeping the story on track and challenging the participants). However, the current limited quality of synthetic audio creates a distraction for participants in simulations, especially when a large amount of realistic prerecorded audio is used. In the end, prerecorded audio that approximates answers to questions appears to be better than synthetic audio that is right on if the voice quality of the synthetic audio is distracting. The inclusion of text messaging from the instructor or a text-driven coaching system was found to be preferable in the three simulations that Paramount and USC did for the US military.