7.1. Communicating with others: the role of the voice

Interaction between two individuals is based on the decoding of verbal and non-verbal elements. Albert Mehrabian’s work (1972) has shown that interpersonal interactions are mainly based on the decoding of non-verbal information, especially in an emotional situation (93%, see Figure 7.1). Verbal elements (7%) still participate in the success of the interaction thanks to the ortholinguistic functions³ that lead to the decoding of the verbal content (from phonological processing to syntactic-semantic integration). Non-verbal cues conveyed by the faces (Burton et al. 1990; George and Conty 2008; Zebrowitz and Montepare 2008), the body (de Gelder 2006), the body odors (Demattè et al. 2007; Leleu et al. 2015) and the emotion-related voice modulation, i.e. emotional prosody (Belin et al. 2004; Beaucousin et al. 2007; Bruck et al. 2011), provide paralinguistic information about the communicative context that will structure and organize the interpersonal interaction. The voice, face and body convey information about the interlocutor such as identity, gender, age, socio-cultural background, personality traits and internal emotional states (Lacheret-Dujour and Beaugendre 1999; Van Lancker Sidtis 2006; George and Conty 2008).

The ability to read these non-verbal social cues, such as voice or gaze, is a basic process that ensures social adaptation and leads to the recognition of emotional mental states, to our capacity for empathy and to social coordination (Lakin et al. 2003; George and Conty 2008; Schwartz and Pell 2012). A difficulty in perceiving non-verbal speech cues greatly diminishes the quality of social exchanges, as it is the case in schizophrenia (Box 7.1) (Brazo et al. 2014), neurodegenerative diseases (Speedie et al. 1990; Drapeau et al. 2009; Obeso et al. 2012) or in patients with brain injury such as aprosody⁴ (Ross 2000).

COMMENT ON FIGURE 7.1.– In an interaction between two individuals, the interlocutor perceives verbal and non-verbal elements (emotional prosody, facial expression, etc.). Verbal elements are analyzed through ortholinguistic functions (phonological, lexical and syntactic-semantic integration). From the non-verbal elements, the paralinguistic functions provide the interlocutor with information about the identity, personality, mental state and communicative intentions of the speaker. Depending on the situation, up to 95% of non-verbal information is thought to contribute to the speaker’s message comprehension (Mehrabian 1972), thus ensuring optimal social adaptability and coordination (Beaucousin et al. 2007; George et al. 2008).

In this chapter, we are particularly interested in the voice as a vehicle for non-verbal information about the person, and thus in emotional prosody. Within the framework of the model we have just posed, it seems interesting to clarify the role of emotional prosody in communication from a phylogenetic and ontogenetic perspective because, although we have the impression that communication is essentially based on verbal content, this does not seem to be the case at all ages of life and even throughout human history. These data allow us to have a better understanding of the place of emotional prosody in interactions.

From a phylogenetic perspective, researchers suggest that the protolanguage of hunter-gatherer tribes, i.e. the form of communication existing before the appearance of a language⁵, could have corresponded to intonational nuances similar to emotional prosody (Locke 1997; Falk 2004). Indeed, it would have allowed, for example, women to calm the cries of their children during harvesting or men during hunting to coordinate their actions in order to kill a large prey. This protolanguage would therefore have allowed the direct transmission of simple information. Gradually, a system of vocal signs (minimal verbal content) organized syntactically would have complexified this protolanguage leading to the emergence of a language allowing an increasingly elaborated communication. In the course of human history, the importance of emotional prosody in the understanding of language would thus have decreased, to the benefit of the decoding of verbal content (Rousseau 1781). But can its contribution be considered as secondary? Before answering this question, let us look at what happens during development.

This precedence of verbal content over prosodic content for language comprehension changes also during development. During in utero development, the baby is able to perceive the sound environment from the third trimester of pregnancy. Intrauterine recordings show that it is mainly prosody that is transmitted through the abdominal wall and the amniotic fluid (Querleu et al. 1988). The verbal content is difficult to distinguish, giving the impression to an adult listening to these recordings that they are hearing through a door. After birth, adults use an adapted mode of communication to interact with the young child, called Child-directed speech – or motherese⁶. This mode of communication is based on the exaggeration of emotional prosody (slowing down, more pronounced intonational modulation, etc.) but without modification of lexical units compared to speech directed toward the adult (Monnot et al. 2005). Child-directed speech conveys information about communication intentions (Fernald and Kuhl 1987; Fernald 1989). Indeed, an adult is able to identify the communicative intentions of delexicalized child-directed speech (without identifiable verbal content) and children can modify their behavior accordingly before the acquisition of the lexical-syntactic code, which appears around the age of 2 years (Boysson-Bardies 1996).

In adults, although some researchers consider that verbal content is more involved in understanding the interaction than emotional prosody (Kotz and Paulmann 2007), studies of brain-damaged patients show that emotional prosody plays an important role in understanding the emotional content (Monnot et al. 2003; Beaucousin et al. 2006), especially for social coordination (Beaucousin et al. 2007; Dickey et al. 2010). The inability to produce or understand emotional prosody, i.e. aprosody (not to be confused with dysprosody, see Box 7.2), leads to a definite disadvantage in interactions with others, insofar as a loss of family support⁷ or a loss of profession⁸ can be observed (Heilman et al. 1975; Ross 2000). Thus, in adults, emotional prosody is used to interpret communication intentions. However, case studies of patients with aprosody have been the subject of only 18 scientific publications⁹ since the first case reported in 1947 by Monrad-Krohn, as compared to more than 6,000 for aphasia. Currently, aprosody is rarely assessed, due to the lack of standardized tests in different languages such as French (to our knowledge, only Elliot Ross’s Aprosodia Battery exists in English; translated but non-standardized versions are used in other languages, raising the question of the quality of translations). Moreover, a neutral prosody during an interview with a health professional, such as a psychologist, or at the beginning of an illness, can be interpreted, sometimes wrongly, as a distancing in order to protect oneself against an emotional outburst (copying strategy).

But, beyond providing valuable information on the context of enunciation, the voice transmits clues that contribute to the recognition of the interlocutor. Indeed, it is essential to know who we are dealing with in order to achieve completely the objective of each person during the interaction.

7.2. Learning to recognize the other person through their voice

Some authors refer to the voice as a sound face (Révis 2017). The face and voice provide valuable elements for speaker identification, and even a redundancy allowing for ambiguities to be removed in degraded interaction conditions (noise or darkness) (Campanella and Belin 2007). We have seen that the voice has only integrated a place in recognition models very late on (Campanella and Belin 2007). Why did this happen?

The first well-documented case of phonagnosia in the literature is recent, dating from 2009 (Garrido et al. 2009), although similar cases had been described before (Neuner and Schweinberger 2000). As we have seen in the introduction to this chapter, phonagnosia is the inability to recognize a person’s voice without any sensory impairment. Unlike the cases described in 2000, the 2009 case is described through a battery of assessments that show a selective impairment in recognizing a (famous and/or familiar) person from their voice. This patient is able to identify environmental sounds, other types of information from the voice, such as the gender of the speaker, or emotions conveyed by the voice, indicating that primary auditory processing is preserved, as well as secondary auditory processing. Furthermore, this case of congenital phonagnosia (without identifiable neurological impairment in adulthood) indicates that identifying a person by voice is a separate process from identifying faces, as this person is perfectly capable of identifying the faces of celebrities, people around them or even people they did not know before (ability to learn to recognize new faces). What impact does this alteration have on the person’s daily life? The article mentions very little information on this subject: the patient is nevertheless described as having a “lifelong social problem” (Garrido et al. 2009). She takes an antidepressant, but the article does not mention whether this is related to her social interaction problem (indication not mentioned). The authors mention that the patient came to the clinic because, upon discovering the description of prosopagnosia¹⁰, she rightly thought that she had found an explanation for her problem. We can therefore legitimately infer that this problem causes a certain form of handicap in her daily life to lead her to seek information and consultation.

Thus, the voice can be used to identify a person. It provides both perceptual (pitch, intensity, etc.) and semantic (person’s name, verbal content, emotional information, etc.) information (O’Mahony and Newell 2012). As with faces, perceptual information forms a unique signature, i.e. unique to each individual (except for monozygotic twins). This is not the case for semantic information, which is variable and not unique to a given individual (O’Mahony and Newell 2012). Moreover, we saw earlier that decoding of this semantic information is acquired later (between 1 and 2 years) than perceptual decoding (possible before birth because perceptual systems are functional in utero).

So we are experts in perceptual decoding, but how does perceptual information allow for identifying an individual as unique? Since the last century, we have known that the voice carries information that identifies the gender, age and socio-cultural background of the speaker. Some authors refer to this as idiosyncratic prosody (Monrad-Krohn 1947). Indeed, depending on the individual’s size, their voice will be more or less acute: the smaller the individual is, the higher their voice will be because the vocal cords are shorter (Révis 2017). We have all experienced this phenomenon when speaking with children. Since women are smaller than men, it is also possible to understand why they also have a higher pitched voice than men.

Size is not the only factor explaining sexual vocal dimorphism; some studies also show that sex hormones modify the elasticity of the vocal cords (Révis 2017). Gender also seems to modulate other acoustical parameters, such as speech rate: women speak more slowly and articulate more than men.

The voice also evolves with age: we have seen that the child’s voice is higher pitched than the adult’s, ensuring a very good discrimination between these ages (much less so within these age groups). During normal aging, the change in voice differs between men and women: women’s voices become deeper and men’s higher than in young adults of the same sex. This phenomenon is called presbyopia and occurs around the age of 60 years (Révis 2017).

Finally, the voice bears the traces of a person’s individual history. For example, excessive smoking and/or alcohol consumption will lead to a voice that is known as “hoarse” in everyday language. The intonation nuances are less harmonious, the voice becomes deeper and deeper and a loss of vocal power can be observed. These substances alter the elasticity of the vocal cords (Révis 2017). But beyond these effects, the voice also conveys territorial and national accents that sometimes identify the geographical origin or socio-cultural belonging of interlocutors. We have seen that dysprosody can lead to the emergence of an accent that is not developmentally specific to a given socio-educational environment (Box 7.2) (Monrad-Krohn 1947). While this information can be changed voluntarily, it is more difficult for biometric information (age, sex).

This information, decoded from the voice, ensures the interlocutor’s recognition. But is it effective? The recognition rate of voice identity is low, compared to facial identity recognition, but a higher inter-individual variability is observed (von Kriegstein 2012; Schweinberger et al. 2014; Révis 2017). However, apart from our own voice, which we find very difficult to identify when exposed to it via a recording, our ability to recognize familiar voices increases with the frequency of exposure (Kensinger and Corkin 2004) and the duration of the presented recording. The recognition rate increases from a chance threshold to over 66% when a sentence is spoken rather than a word (Ladefoged and Ladefoged 1980).

A final element can enhance this recognition ability: the presence of a face (Belin et al. 2004; Campanella and Belin 2007; von Kriegstein 2012; Schweinberger et al. 2014). The ability to correctly associate a voice and a face develops between 4 and 7 months of life (Campanella and Belin 2007). It is therefore an extremely early learning ability. In adults, one study shows that the presence of a face can serve as a primer for recognizing a person’s voice (O’Mahony and Newell 2012). Indeed, we make more errors and go more slowly to identify a voice when it is presented with a face that has not been learned as belonging to that same person. On the other hand, the presentation of a face previously associated with a voice will subsequently facilitate its recognition. This is true even if the voice-face association is new (voice and face unknown to the participant before the experiment). This effect is not present if, instead of the face, the first name is presented with the voice (O’Mahony and Newell 2012). This beneficial effect of faces on speech recognition may be selectively achieved in prosopagnosia, even though the presence of a face in these patients may improve language comprehension (von Kriegstein 2012). Moreover, the voice-face association is robust and rapid, as a brief simultaneous presentation is sufficient for the two to be linked (von Kriegstein et al. 2005; von Kriegstein and Giraud 2006).

To date, there are two models of person recognition that integrate speech and facial information (Campanella and Belin 2007; von Kriegstein 2012). Both models are supported by behavioral, brain imaging and neuropsychological data, but here we will only detail the cognitive processes. The main difference between the two models is that one postulates that sensory decoding will activate the individual’s memory trace (Campanella and Belin 2007), while the other assumes that percept will lead to the formation of predictions about the origin of sensory information and will constrain the interpretation of the sensory signal with respect to certain signal regularities (von Kriegstein 2012). Regardless of the model, identification is based on the decoding of speech information on the one hand and facial information on the other hand. These are then integrated to activate the speaker’s identity (Figure 7.2). It should be noted, however, that although identification is based on idiosyncratic information (allowing the speaker’s age, gender, etc., to be determined), there is a distinction between the decoding and identification stages since patients with phonagnosia or prosopagnosia are able to decode this information. The decoding of vocal and facial information takes place in distinct brain regions¹¹ but in parallel (before 100 ms). The networking of these regions leads to the identification around 170–220 ms.

COMMENT ON FIGURE 7.3. – During an exchange between two people, the speaker will perceive the interlocutor’s face and voice. The speaker will define a facial and vocal identity from the visual analysis of the face and the auditory analysis of the voice, which will allow them to recognize their speaker. At each level of analysis, interactions can be observed between the specific face processing pathway and the voice processing pathway, even in the case of a unimodal interaction (such as a telephone conversation).

Finally, it seems that the ability to recognize an interlocutor increases when they are encountered in a strong emotional context (Kensinger and Corkin 2004). The importance of emotions conveyed by the voice is not yet well integrated into existing models but we will see that it could be of importance in the case of memory pathologies.

7.3. Emotions and the recognition of others

During a social interaction, a lived experience encoding will modify subsequent experiences (Feldman Barrett et al. 2011). Voice can therefore help define an enunciation context that will modify future interactions because it conveys different types of information, as we have just seen. We will first define what we mean by context. There are several types of context, but in this chapter we will only be interested in the one related to the environment (or the experimental stimulation). It is called “stimulus-dependent context” (Feldman Barrett et al. 2011). For example, we were able to show that gaze contact with a face alters the memory of the face and the judgment of the individual (Lopis et al. 2017). This stimulus-dependent context evoked by emotional language stimuli, including voice, can alter the memory trace of a face and the person’s judgment.

Emotional expression from the face is a prolific field of research with a long history (for a review of the literature, see George (2013)). But the study of emotional contextual effects on facial perception is more recent and still very underdeveloped (for a review, see Feldman Barrett et al. (2007) and Wieser and Brosch (2012)). Most work has focused on emotional facial expressions but very little has focused on emotional cues outside of faces. Yet these can alter the way faces are perceived, judged and remembered. For example, the presence of emotional prosody facilitates the decoding of emotional facial expressions (Schwartz and Pell 2012). This facilitative effect is likely related to the activation of emotion conveyed by the voice (Hietanen et al. 1998; Magnee et al. 2007).

But beyond the emotional evaluation, a linguistic emotional context can modify the neural trace of the face previously perceived in the emotional context evoked by the voice (Morel et al. 2012) or evoked in writing (Todorov et al. 2007). In particular, early effects evoked by a vocal emotional context have been observed through audio recordings of voices evoking emotional situations (Beaucousin et al. 2006, 2007). Participants in a magnetoencephalography (MEG) protocol perceived faces expressing no facial emotion while listening to these recordings. When these same faces were presented a second time alone (without recordings), we observed that the neural trace of faces perceived after a single presentation in a positive context was different from that of faces presented in a neutral or negative context around 50 ms (Morel et al. 2012). Thus, the single presentation between an emotional voice and a face is sufficient to alter early brain activity in response to that face presented alone. Why was this change in prior experience only observed for the positive context? We hypothesized that a negative context, because of its perceptual salience, may have diverted attention from the face (Vuilleumier 2005). Moreover, positive emotions have a particular social salience for humans, promoting the association of emotional voice and face (Sander et al. 2005; Grandjean et al. 2008). For the first time, extrinsic stimulus-dependent context effects on the face were demonstrated in early steps. Prior to this work, it was accepted that emotional processes could modify the perception of the visual environment as early as 100 ms after stimulus presentation (Galli et al. 2006; Pourtois et al. 2012; Wieser and Brosch 2012), but now we know that these modifications appear as early as 50 ms (Morel et al. 2012).

However, it remains important to know whether these early brain effects shape people’s behavior. Indeed, a language context is considered as an ecological context because it is part of our daily experiences (Feldman Barrett et al. 2007). However, the experimental constraints, especially in MEG (large number of trials, positioning of the participant in a chair with a heavy device to place the magnetometers around the head, etc.), do not allow direct inference of possible impacts at the behavioral level. By reproducing this protocol but outside of MEG, Shasha Morel was able to show that this prior experience modifies the judgment of the person’s mood (Morel 2009). Thus, faces previously perceived in a positive vocal context are judged as being in a good mood, whereas a neutral or negative context induces an evaluation of bad mood on faces that do not express any emotional expression (Morel 2009). This effect remains present even if the person does not voluntarily report having noted the association between the face and the voice.

These results show that the information conveyed by a face is not only constructed from the immediate context of presentation (Feldman Barrett 2011). It is not only the emotional meaning evoked by the face that is constructed from the context but also the value of that face for the individual. This value constructed by the stimulus and its environment thus comes to modify the judgment on this person. Thus, it seems conceivable to use the emotional context to modify the value of a person’s face in order to improve its learning (Box 7.3), as is the case for other non-verbal social cues (Lopis et al. 2017). But further studies will need to confirm these observations in order to generalize the results, before a new and more integrative model can be proposed.

7.4. Conclusion

The voice carries different types of information: idiosyncratic, emotional and linguistic. This information actively participates in the recognition of individuals. In particular, the emotional dimension of the voice could play a contextual role in facilitating the memorization of people’s identity, thus opening up paths to remediation program for memory pathologies (e.g. Alzheimer’s disease) or communication pathologies (e.g. schizophrenia).