The Science of Emotion From the Inside Out

“Do you ever look at someone and wonder: What is going on inside their head?” Joy asks in the opening line of Pixar’s film Inside Out.

Inside Out is the story of 11-year-old Riley and her emotions: Joy, Sadness, Fear, Anger, and Disgust. When Riley moves with her family to California, her emotions struggle to keep her happy, and when Joy and Sadness accidentally get sucked out of Riley’s Headquarters her emotions are thrown into chaos. Can Joy and Sadness learn to work together to find their way through Riley’s mind and back to Headquarters? Can Riley come to terms with the complex emotions of growing up and learn that sometimes it’s okay to be sad?

As with most Pixar movies, Inside Out is a beautifully entertaining family film whilst also being subtly thought-provoking. Viewers of all ages are encouraged to reflect on their own emotions and consider the ways in which our emotions shape who we are and how we take ownership of them.

As such, Inside Out was a perfect fit for the Sci/Film series we run at Birkbeck using movies to provoke public discussion about science. We recently ran a special screening of the film in conjunction with the Birkbeck Institute for the Moving Image and Children’s Film club, funded by Wellcome Trust and King’s College London where children had a chance to participate in interactive games and science studies. After the film, we had an audience Q&A about how we learn to recognize emotions in ourselves and others.

In this post, we cover some of the topics about the development of emotions discussed during the event and how they are featured in the movie.

Can babies feel and recognize emotions?

Inside Out begins with Riley as a baby and her first emotion, Joy. As the other emotions gradually appear, life gets a whole lot more complicated!

This mirrors how emotions really emerge in infancy, with more complex emotions such as anger or frustration requiring a baby first to be able to build up an expectation for what is going to happen — "every time I throw this toy Mum and Dad pick it up and give it back to me," and second to remember that this expectation has been violated — "Mum and Dad aren’t picking up my toy like usual." Anger and frustration, possibly followed by sadness, ensue.

We know that babies love to look at faces from the second they are born. And young babies are able to discriminate between different emotional expressions. Montague and Walker-Andrews (2001) used the peek-a-boo game, but in addition to the typical happy/surprised expression, they also showed babies anger, sadness, and fear. Four-month-olds could tell the different emotions apart, looking more than usual when faces showed anger or fear, and significantly less than usual when faces were sad.

How do we recognize other people’s emotions?

In order to recognize emotional facial expressions, it’s important to pay attention to key features such as the eyes and mouth. Different features are more relevant for different expressions, for example, the large white of the eyes indicates fear, whereas a smiling mouth suggests happiness.

In the movie, the filmmakers have used exaggerated facial features to emphasize the characteristics of joy, sadness, anger, disgust, and fear.

To explore how different parts of the face convey emotions with the children at our event, we gave them a face template and they chose different facial features (e.g., smiling mouth, eyes with teardrop) together with different angles for the eyebrows to create different emotional expressions.

We can also see how children view these different cues by using a technique called eye-tracking. This allows us to record where a person is looking using high-speed infrared cameras.

During our event, we set up an eye-tracking tent in the foyer of the cinema and recorded how people watched the trailer for Inside Out. The data demonstrated how quickly our eyes seek out faces in each shot of the trailer, making very rapid movements (called saccades) after every cut to find a face, scan its features, identify what the character is feeling and saying and then look at what the character is interacting with. (For more discussion on how we watch movies you can read the blog post and Smith, 2012).

Despite infants’ very early capacity to discriminate between different emotional expressions, the ability to accurately label expressions continues to develop throughout childhood into adolescence.

Batty and Taylor (2006), looking into the neural responses of occipitotemporal areas of the brain, (which are critical for understanding other people’s facial expressions) show continuously changing patterns from early childhood into adolescence, with the adult pattern of neural responses only emerging in late adolescence. This has a consequence on children’s understanding of emotions. For example, children often mistake negative emotions such as anger or sadness with disgust.

We know context also plays an important role in emotion recognition. For example, a teen girl screaming at a concert for her favorite pop star may have the same facial expression as somebody scared by a ghoul on a ghost train, but her feelings are of extreme joy, not fear.

At our event, we ran a small study to demonstrate this emotional context effect. Our Inside Out Emotional Stroop Task was a twist on the classic Stroop Task. In the original task, participants are asked to name the font color of a printed word. This task is much harder than you might think because you need to inhibit reading the words themselves in order to say the color. When the required response, the color of the ink, does not match the meaning of the printed word (e.g., “GREEN” printed in red ink) there is interference making people much slower to respond.

We created an Inside Out themed version of this task: Children were presented with pictures of joy or sadness feeling either happy and sad. They had to respond as quickly as possible with the emotion that the characters were expressing. As expected, performance was worse (showing longer and less accurate responses) when the character identity did not match their emotion, i.e. when joy was sad and sadness was happy.

How does a movie guide our attention to the key emotions in a scene?

Real life is complex. We are often bombarded by a wealth of confusing and conflicting sights and sounds, which we have to learn to sift through in order to pick out which details are relevant to a particular task e.g., to understand how someone is feeling.

In the real world, emotional facial expressions may be fleeting and occur in a busy scene with many other competing distractions such as traffic passing in the street, the siren of an ambulance or our own overriding interest in the fox that just leaped over a fence. We have to learn to inhibit these distractions to focus our attention on the subtle but socially rich emotional cues in other people’s behavior.

An area of the brain call prefrontal cortex plays an important role in both inhibitory control and our ability to regulate and control emotions, and this region, as well as its connections to other parts of the brain (like the amygdala which is particularly important for processing fear), are still developing into adolescence.

In film, directors have developed a cinematic language that they can use to simplify the task of inhibiting distracting information to focus on information relevant to a particular goal.

At the beginning of film history, directors would simply shoot a scene from a single distant camera position, capturing all the action in one take. Very rapidly, audiences got tired of not being able to see the subtle details of a scene, such as what a character was holding, or most importantly, their facial expressions. Directors learned to edit together multiple viewpoints of a scene (known as shots), gradually moving the camera closer to the object of interest at key moments to accentuate the drama, emotion and empathy audiences felt for a character.

A key technique used to communicate how a character feels about something is the shot/reverse-shot technique. A character is shown looking at something in a long shot (known as an establishing shot, as it establishes character locations) and then a cut to a close-up of the object they are looking at allows the viewer to adopt the point of view of the character. Critically, we then cut back to the character’s face, now in a close-up so we can see their emotional reaction.

This editing pattern creates and answers a series of questions in the mind of the viewer: What is Joy looking at and what does she think and feel about it? The viewer becomes cognitively engaged with the action of the scene and emotionally involved with the character. A scene that could have been emotionally confusing from a distant viewpoint is now completely unambiguous and universally understood (for more discussion of how “universal” film comprehension is, see the work of Sermin Ildirar here and here).

Inside Out is a fantastic portrayal of how our ability to feel emotions and recognize them in other people develops from infancy through to adolescence. Of course, the filmmakers had to take some artistic license when portraying the inner workings of Riley’s mind — we don’t actually have little people in our heads watching what we watch (see the homunculus fallacy) — but these storytelling tricks are essential to ensure young viewers understand the film’s difficult themes about emotional development and hopefully provoke families to discuss what constitutes a healthy emotional life.

Guest contributors: Tim J. Smith, Ines Mares, Ana Maria Portugal, Sermin Ildirar and Claire Essex.