Your brain on movies

You might think it would be easy to see how our brains function while we are watching a movie. Just hook some viewers up to an electroencephalograph or a magnetic resonance imager (MRI) and see what happens when they watch a movie. But whoever said it would be easy?

Neuroscientists themselves make it hard by insisting on tightly focused hypotheses and predictions within narrowly controlled experimental parameters. Movies aren't like that.

Car chases, weeping lovers, affectionate dogs—a given movie might have a million different stimuli. Neuroscientists call these "dynamic natural scenes." And how would you frame a hypothesis or a prediction given all these different stimuli?

Yet that's what we'd like to know about, how we humans process natural scenes, not the artificially constrained stimuli of the laboratory. And, for a film buff like me, I'd like to know how our brains process ordinary feature films, not some film manufactured in a laboratory for experimental purposes.

One researcher and his team have sidestepped that difficulty. Uri Hasson began working on movies with Rafael Malach at the Weizmann Institute in Israel. They produced one seminal paper in the field in 2004.

What Hasson and Malach did in Israel was put the usual method of hypothesis-prediction-experiment-conclusion aside. Instead, they simply had five subjects watch the first 30 minutes of The Good, The Bad, and the Ugly, the Sergio Leone spaghetti western starring Clint Eastwood, while lying in an MRI scanner.

In lieu of hypothesis and prediction, they asked: To what extent can we predict the activity of one viewer's brain from the activity in another viewer's brain? Intersubject correlation they call it, or ISC. In effect, they are measuring similarities in the viewers' brain activity millisecond by millisecond as they watched the movie.

What they found, not surprisingly, was that viewers' brains behaved alike in some respects and differently in others. What was interesting was the regional differences. Viewers' brains behaved alike (about 45 percent of the neocortex!) in the occipital, temporal, and parietal lobes—that is, regions involved in primary sensory perception plus some multimodal complex response functions. Specifically viewers' brains behaved alike (high ISC) in the primary visual areas of occipital and temporal cortex, Heschl's gyrus (auditory region), Wernicke's area (language processing), some limbic areas (emotion), the fusiform gyrus (face recognition), and the association cortices that partially integrate primary sensory data.

I'd sum that up by saying the viewers' brains behaved alike at the level of sensory processing and simple comprehension of the plot of the film. For example, the viewers' eye movements closely resembled one another's. I think of this as low-level processing. No subtle film criticism here, just a plot summary. (And, if you teach movies, as I have, you know that students and ordinary viewers often have trouble agreeing about "what happened.")

By contrast, the viewers' brains behaved differently (low ISC) when it came to the more advanced areas of information processing. They did not share activity in the supramarginal gyrus, the angular gyrus, and prefrontal areas. These “consistently failed to show intersubject coherence.” The first two of these are polymodal areas where our brains put perceptions together to achieve a complex perception of a whole environment. The prefrontal areas presumably directed the viewers’ emotional and intellectual understanding of what they were seeing.

Hence the experiment says something about film form: The form has a widely shared effect on viewers, but the total aesthetic experience will vary considerably from individual to individual. “The ‘collective’ coherence,” the experimenters conclude, “naturally divides the cortex into a system of areas that manifest an across-subject, stereotypical response to external world stimuli versus regions that are linked to unique, individual variations.”

Science Magazine's commentator on the paper adds: "Perhaps just as important as Hasson et al.’s findings about interbrain correlations is their observation that large regions of activated cortex could not be predicted from another individual’s brain activity. These regions include portions of the parietal cortex and, notably, the majority of the prefrontal cortex. Thus, there might be, after all, ample cortex for you and [me] to experience The Good, the Bad, and the Ugly in a unique way."

That was Israel, 2004. Since then, Hasson has moved on and currently works at Princeton and at New York University with its famous film school. (Spike Lee, Oliver Stone, Martin Scorsese, Jim Jarmusch are grads.)

Hasson and his team have done more experimenting with ISC using various kinds of films. Inevitably, they have focused on the brain activity that the viewers shared, not the far more challenging and perhaps impossible task of tracing viewers' individual responses. He sums up their work in an article in Projections: The Journal for Movies and Mind in the summer of 2008.

By the way, if you're interested in the psychology of responses to movies or just in movies and psychology, that's a journal you should know about. Edited by Ira Konigsberg at the University of Michigan, it publishes articles using psychological approaches that vary from the psychodynamic to heavy-duty neuropsychology like Hasson's experiments. I think it's indispensable for serious students of film.

In one of their subsequent experiments, Hasson had his subjects watch the Sergio Leone film and an unedited 10-minute clip of people at a concert in Washington Square, a New York park, just people milling around. The experimenters compared viewers' brains as they watched these two very different films. The clip had no editing, no camera movement, nothing of the sophisticated film techniques of The Good, the Bad, and the Ugly.

First, Hasson's group found that viewers' brain activity was "time-locked" to events on screen. That's not surprising. Our brains are designed, as all animals' brains are designed, to turn our attention to whatever is new in our environment. It could be something good for survival or sex, an evolutionary plus. We need to focus on it and, if need be, cope with it. Since a movie, even an unsophisticated unedited movie, is a motion picture. Because it is a moving picture, it constantly presents us with something new, and we pay attention to that new thing.

Second, with both films, viewers' brains behaved alike in some visual and auditory areas and in a region (lateral occipital cortex) active in object recognition. As with the earlier experiment, we viewers all process the basic sounds and sights of a film (even the unedited Washgton Square film) the same way. But there was a lot more ISC with the directed and edited film.

Conclusion: In order to control viewers' responses, you have to construct the film's sequence of images. We are not responding just to a particular image on the screen. We are responding to a sequence of images, a story. (Movies are, to my mind, the greatest method of telling a story since language itself.)

Third, the experimenters contrasted a purely visual narrative (Chaplin's City Lights with its soundtrack removed) and a purely auditory narrative, someone reading Alice in Wonderland. They found that the viewers' visual systems correlated during the movie, but not their auditory systems. And vice versa for the story reading: Their auditory systems correlated, but not their visual systems. By contrast, other regions (superior temporal sulcus, temporal-parietal junction, and part of the left intraparietal sulcus) showed activity shared by the viewers during both presentations.

As I read this result, what it implies is that at a primary sensory level, we process basic sights and sounds in movies independently of each other. But even when it comes to the two modalities' synthesis, processing sequences of events, for example, we still process the film like other viewers. In other words, Hasson's group is showing in more detail the similarity in viewers' responses at the basic levels of film form, sight vs. sound, camera angle and movement, and editing for sequence. He is getting more detail about two different ways of processing movies, like other viewers or according to our individual styles of response, that were so sharply differentiated in the 2004 experiment.

Hasson has gone on to other, more complex experiments dealing with the role of editing, for example. I want to blog about those, but at a week or two from now. I've gone on long enough in this posting. And we've got more opera this coming Saturday.