In my last blog on this subject, I reported on some fascinating work by Uri Hasson, Rafael Malach, and their co-workers. Cutting the Gordian knot of neuroscientific hypothesis and prediction, they simply put their subjects in an MRI scanner and compared their brains as the subjects watched different movies. They measured how much one could predict one subject's brain activity from another's: intersubject correlation (ISC). In other words, how alike were these people responding? And that tells you how much control the moviemaker had.
I'd sum that up by saying the viewers' brains behaved alike at the level of sensory processing and simple comprehension of the plot of the film. But in later experiments, this group refined those findings.
They compared eye movements while viewers watched a segment of Sergio Leone's spaghetti western, The Good, the Bad, and the Ugly, and a 10-minute, unedited one-camera video of people milling around in Washington Square Park. They found that gaze was more tightly correlated among viewers, that is, more tightly controlled by the director, for Leone's edited film. The feature film produced lots of ISC, the unedited film, not much.
They then compared ISC for purely visual film and for purely aural storytelling. They observed brains responding first, to Chaplin's City Lights with no sound track and second, to a segment of an audio book of Alice in Wonderland. They found high ISC in the visual areas for the visual film and high ISC in the auditory regions for the audiobook. In effect we process the sight and sound of media in separate channels.
The separateness of the channels suggests to me how we can accept non-diegetic music. We filmgoers can believe in a symphony orchestra playing while the screen shows only a barren landscape or a soundtrack of ominous theremin music while the blond starlet opens the door in the haunted house. We can believe because we process sight and sound in different channels.
At the same time, though, Hasson's group found that polymodal areas, regions of the brain where sight and sound come together (superior temporal sulcus, tempoparietal junction, and intraparietal sulcus) were involved in integrating these visual and auditory streams of information. The barren plain becomes majestic because of the symphony orchestra, the door scary because of the theremin, but we don't think the door or the plain is making the sound we hear. We integrate sight and sound, but in a rationally controlled way.
The researchers now tried movies chopped up into different time scales, long-term and short-term. These were pieces of two Chaplin silents presented in their original form and as scrambled sequences of 36-second segments, 12-second segments, and 4-second segments. All these sequences showed high correlation (ISC) in the processing of instantaneous information within single shots, but only the long, 36-second sequences showed high ISC in cognitive functions. As a result, the experimenters could give us pictures of the brain regions that are putting the movie together into a coherent story (the brains' supplying things like causation and motivation). The subjects did this even if the film clip was only 36 seconds long. (Regions involved were the lateral sulcus, tempoparietal junction, and frontal eye fields.)
The researchers also found some indication of brain regions involved in integrating information from one 12-second shot to the next (think Eisensteinian montage): lateral occipital cortex, parahippocampal place area, fusiform face area, superior temporal sulcus, and precuneus, but these are regions whose functioning is less well understood. It's interesting to speculate, though, as to what this experiment might say about the way we understand montage. It may be that one set of regions specialized for certain things (faces, places) and other regions (perhaps!) involved in theory of mind, that is, understanding what is going on in someone else's mind, that these two different kinds of systems combine to create meaning from montage.
Another important factor: the experimenters found high correlation among viewers' eye movements. as they watched all these film segments. We are programmed, as all animals are, to pay attention to whatever is new in our environment. A director, by focusing first on one character or then on another, can effectively control what we pay attention to. But it takes more from the director to control our cognitive and emotional responses.
The experimenters explored this issue by playing the same movie forward and backward, They found high correlations (ISC) of brain activity with eye movements either way, playing the film forward or backward, but much smaller ISC in regions involved in synthesizing the movie (causation, motivation, and so on) when the movie was played backward. As anyone who has taught film knows, students agree at the simple visual level of what the screen shows, but it is at these more complicated readings of a film that students (and critics!) usually differ. These readings come from areas of the brain more involved in synthesizing perceptions toward eventual action (precuneus, lateral sulcus, temporoparietal junction and frontal eye fields).
What emerges, then, from Hasson and his group's work is a teasing apart of the different brain systems and their interactions that go into our watching a movie. I've covered only some of this work at this point, and I'll devote a third blog to it soon.
Psychological items I've referred to:
Hasson, Uri, et al. 2004. "Intersubject Synchronization of Cortical Activity During Natural Vision." Science 303.5664 (12 Mar 2004): 1634-40.
Hasson, Uri, et al. 2008. "A Hierarchy of Temporal Receptive Windows in Human Cortex." Journal of Neuroscience 28 (10): 2539-50.
