Like other soft tissues of the body, brains don't generally fossilize. So in reconstructing the evolutionary history of brains, in various species and lineages, scientists largely rely on indirect evidence. Such evidence is obtained by studying the anatomy or biochemistry of animals living today, and in uncovering the common pathways of embryonic development and life history as they are expressed in a wide range of species. "Comparative neurobiology" is a powerful and effective tool for understanding brain evolution as it has been expressed in species ranging from the common lamprey to bald eagles to human beings (see Georg Striedter's Principles of Brain Evolution, Sinauer Associates, 2005, for a comprehensive overview of the field).
Comparative neurobiology provides us with a broad overview of the patterns of brain evolution, but sometimes we are more interested in a specific twig of the evolutionary tree. The evolution of our own brain represents just one such twig on the tree, but it is a twig that, naturally enough, is particularly interesting to us. By comparing our brains to those of our closest relatives, the other primates (including monkeys and apes), we can find the ways in which our brains are just another primate brain and the ways in which our brain has diverged from the primate pattern during the 6 million years or so since we split off from our closest relative, the chimpanzees.
But we do not just want to know how our brains are different from those of the other animals, we want to know how those differences arose. The key questions are: What were the critical events during human brain evolution over the past 6 million years and when did they occur? The comparative perspective is less valuable in answering these questions, because although it provides a framework for understanding human brain evolution and identifying the unique features of our brain, it does not inform us about the unique events and contexts in which human brain evolution occurred.
So this gets us back to the unfortunate reality that brains do not fossilize. The fossil record gives us information about the time and place for critical events in human evolution, but if brains do not fossilize, how can we use paleontology to learn more about the evolution of our brain? Well, as many of you are probably already aware, the fossil record does a pretty good job of preserving at least one critical measure of the brain--its size. One of the most striking and unique (at least compared to our close relatives) characteristics of our brain is its large size. Cranial capacity is simply the volume of space contained within the cranium (the part of the skull housing the brain). As you might expect, brain size is pretty well-correlated with cranial capacity, so we can look at the cranial capacities of our ancestors and get a good idea of where, when, and how brain size changed over time. In a nutshell, so to speak, until about 2 million years ago, the brains of our ancestors were essentially the same size as those of the great apes today; however, starting at that time, a trend towards increased cranial capacity began, culminating in the modern human brain that is about three times the size of a great ape brain. This was not necessarily a smooth trend, and the rate of brain size increase was not steady as it was expressed in several distinct extinct species of the genus Homo.
Size isn't everything, of course. The human brain differs from the brains of other primates in many aspects of its functional organization: it is not simply a chimpanzee's brain blown up to three times its size. As anthropologist Ralph Holloway has argued for many years, changes in the functional organization of the brain have undoubtedly been as important as changes in the size of the brain. Thus we should not necessarily regard those first 4 million years of human evolution for which we have no evidence of a substantial increase in size as being a time when there was no brain evolution. The problem is in finding evidence of these changes in organization. One possible, but contentious, source of information comes from casts made of the inner surface of the cranium. These are called "endocasts," and sometimes they form naturally, although they are most often made by paleoanthropologists (formerly using some sort of molding material, but now they are made virtually via CT scans). Some endocasts preserve information about some of the sulci and gyri on the surface of the brain, and in some cases these patterns may correlate to the functional organization of brain. The interpretation of endocasts is fraught with controversy, however, given that even under the best of circumstances, the preservation of surface brain structures is highly variable. Combine this with the fact that the fossil material from which the endocasts are made may themselves be more or less complete, and it is easy to see why it is difficult to make any definitive statements about brain organization based on endocasts. Nonetheless, endocasts are a kind of "hard" evidence about brain evolution, and even if imperfect, they can be quite striking and evocative remnants of our biological past (see R. Holloway, D.C. Broadfield, and M.S. Yuan, The Human Fossil Record Volume Three: Brain Endocasts, The Paleoneurological Evidence, Wiley-Liss, 2004).
In contrast to brains, teeth are the part (or parts) of the body that are most likely to become fossils. They are very hard, and not particularly appetizing to predators or scavengers. Compared to other parts of the skeleton, paleoanthropologists find a relatively large number of teeth and jaws, and they spend quite a bit of time thinking about and analyzing teeth in order to get the most information possible out of them. Is it then possible that fossilized teeth can tell us something about brain evolution? Strangely enough, the answer to that question appears to be "yes."
There is one aspect of brain reorganization that has occurred during human evolution, which is readily apparent to everyone. Humans are right-handed. Well, not everyone, but a good 90-95% of people are genetically predisposed to becoming right-handed. Motor and sensory control of body is mediated in the opposite side of the brain (contralateral control), therefore handedness reflects one aspect of functional brain organization. Historically, the left side of the brain has been considered to be "dominant" to the right, reflecting not only handedness but also the fact that speech production is also typically most obviously represented in the left hemisphere. Although such notions of cerebral dominance are now generally regarded as overly-simplistic, they do reflect real differences between the hemispheres in terms of their functional roles. Intensive studies of other primates demonstrate that there is no such species-wide bias towards right-handedness in any other primate, although individuals can develop hand preferences and there may be some biases towards right-handedness for some tasks (for different perspectives, see W.D. Hopkins and C. Cantalupo, Laterality 10:65-80, 2005, versus L.F Marchant and W.C. McGrew, Primates 48:22-26, 2007). Even if other primates are right-handed for some tasks, the degree to which humans are right-handed far exceeds the level seen in any other primate species.
At some point over the past 6 million years, the human brain underwent an organizational change that is now reflected in a species-wide, genetically-mediated predisposition towards right-handedness. What can teeth tell us about this process? In northern Spain, there is an extraordinary fossil site that has been excavated since the 1990s. Known as Sima de los Huesos (the "bone pit"), it is dated to 500-600,000 years of age, and has yielded the remains of at least 28 individuals from over 4500 fossil specimens. These individuals have been ascribed to a species called Homo heidelbergensis (also known less formally as "archaic Homo sapiens"). H. heidelbergensis is very human-like in many ways, but the bones of its skull were much thicker and more robust than in modern humans and the shape of its cranium was long and low, with a sloping rather than upright forehead ending at rather massive browridges. Its cranial capacity was on the order of 1100 to 1390cc, close to, but somewhat smaller than seen in modern humans or Neandertals.
Homo heidelbergensis used stone tools, although they were not as sophisticated as those used by modern humans or Neandertals. At the Sima de los Huesos site, only a single stone tool has been recovered, which may be consistent with the interpretation that it was a burial pit of some kind. Despite the absence of stone tools, the dental remains from the site offer us an intriguing perspective on the behavior and functional brain anatomy of the members of this population. Marina Lozano and her colleagues have recently published a fascinating study of the microwear patterns (visible only under microscopic examination) found on the surface of teeth found at Sima de los Huesos (M. Lozano, M. Mosquera, J.M. Bermudez de Castro, J.L. Arsuaga, and E. Carbonell, Journal of Human Evolution 30:369-376, 2009). Based on earlier tooth wear analyses, it is generally thought that some earlier hominid species used their mouths as a third hand, holding materials with their teeth while they cut them using stone tools. Via experimental studies, Lozano and her colleagues have found that stone tools make characteristic scratch marks on dental enamel; they hypothesized that if the Sima de los Huesos individuals were holding materials in their teeth while cutting them with a stone tool, then such striations should also appear on the surface of certain teeth (namely the incisors and canines). This is exactly what they found when they examined the dental remains. Furthermore, the orientation of the striations on the teeth suggest that none of the 20 individuals they examined was left-handed and that the vast majority had striations consistent with right-handedness.
The study by Lozano and colleagues therefore strongly suggests that right-handedness, and the brain organization that goes along with it, was present as a high-frequency trait in an ancestral hominid species at least 500,000 years ago. Although analyses of chipping patterns of stone tools from 1.5 million years ago suggest that their makers were right-handed, this study provides the earliest somatic evidence of handedness in an ancestral human species. Integrating this discovery into the broader picture of human brain evolution remains to be done, however this window into brain evolution provided by the teeth puts us a little closer to understanding functional changes in the brain as they occurred along with the evolution of increasing brain size.