The assimilation of the idea of virtual reality into the popular culture over the past couple of decades has created a cozy protected niche in our collective thinking for a rather extreme idea: that the world we perceive and interact with is illusory. The irreality stance has been discussed by philosophers for millennia and has become a doctrinal mainstay of at least one major religion. As it happens, cognitive science has much to say about these matters, and I plan to explore its contibutions in the coming posts. But first things first.

Getting to the bottom of the irreality idea is bound to be a long slog, so it makes sense to start by seeking some instant intellectual gratification that would motivate us at the outset. This, it turns out, is very easy: all we have to do is have a fresh look at some well-known facts about visual perception in animals that, like us, possess two functional forward-facing eyes and that use their two eyes to better see depth.

Popular accounts—and, embarrassingly, most introductory psychology textbooks—describe depth perception as the recovery of 3D information, which is lost when the world is projected by the eye's lens onto the retina, from the 2D retinal image (of which, in a binocular system such as ours, there are two). It is hard to imagine a more misleading way of stating the problem of depth perception. True, depth is lost in the optical projection of the world onto the retina, but the signal that the retina creates out of the image that is projected onto it is in absolutely no relevant sense "two-dimensional." The retina transduces the image by means of a dense mosaic of photoreceptors, which are connected, via some pretty sophisticated circuitry, to a layer of ganglion cells—neurons that send their axons to the rest of the brain, in a million-strong bundle that is the optic nerve. The nominal dimensionality of the signal that the brain gets from each eye is, therefore, not 2 but 1,000,000 (in digital camera parlance, that's about a megapixel—not a biggie for the bundle of computational sophistication that is the human brain, but not a trivial matter either).

The computations that support binocular stereopsis—the process through which the brain integrates two sets of 1,000,000 numbers each (which, moreover, change every time we shift our gaze, which happens several times a second) to recover much of the lost depth information—are pretty well-understood and make for a fascinating story. Furthermore, stereopsis is quite an important behavioral cue: try to thread a needle, or hit a ship at sea with a boulder hurled from the shore, with one eye closed, and you'll see why. Still, the computational processes that constitute 3D vision are not my focus today. Let us therefore return to the question of the nature of perceived reality. Clearly, the two million-long lists of numbers together contain rich depth information (otherwise binocular depth perception would have been impossible). Note, however, that the world as you perceive it corresponds to a view from a vantage point that is identical neither to that of your left eye, nor to that of your right eye.

To ascertain that, hold your hand in front of you and focus on the tip of your finger. If you now close your left eye, the perceived location of the finger will shift to the left; likewise, if you close your right eye, the finger will seem to shift to the right. When you look at the world through both eyes, your perceived vantage point is between your eyes, a couple of centimeters behind the bridge of your nose—in a place where you don't, in fact, have an eye at all, and where in reality it is perpetually dark (unless your skull is cracked open). Thus, as Björn Merker pointed out in a 2007 paper, your phenomenal impression of the visual world, as seen from that vantage point, has no trace of the entire front portion of your skull, which, along with your nose, just does not seem to be there.

Polyphemus hurling a boulder at Odysseus' ship

The Cyclops Polyphemus hurling a boulder at Odysseus' ship

You may counter that, its phenomenally Cyclopean appearance aside, the content of your view of the world is definitely real, being simply the result of binocular fusion, which combines the disparate views of the two eyes into a single whole. A close investigation will reveal, however, that the visual impression of the world that you experience with both eyes wide-open differs in several respects from the actual contents of the two images registered by the eyes. The most important difference is, of course, that the binocularly perceived view contains an impression of depth, which the brain works hard to recover from the millions of numbers that it receives from the retinas; as always the case with mere metaphors that pose as explanations, "fusion" does not begin to account for the computations that make binocular stereopsis possible.

All this can only mean one thing: your normal view of the world is virtual, in the same literal sense that the image on the screen of a computer game console or a scene from Avatar is. First, it is computed from a variety of of data sources, only a few of which are derived from the immediate external reality. Second, it would not exist at all, were it not for the computations that gave rise to it. How far can or should we trust a reality that is decidedly virtual already in our very first encounter with it through the senses? I shall return to this question during the following weeks.

You are reading

The Happiness of Pursuit

AI We There Yet?

A behavior-science perspective on why it's still a long way to human-like AI

From Beginnings to Hope

The human mind/brain has what it takes to make the world a better place.

How to Make a Corporation Happy

If corporations are people, how likely are they to have human feelings?