Penguin egg

An emperor penguin.

My laboratory is the Science of Imagination Laboratory. In it, we try to recreate the ability humans have to imagine things. Yeah, it's not easy. 

We think that when a person imagines a visual scene, what they are doing is drawing upon their visual memory-- all of the memories of all the things they've ever seen. So if you've seen roads beneath cars a lot, when called upon to imagine a car, you're likely to imagine a road beneath it. 

Here's the problem for modeling this on a computer: how do we get a computer model of everything a person's ever seen? One way we tackle this problem is by using pictures on the web. We assume that the pictures represent a typical person's visual memory (or close enough). We have databases of images that are labeled. This means that for a given picture, we know what's in it, and where in the picture it is. This took a huge amout of work; luckily other people did it. 

But we want 3D information too, because we want our programs to be able to imagine three-dimensional scenes. But there are no libraries of 3D scenes that are labeled. Or are there?

My Ph.D. student Sterling Somers came up with a brilliant idea: to mine 3D worlds people have created for information about what's out there. Sure, a lot of made-up worlds are sci-fi or fantasy, but a whole lot of them are things like living rooms, and penguins, and other real-world stuff. So we are now working on a program that takes in these 3D worlds people have created and looks for regularities-- what tends to be in a bathroom? What objects appear with apples? The program asks those kinds of questions. So (in the future) when we ask the program to imagine a sink, it will have an idea of what goes next to it.

One problem is that not all of these 3D scenes are labeled. Some of the objects are labeled in ways that are problematic. Some might be "chair," which is great. Others might be "chair23" which is not hard to deal with. Others might be "foldingChair" which might require us to use another knowledge base so the program can understand class-subclass relationships. The worst is when it has no meaningful name at all, like "jjdk67588." What to do then? We need yet another program to try to figure out what that thing is. 

We suspect that it might be easier than computer vision, which works from pixels. We already have the 3D structure of the object (this is normally a big job in computer vision) because they were created using 3D structure software in the first place. Perhaps we can use some kind of learning process to use labeled 3D models to identify unlabeled 3D models. 

That's a longer introduction than I'd planned to give.

Anyway, I have a student working on this (possibly impossible) problem in my lab. At one of our meetings he noted that some things, such as chairs, are best described functionally-- it's a chair if you can sit on it. We talked about the possibility of using functional descriptions to identify objects. But what about penguins? What are their functions? 

In the context of this problem, they have no function. You might be able to identify tables and other artifacts functionally, but some things are what they are because of how they look. 

Philosophically, though, it got me thinking about the function of a penguin, or an ocelot, and I figured out that it's not that penguins have no function-- penguins have lots of them. 

The an ecosystem, a penguin clearly has a function, as all animals that evolved in that ecosystem do. Penguins keep fish populations under control by eating them, and they feed seals and orcas, but not polar bears*, by being eaten by them. 

A penguin might have a function in its family-- to get food, or to keep an egg warm. 

Thinking about an evolutionary system, the function of a penguin is to turn food into more penguins. 

They don't appear to have a human function, like a chair does, unless you're a hunter who plans to eat them. 

So things have lots of functions-- but each one is relative to a particular system. 

But as far as our system is concerned, penguins are penguins because of how they look. They have wings, and are black and white colored. Right? 

Pictured: An emperor penguin. From Wikimedia Commons.

Jim Davies


* There are no penguins at the north pole, and there are no polar bears at the south.

You are reading

The Science of Imagination

How We Can Catch a Ball When We Don't Know Where It Will Be

Our minds have two tracks, when it comes to physical motion

Might We Live in a Simulated World? (A Response to Manzotti)

It depends on what you consider to be really real.