Perceiving Stickiness in Augmented Reality

Many aspects of human perception are multisensory. For example, the sound we hear when a ball is moving can affect how we interpret its path (Sekuler et al., 1997). Conversely, visual cues such as someone's mouth movements can modify how we hear a spoken syllable (known as the McGurk effect; McGurk & MacDonald, 1976). However, it may seem that certain sensations, like the stickiness of an object, would be primarily tied to the tactile modality: It seems that we sense the stickiness of things by testing how it feels to interact with them and subject them to physical deformations.

Virtual stickiness: a pseudo-tactile sensation

A recent study published in the May issue of Perception challenges this notion by showing that when visual cues are mapped onto a person's finger movements, they can evoke sensations of stickiness, even without the accompanying tactile stimulation. Furthermore, the perceived magnitude of stickiness can be systematically manipulated by adjusting the "break distance" of a virtual object, or how far it takes for an object to "break" in response to particular finger movements.

Researchers Takahiro Kawabe and colleagues used Augmented Reality (AR) to conduct their experiments. Participants placed their hand on a desk and observed a video feed of their hand on a screen in front of them, appearing around the same size as their real hand. Using finger tracking software, the experimenters could calculate the distance between the participant's thumb and index finger in real time and use this information to project a virtual deforming object onto the video feed. The object consisted of two white disks joined together when participants' index finger and thumb were touching. As participants separated their fingers, the two white disks' connection became continually thinner until, at a certain point, they broke distance and the two disks separated completely. Here is a video demonstration of the experimental paradigm.

Stickiness relates to break distance

The researchers found that perceived stickiness tracked continuously with the break distance of the object. In other words, a highly sticky object might require a large distance, perhaps several centimeters between the fingers, before it breaks into two, whereas a less sticky object would reach the break point sooner, for example with an inter-finger distance of just one centimeter.

In two subsequent experiments, the researchers demonstrated the robustness of this effect. First, they showed that the same stickiness illusion could be produced on a smartphone, using the rear camera to capture the hand and having participants look at the rendered object on the phone screen. Second, they showed that the experimental manipulations of virtual stickiness corresponded well to the perceived stickiness of real-life substances like dish soap (low stickiness), ketchup (medium stickiness), and honey (high stickiness).

This intriguing study shows that even sensations that we consider to be fundamentally tactile in nature can be induced without real tactile stimulation. The sensation of stickiness can be recreated by establishing a correspondence between performed finger movements and deformations in the shape of a virtual object. By manipulating how much the virtual object can stretch before it breaks, observers acquire a pseudo-tactile sensation of stickiness. This finding adds to a growing literature that highlights the multisensory nature of perception and the power of augmented reality to simulate sensory experiences.