Generative AI in the Metaverse

Some assume that the metaverse will be a single immersive environment like in the movie Ready Player One (2018), while others, myself included, assert that real life is the metaverse, and we will use different interfaces throughout our day to combine our physical and digital worlds to varying degrees—from augmenting our reality with virtual overlays to immersive VR. [1] Regardless of the perspective, our social connections and overall well-being will depend on how well we can interact with each other and build relationships as the metaverse evolves. Generative AI will play a pivotal role in this transformation.

There has been significant discussion surrounding generative AI algorithms, such as ChatGPT for text generation and DALL-E for image generation. Recently, Jaron Lanier, a pioneer in the field of virtual reality, wrote an article for The New Yorker where he says that AI is just a tool used by people and that “the most accurate way to understand what we are building today is as an innovative form of social collaboration.” [2] The potential for the large language models (LLMs) that power generative AI to build dynamic content for social collaboration can accelerate the development of the metaverse beyond what we might have predicted just one year ago.

Imagine learning, playing, and working collaboratively with friends, family, and co-workers in a virtual environment but with the assistance of multiple LLMs to make our avatars more realistic and interactive.

The metaverse I envision fosters collaboration, communication, and connection with the virtual and mixed-reality worlds, adapting as we move through them.

Empathy Amplified

The metaverse could serve an important role in building empathy, which is key to promoting and encouraging social interaction. Empathy is understanding another person’s point of view or values, even if they are different than your own, and it has been referred to as a muscle that needs to be exercised. [3] Successful collaborating, whether for personal or professional reasons, is a form of social interaction that could be enhanced through generative AI that facilitates empathy.

For example, I might work in a mixed-reality environment with my colleagues on a collaborative design project, but one of the challenges we face in virtual and distributed collaboration is the absence of non-verbal cues and emotional nuances that form an integral part of face-to-face interactions. Generative AI has the potential to bridge this gap by imbuing virtual avatars with realistic emotions and body language, creating a more immersive and emotionally resonant experience. Through interacting with these AI-driven avatars, we can enhance our ability to recognize and respond to emotional cues, ultimately strengthening our empathy muscles.

As a bigger database of interactions forms with the goal of effective collaboration that builds empathy, the virtual worlds could adapt depending on the needs of an individual as well as the community. In a professional collaboration through the metaverse, standards of communication could be upheld by promoting input from everyone and enabling individuals to contribute equally. Participants with domineering personalities might learn to listen, while those who haven’t contributed could be put in the spotlight and empowered to lean in. A well-trained LLM could be the ultimate facilitator of a meeting.

Expanding Connections

In a recent interview, Apple’s Tim Cook said, “…the idea that you could overlay the physical world with things from the digital world could greatly enhance people’s communication, people’s connection… It could empower people to achieve things they couldn’t achieve before.” [4]

Connection and communication are critical for enhancing social interaction, and the metaverse presents us with an unprecedented opportunity to harness its potential as a powerful tool for forging these connections and enabling communication. By leveraging generative AI, we can create virtual communities and platforms that facilitate authentic connections and foster a sense of belonging.

Imagine a metaverse where AI-powered algorithms analyze user preferences, interests, and personalities to connect like-minded individuals in virtual social spaces. These carefully curated interactions would enable individuals to find companionship, support, and shared experiences, irrespective of physical distance.

Generative AI can also play a pivotal role in building bridges between generations. As the metaverse transcends age barriers, AI-generated characters can act as conduits for intergenerational communication, facilitating dialogue, understanding, and mutual learning between different age groups.

Engaging all of our senses will be critical if we want to expand our connections and social interaction through the metaverse. Otherwise, Zoom would do! But if we can apply LLMs not only to text and images but also to smell, sound, and touch, these connections made through the metaverse will be even more powerful when they engage all our senses. A recent announcement from Disney that it is building a multi-sensory communication platform, starting with touch feedback so that family and friends can interact, will help build the corpus of data needed to do this. [5]

Artificial Intelligence Essential Reads

AI Has Started Lying

AI Weakens Critical Thinking. This Is How to Rebuild It

Building the database of multi-sensory immersive virtual worlds is the biggest gap right now to the potential for LLMs to generate dynamic virtual worlds. Lanier says, “Right now, it’s much easier to get an LLM to write an essay than it is to ask the program to generate an interactive virtual-reality world because there are very few virtual worlds in existence.” This will change in the near future as hardware and software platforms stabilize and standardize. In the meantime, it’s imperative to explore how to use generative AI as a catalyst for improving communication and fostering stronger connections within the metaverse.