Autism App Targets the "Holy Grail" of Communication

The first app to use artificial intelligence to power an alternative and augmentative communication (AAC) tool for profoundly autistic and other minimal- and non-speaking users was recently unveiled by Howard Shane, a professor at Harvard Medical School and the former director of the Autism Language Program and Center for Communication Enhancement at Children’s Hospital in Boston.

QuickPic AAC was created by Shane and Christina Yu from Boston Children’s Hospital and Mauricio Fontana de Vargas, a human-computer interaction (HCI) research scientist. The app facilitates the instantaneous generation of topic displays—sets of vocabulary relevant to a particular image—that used to take teachers and speech language pathologists up to half an hour to create. For example, if a therapist wanted her client to work on building sentences and answering questions about a birthday party, QuickPic could search the web for a photograph of a child blowing out birthday candles or even upload a photograph from that child's birthday party. Then AI would interpret the image and arrange it on a display alongside picture icons representing relevant words the client might choose, such as girl, boy, bake, cut, light, blow, yummy, cake, etc. (see accompanying image for an example). A facial recognition feature allows the inclusion of friends and family members.

“AI doesn’t make the sentence,” Shane explained to me in an interview. “The child still has to choose from the icons to build a sentence.”

For profoundly autistic children—those whose vocabularies are typically dominated by nouns—this can be very difficult. But it’s a critical skill. Shane notes that research has found that syntax is “like the holy grail… If we can teach autistic kids syntax, we can greatly expand their ability to understand and use language.” Since many autistic kids have better visual than auditory processing, tools like topic displays can be particularly effective. Building them through an app not only frees up the time of speech language pathologists for more direct client interaction, but it increases accessibility to parents, teachers, and others who may want to generate new topic displays to promote conversation in more natural settings.

A sample topic display

Source: Howard Shane

Just the Beginning

Perhaps this seems like an underwhelming use of AI—after all, QuickPic AAC is streamlining an existing tool, not creating something completely new. But Shane believes that AI will “revolutionize the field.”

“As you know, AAC can create a synthetic voice that speaks,” he said. “AI will be able to use the measurements of your vocal tract to manifest what your voice would sound like if you were able to talk. It will also be able to take some basic words chosen by the user—for example, grandma, train, thank you—to generate a coherent message.”

An increasing role for AI is not without its risks, however. As one of the longest and most vocal critics of facilitated communication—a debunked intervention in which a non-disabled facilitator helps a minimally- or non-speaking person spell out words on a letterboard or keyboard—Shane knows how easy it is for authorship to be hijacked.

“Obviously, it’s essential to ensure that autistic users are the ones actually creating the input,” he emphasized. “But too much is at stake not to integrate AI into AAC. There are few things more life-changing than giving someone who couldn’t communicate before the ability to do so.”