Prof. Karahalios Tests Speech Visualization for Autistic Children

4/28/2009

CS prof devise a way to digitize conversations and spit them back as images that let people "see" on monitors

Written by

Prof. Karrie Karahalios has devised a way to digitize conversations and spit them back as images that let people "see" their own conversations on computer monitors. The computer program, which she calls a "conversation clock," has been tested with low-functioning autistic children and in marriage counseling and is being prepared for use with Asperger's Syndrome. Her work was recently profiled by the Washington Post.

Her tool, called a Conversation Clock, visualizes conversations on a computer terminal as vibrant colors -- red, yellow, blue, green -- the image growing in size if the voice gets louder, overlapping another color as it interrupts or abruptly narrowing with silence.

Karahalios says her method provides feedback in real time and can act as a type of social mirror, allowing people to adjust their speech in the same way they adjust their appearance before a glass mirror.

"You look into a mirror and you change your dress, your expression, because you see exactly how it's happening in real time," she said.

The colors linger so people can see the progression of an entire conversation, not only the present moment.

Karahalios will begin work on a trial of the technology to see if children with Asperger's Syndrome might be able to learn new conversational patterns of speech, and to help them overcome the monologing or lecturing behavios associated with the disorder.

Karahalios describes a test of a previous version of her Conversation Clock in which four people sat at a table, each fitted with a lapel microphone matched to a color and attached to a computer. The conversation clock charts a colorful, real-time record of their conversation that is projected on the table, instead of on a wall or a computer screen.

The "clock" shows the progress of the talk. Three times a second, a color bar pops up showing who was speaking. The louder the speech, the longer the bar. Interruptions are shown as overlapping color bars. Every minute, a new circle of bars is rendered in a visual record akin to the rings of tree trunk.

Karahalios found that once it was turned on, people tried to balance their conversation.

"Everyone sees the exact same thing and people almost felt like it had to be a balance of color," Karahalios said. "I'm not saying that balanced conversation is good conversation. It's fascinating how people behaved."

Once she had established her techniques, Karahalios worked to apply them to children with autism.

Research has shown computers can teach in ways that reduce human-to-human interaction, which tends to be a source of anxiety to people with autism. That came into play in an experiment Karahalios did with some low-functioning autistic children, age 3 and 7, who did not talk.

The idea was to help children "vocalize," using sounds that are the basis of speech instead of screams or grunts.

When a child watching the computer monitor vocalized, a cartoon character would spin, or a cone would erupt in fireworks. The longer the vocalization the greater the reaction: the cartoon character spun more quickly, or more fireworks appeared.

As for audio, the computer would give the child an "echo," a short sound for a short sound, a longer one for a longer sound.

"We got him to say 10 words, which his mother had never gotten him to say," said Joshua Hailpern, a doctoral candidate working with Karahalios and two other professors.

Karahalios hopes for the day her work can become widely available to help children and families, through iPhone programs and as toys for children. A toy for autistic children who have trouble working with doctors, or who are far away from specialized help, could give "rewards" to kids as they play with the toy and make sounds.

"We don't want it to look like a small computer," she said. "It needs to be inviting ... fun and engaging."

Karahalios presented her research at the recent eComm conference.


Share this story

This story was published April 28, 2009.