6/9/2022 Aaron Seidlitz, Illinois CS
Beginning in January, Ji officially became part of a project led by Columbia University, involving four different institutions, and funded by DARPA.
Written by Aaron Seidlitz, Illinois CS
For the past six months, Illinois Computer Science professor Heng Ji began work as part of a collaborative research effort designed to produce a system called CHARM – which stands for Cross-Cultural Harmony through Affect and Response Mediation.
After preparing for the project proposal in October of 2021 and learning of its funding through DARPA shortly thereafter, Ji started working under the lead of Kathleen McKeown, a longtime collaborator and mentor from Columbia University.
The primary goal of this effort is to “use theoretical frameworks of cross-cultural communication and entrainment, to guide the selection of communicative behavior indicators associated with sociocultural norms and communication failure.”
DARPA and the Department of Defense (DoD) funded this effort with the goal that CHARM will utilize artificial intelligence to help deliver better “communicative understanding.” The resulting automated systems could assist interpreters and others, whose discussions often fall short of understanding and proper practice of social norms through misinterpretations.
After detecting the instances in which communicative norms were violated, the system can also suggest alternative appropriate responses for conversations.
“What we have done thus far is try to collect some compositional video, which can include anything from international negotiations to movies and TV shows that discuss and analyze cultural stereotypes,” Ji said. “The idea is to use our models to find the pattern and norms needed in these conversations. The second task will be applying those norms to a model that detects places in new conversations that violates those norms.
“So, in the future, we can imagine users having a device they carry with them that allows them to communicate properly with other cultures through this system.”
The first phase began in January of 2022, and includes Ji’s expertise in multi-lingual, multimedia knowledge extraction.
She has teamed with a group that has McKeown as its lead and includes:
- Bo Feng, University of California Davis
- He He, NYU
- Julia Hirschberg, Columbia University
- Colin Wayne Leach, Columbia University
- Smaranda Muresan, Columbia University
- Michael Picheny, NYU
- Owen Rambow, Stony Brook University
- Carl Vondrick, Columbia University
- Changxi Zheng, Columbia University
Ji said the team first analyzed two days of meetings between governmental officials from the United State and China that took place on March 18-19, 2021.
These meetings in Alaska caused a stir as both parties’ dialogue grew adversarial. In instances like this, Ji and the team of collaborators believe that indicators from the CHARM system could help involved parties converse better.
“What we noticed when studying the Alaska meetings were two teams that both became very angry, and then they both veered from the agenda,” Ji said. “The US team began talking about human rights and then the Chinese team took the conversation another direction entirely. The point of our multimodal monitoring is to not only account for norms in dialogue but help provide reminders of what proper protocol is.
“Without these reminders, there are so many slight differences that can be taken the wrong way when two cultures do not have much familiarity with each other.”
While this portion of the initiative represents the inspiration detailed by sponsoring parties like DARPA and the DoD, Ji also has personal motivations for the research expanding beyond this need.
She could also see the work delivering a helpful educational tool for children and students adapting to new cultures.
Herself a Chinese student who came to the US to study at New York University, Ji said her first workgroup as a graduate student was a diverse set. Almost all 30-plus members were from different countries, resulting in a group of individuals who accepted others for their differences to ensure proper communication.
However, when moving to other areas of the country, Ji and her family came to understand it wasn’t always that way.
In those instances, a device and model like CHARM could help others better understand each other.
“I’m excited and proud to be collaborating with this team and with Kathleen McKeown once again,” Ji said. “We have several people who are technically strong – from those that focus on computer vision, others who focus on language processing and, more still, who do speech recognition in real time. I primarily analyze the content and extract knowledge from it.
“All together, this is an excellent group to be working with, and a great goal to be working toward.”