Illinois led DARPA Project Earns $5.8 Million to Advance Conceptual Learning

Illinois CS professor Heng Ji will lead a multi-university, interdisciplinary team that includes fellow departmental professors Derek Hoiem, Karrie Karahalios, and Camille Cobb to develop MIRACLE: Multimodal InteRActive Conceptual Learning.

After years of research focused on natural language processing (NLP) within Artificial Intelligence (AI), Illinois Computer Science professor Heng Ji has become a trusted source to answer some of the most relevant questions about the proper development and application of natural language processing. 

Ji aims to implement a new concept learning framework for her next project. She will do so as lead investigator for a new three-year grant worth $5.8 million from DARPA’s Environment-driven Conceptual Learning (ECOLE) program. Through this work, Ji has formed a research team to develop “MIRACLE: Multimodal InteRActive Conceptual Learning” and discover “comprehensive, global-scale schemata of objects and events for open-world recognition of images and videos.”

“Part of the reason why this project is close to my heart is because we are introducing a new way of addressing a problem that we’ve become increasingly aware of,” Ji said. “It’s kind of funny, but its inspiration comes from years of watching kids learn languages beginning at about six months old. I have since wanted to know why humans can learn concepts and multiple languages so quickly compared to a machine. 

“I’ve thought something is wrong with the current vision-language learning paradigm, and we believe MIRACLE can be a solution.”

DARPA values it during a time in which automatic analysis of large amounts of multimodal streaming data from diverse sources is crucial for situational awareness of events around the world.

To approach the problem, Ji has formed an interdisciplinary and multi-university team with co-PIs professors Carl Vondrick of Columbia University, Mohit Bansal of the University of North Carolina Chapel Hill, and Nanyun Peng of UCLA. 

Within Illinois CS, professor and computer vision expert Derek Hoiem joins the team and fellow professor and expert in human-computer interaction (HCI), Karrie Karahalios. Additionally, Illinois CS professor Camille Cobb will present another HCI expert with experience researching security and privacy issues.

The project includes three main tasks:

• A distributed interactive curriculum learning framework

• Unsupervised hierarchical multimodal representation learning, knowledge reasoning and association

• Attribute and affordance discovery and description generation through human-machine collaborative analysis

“This entire project builds off previous work on a smaller project I’ve conducted with professor Vondrick, for DARPA called GAILA,” Ji said. “We did some work on learning concepts, but that was limited to certain domains. This work has expanded to an open domain goal, and I’m so proud of the team we have built. I aimed for training and supporting next generation of young woman PIs. The MIRACLE PIs are primarily a very young group and there are six female Co-PIs. The technical ability of each member is extraordinary and brings an energy to this effort that is so exciting to work alongside.”

As the team works to build MIRACLE, Ji and her collaborators, remain engaged with several potential applications.

Among potential applications, Ji sees potential within education to create an AI tutor to help children with learning problems. She also forecasts the possibility of MIRACLE working well with autonomous cars to enhance safety and reliability.

Finally, Ji has allowed herself to think even bigger about this project and the impact it can have on the computing community.

“MIRACLE has the potential to change an entrenched paradigm in the sense that it will not require a lot of human annotation,” Ji said. “Most of the current systems are very expensive because they need a lot of machines to train and a lot of annotated data to form examples. So, only a few large research groups can do this kind of work.

“But we believe that if we make MIRACLE work the way we envision, then the AI research can be ‘greener’ because the development cost will dramatically decrease. We would love to be able to democratize work like this moving forward.”

DARPA has also shared news of this award: “Class is in Session: Teams Selected to Teach AI Agents to Interact with People & Learn”.