6/2/2021 Aaron Seidlitz, Illinois CS
Now in her third year of the Illinois CS PhD program, Li produced an event graph schema model that predicts future events through several applications.
Written by Aaron Seidlitz, Illinois CS
Manling Li is now in her third year as a PhD student at Illinois CS. Before even applying to doctorate programs, though, she considered her current adviser, professor Heng Ji, a role model.
The first reason being the way Ji cared about and encouraged female students to pursue ambitious career paths in computer science. Li also had a positive early impression of the professor’s research in Data and Information Systems. Li found it to be comprehensive and focused on real world applications addressing unsolved problems.
To study under Ji at “one of the most prestigious PhD programs in the world” was an opportunity Li gladly accepted. Their efforts together first focused on Ji’s work to expand information extraction beyond text-based information and into multimedia information.
But a more current project has earned Li recognition validating her path forward in academia. With Ji’s guidance, Li has showcased her expertise with a new event graph schema model. This model seeks to discover the knowledge needed from available data to predict events that will happen next.
The professor said that Li has been responsible for about 90 percent of the work on this project. Their group includes two other Illinois CS students and researchers from New York University, University of Southern California, US Naval Academy, and the Army Research Laboratory.
To obtain historical events to support event graph schema model, Li leads students from the group and Columbia University to develop a multimedia event extraction system. In 2019 and 2020 this effort produced a multimedia event graph ranked first in a Streaming Multimedia Knowledge Base Population evaluation at the 2019 and 2020 Text Analysis Conference through the National Institute of Standards and Technology. It also earned the Best Demonstration Paper Award from the 2020 meeting of the Association for Computational Linguistics.
“I can tell that our work together has been very exciting to Manling, because it can make a difference on the lives we live,” Ji said. “I can see the connection she has with the work, and, as a result, she is becoming an expert on graph modeling and graph mining.”
Li credits their approach and execution to the guidance she received from Ji.
“Professor Ji has always believed that a good professor is a reflection of her students and their success,” Li said. “She exerts her best effort to make the student visible in the community. I am truly grateful to have her as my adviser. Working with her not only built my basic skills of writing, presentation, and communication, but also helped me determine how my values and goals align with a faculty career.”
To prove the validity of their model, Li and the co-authors compared its performance to a human schema.
Ji said that they asked people participating to write down what they thought would happen next when provided a certain scenario.
Of course, Ji said, there are several limitations to human schema. First, human knowledge limitations meant they could not make all the potential connections for a fair prediction. Second, they were not using data-driven knowledge, thus unable to produce probabilities for what happened next.
Meanwhile, the researchers’ event schema model works off an acute method. This begins with an initial graph that shows, for example, when an attack occurred and how many people died. The model then intakes all previous articles and multimedia instances from similar events.
From there, it deduces similar issues related to the event. The result is a timeline of the event that not only looks back at what happened but also predicts future events based off this occurrence.
“The whole idea here is that history tends to repeat itself,” Ji said. “We can look into the events that have happened throughout history and summarize the patterns – how events relate to each other, how they evolve over time. Then, hopefully, when we are facing a new scenario or complex event, we can predict what will happen next.
“From there, we can work toward preventing bad things from happening.”
Both Li and her professor see political applications as possibilities for this event schema. But Ji also believes that it can benefit different efforts, like infectious disease management.
Its benefit in a scenario the COVID-19 pandemic includes predictions that can help outcomes. It can identify supply issues at a grocery store, for example, because people aren't eating at restaurants. Or it can estimate the impact on local businesses due to previous financial decision.
Beyond that, Li sees application possibilities benefiting misinformation detection techniques. This can also enhance information extraction techniques.
Regarding political applications, events such as shifts in Europe can trigger the "international conflict" schema. Li said one such instance they investigated was the Ukraine declining to join the EU in 2013.
Their model uncovered a couple different impacts. First, "evidence of Russian influence would suggest a Ukrainian revolution, or pro-Russian unrest with respective probabilities.” Additionally, she said, the model identified the "president's removal by parliament." It’s forecast for the situation predicted civil unrest and additional Russian involvement.
“A typical question from analysts would be ‘Can you anticipate Russians' reactions to Ukraine's decision not to join the EU?’ This requires an event understanding system to match events to schema representations and reason about what might happen next,” Li said.
As the project continues to progress, Ji believes their next step includes involving multilingual and multimedia data. This will help produce an even more fair and complete prediction.
Li plans to continue striving for her PhD with a goal of becoming faculty herself one day. Furthering this process, Li was recently selected as one of the next Mavis Future Faculty Fellows through The Grainger College of Engineering.
She also earned this year’s C.L. and Jane W.-S. Liu Award from Illinois CS.
“I am determined to be a faculty to work on multimedia event extraction and reasoning,” Li said. “I envision a future in which I can lead my research team to overcome the knowledge understanding barrier of different modalities. I want to work toward the realization that machines are capable of taking full advantage of multimedia data to obtain both physical and abstract knowledge jointly, such as multimedia event understanding.”