University of Illinois at Urbana-Champaign
The Grainger College of Engineering
Computer Science

Suleyman Ates

Suleyman Ates

Suleyman Ates

Home Institution
Middle East Technical University

Year Participated
2022

Year in School
Undergraduate

REU Faculty Mentor
Reyhaneh Jabbarvand

Research Area Interest
Artificial Intelligence

Project Title
ML Pipeline Testing

Biography & Research Abstract

Abstract:

Developers of ML libraries often write end-to-end/integration tests that check the validity of the ML pipeline – this includes loading the data, initializing the model/algorithm, training the model, and asserting that some metrics are above a pre-selected threshold. To make testing cost-efficient, developers often use a smaller/toy version of the data (selected from the training dataset). However, it is currently unknown if the choice of data is optimal, i.e., whether it is sufficient to test that the model is learning something useful and catches potential “accuracy” bugs. Hence, the goal of the project is to study and understand the role of training “data” used for testing various ML algorithms/models and their implementations. We aim to study and validate the following Hypotheses/Questions:

  1. Given a (training) dataset D for a model, can we derive a smaller dataset D’ that has (1) a similar accuracy (say within ~5%) as original data, (2) has high fault-detection ability, and (3) can be trained with within a reasonable time (for CI)?
  2. Can we validate whether the model is learning something “useful” (e.g. via some relevant metrics) with the small dataset (derived by us/developer) used in the test?
  3. Can we develop data attacks that perturb the existing dataset and evaluate how that impacts the test in terms of, for instance, (1) its passing probability, (2) fault-detection ability, and (3) code coverage?
  4. Can we develop some interpretability metrics with respect to how the models are tested? 

Bio:

My name is Süleyman AteÅŸ, and I am from Turkey. I am currently sophomore METU Computer Engineering student. I love computer science and programming, and I have deep interests in the field of deep learning. Other than that, I like watching and playing football, and trekking with my friends at our campus. I like meeting new people, and getting to learn about the cultures and traditions that are new to me.