7/20/2016 By Laura Schmitt, CS @ ILLINOIS
Written by By Laura Schmitt, CS @ ILLINOIS
When CS Associate Professor Brian Bailey began teaching the department’s user interface design course (CS 465) nearly a dozen years ago, 20 students were enrolled in the class. Today, more than 200 students take the class, where they work on self-selected projects like mobile interaction, web design, interactive games, or product design.
As enrollments increased over the years, Bailey found it increasingly difficult to provide timely feedback to the students on their concepts and designs—a critical step in successfully completing a project. “What we’d teach the students is to produce an initial concept as soon as they can, show it to stakeholders, and try to get feedback,” said Bailey. “The problem is in an isolated classroom, students turn to the TA or instructor for feedback, and there’s too many students to do that at scale.”
An expert on crowdsourcing and design innovation, Bailey had previously developed a web service that allowed people to get feedback on visual designs from either social media or a paid online audience. He wanted to explore whether this type of crowdsourcing solution could address the scaling problem in his course.
“The crowd-based systems give us an opportunity to connect with potential stakeholders at scale and very quickly,” Bailey said. “We saw this type of crowd feedback system as a chance for students to acquire feedback on their projects whenever they want and as often as they want.”
Bailey will test his crowd-based approach thanks to a recently awarded $1.3 million NSF cyber-learning grant that he shares with faculty colleagues Steven Dow at Carnegie Mellon University and Elizabeth Gerber at Northwestern University. They are investigating whether leveraging online crowds through social media and paid work platforms can be used effectively to provide formative feedback in the classroom and enhance design learning and innovation.
Bailey and his collaborators are developing a Crowd-Aided Feedback Technology (CRAFT) and studying its effect on design learning. “There aren’t any principles on how to do [crowd] feedback optimally, and no one has studied the crowd’s influence on the design process,” said Bailey, who has found some evidence that his earlier crowd feedback system led to improved designs.
Crowd-based feedback has been popular for about five years, according to Bailey, but there’s a shortage of knowledge on how to acquire, understand, and reflect on it. Students will upload their designs through the CRAFT website, provide information on project goals, and explain where they are in the design process.
One goal of the research is to determine which type of crowd provides the most valuable feedback and which point in the design process is optimal for soliciting feedback. A second goal is to translate the feedback into an action plan by using annotation tools that groups the reviews into categories so that the students can understand the aggregate feedback more quickly.
“We’ll be looking at new methods for making sense of the feedback,” said Bailey. “[We] can begin to cluster the feedback . . . from various sources and over time you can think more carefully about important things, and remind yourself of things that you wanted to do.”
Finally, Bailey said, CRAFT will enable the creation of a design canvas, where designers can construct a visual representation of their creative process. “The crowd would have access to the designer’s history,” he said. “They wouldn’t just see what someone had designed now. They’d see different iterations of the design up to the present.”
Not only will educators have access to CRAFT, but designers and engineers anywhere will be able to access it online. “The students can continue to use it, but people pursuing entrepreneurial activity, hobbies, or their own professional interests can also get feedback on their ideas.”