7/11/2013 Mike Koon, College of Engineering
Written by Mike Koon, College of Engineering
The look home entertainment continues to evolve, but few innovations in the field have been more talked about in the last few months than Microsoft’s IllumiRoom project, where CS graduate student Brett Jones is one of the principal researchers.
IllumiRoom is “a proof-of-concept system that augments the area surrounding a television with peripheral projected illusions to enhance traditional gaming viewing experiences.”
IllumiRoom is one of many major research projects Microsoft Research is presenting at this week’s Association for Computer Machinery (ACM) SIGCHI (Special Interest Group on Computer-Human Interaction) Conference in Paris, France.
Watch “IllumiRoom: Peripheral Projected Illusions for Interactive Experiences” video.
As outlined in the research paper, IllumiRoom utilizes an ultra-wide field of view projector and a Kinect sensor that sits on a flat surface to create a virtual 3D model of your room. The goal of the project is to virtually project an immersive experience out of the screen, bringing the entire room into the experience. According to IllumiRoom’s research video, “those illusions could distort reality, extend field of view, and enable entirely new experiences, such as those in gaming or film.” This could be done in several ways at the player’s discretion. For instance, it could change the appearance of the room into a cartoon world or extend the content from the screen (or just some of the elements) into the room.
“IllumiRoom can turn a forty-inch TV into a fifteen-foot TV, so you can have a more immersive experience,” Jones said. “What’s more exciting and interesting is that you can mix virtual and physical things in your living room, a process called augmented reality. The experience no longer just exists in just your TV. It exists in your living room as well.”
The industry got its first look at IllumiRoom in January through a Samsung keynote at the annual Computer Electronics Show (CES) in Las Vegas. Since that time, the technology industry has imagined how it could be applied and has generated a lot of social media discussion.
“It’s like running a user study with a couple of million people weighing in with ideas,” Jones said.
Tuesday Jones and his collaborator Hrvoje Benko, a researcher at Microsoft Research, shared more details of the IllumiRoom project at the CHI conference. To read more, visit http://research.microsoft.com or download the full IllumiRoom paper.
“IllumiRoom has created a lot of buzz and generated a discussion about what living room entertainment could look like in five years,” Jones said.
IllumiRoom automatically identifies where the television screen is and determines the color, shape, and dimensions of the items in your room. It then uses a process called radiometric compensation to adjust the projection, so it can work with the existing furniture in your room. It can apply that information in many forms, such as making it look like it’s snowing with the snow accumulating on the furniture. To do that, it uses a real-time physics simulation.
Jones, Benko, research manager Andy Wilson, and senior researcher Eyal Ofek conceived the idea during a brainstorming session while Jones was interning with Microsoft Research last summer. All four principals had been involved in other projects involving projection space or as Jones puts it, “The idea of taking your computer out of this little box that sits on your desk and making it interact with things in your environment.”
Jones worked on some early video prototypes of IllumiRoom, getting feedback along the way, and constantly added more to the system, producing several iterations throughout the summer.
“Brett was instrumental in bringing IllumiRoom to life during his internship last summer,” Benko said of Jones’ involvement. “It was a super ambitious project for a twelve-week internship, and like most research projects at the beginning, we had no idea whether it was going to work or not. Brett did a fabulous job. Incorporating different game engines into the experience, and contributing to the camera-projector calibration, and real time effects pipeline.”
Although the research video mostly shows a first-person shooter game experience, the research team envisions how the concept could be applied to other genres, like education and sports. For instance, Jones said a designer could make a “Dora the Explorer” game where Dora asks players to find a backpack somewhere else in the room. Applications could be used for sporting events on television where the scoreboard box that usually appears in the corner of the screen instead appears behind the television along with a Twitter feed, creating an even greater interactive encounter and allowing for the actual high-definition screen to include just the video itself.
“We were interested in this global concept shift where having this in your living room changes everything from playing a board game with your family to watching TV to enhancing your gaming experience,” Jones said. “We also wanted to make it practical. It needed to be easy to use for both the consumer and the designer.”
A native of Homewood, Illinois, Jones is now in his ninth year at Illinois, earning both bachelor’s (2008) and master’s degrees (2010) in computer science from the university. He has been working under Professor David Forsyth in the area of computer vision and Professor Brian Bailey in the area of human-computer interaction (HCI).
Each year, Microsoft Research invites about 700 PhD-level interns from around the world, many of whom come from the University of Illinois, to apply research and theory into practice. Jones is returning to Microsoft Research headquarters in May to continue his work there before completing his PhD at Illinois in 2014.
“What is amazing about a project like this, as each piece came into place, we kept being wowed by the experience and subsequently getting more ideas about how to push it further,” Benko added. “It was a super exciting twelve-week collaboration, and I am really looking forward to working with Brett again this summer,”
“The experience has been kind of crazy because this started out as just an internship project,” Jones said. “All of a sudden it’s at CES and everyone is talking about it. It’s been neat to get the research out there and have people react to it. I am excited to see where it leads.”