Picture in Picture: Adding 3D Objects to 2D Images

Inserting objects into existing photographs or videos is extremely difficult because the object's appearance depends on the complex lighting and geometry of the environment.  Researchers from University of Illinois show how to realistically render 3D objects into an existing photograph, requiring no special equipment and much less effort from the artist than previous approaches. 

Researchers at Illinois have shown how get the lighting right, by combining methods from computer vision, human computer interaction, and computer graphics.   A user who wanted to insert an object or an animation into a picture would sketch the main layout of the space (or use a computer vision program to estimate it) and the lights using an interface, and use a standard modeling tool to make the object or the animation.

“Detecting photo fraud is often easy, because it is very difficult to ensure that the new object has the right shadows, highlights, and shading; casts shadows in the right places; and is reflected in mirrors,” said computer science professor David Forsyth.  “In contrast, experimental work conducted at Illinois suggests that this new system (a) produces accurate estimates of shading, tested by comparison with real photographs and (b) fools human observers into believing that an edited image is real. “

The technical breakthrough is getting the lighting right by taking advantage of small amounts of annotation to recover a simplistic model of geometry and the position, shape, and intensity of light sources. The team’s approach applies a computer vision program to estimate strong directional sources of light like the sun through windows;  applies a computer vision program to estimate how much light the surfaces in the picture reflect, and to correct the user sketch of light; and then uses a standard computer graphics program to generate the picture from detailed physical models of light behavior. 

In addition to the overall system, the research team also developed a semiautomatic algorithm for estimating a physical lighting model from a single image. The Illinois method is able to generate a full lighting model that is demonstrated to be physically meaningful through a ground truth evaluation. As part of their work, the team also introduced a state-of-the-art image decomposition algorithm for single image reflectance estimation. 

“With a single image and a small amount of annotation, our method creates a physical model of the scene that is suitable for realistically rendering synthetic objects with diffuse, specular, and even glowing materials, while accounting for lighting interactions between the objects and the scene,” said computer science PhD student Kevin Karsch, lead author of the approach. “Our approach has applications in the movie and gaming industry, as well as home decorating and user content creation, among others. Imagine being able to slay dragons right in your own living room.”

Kevin’s work, joint with Varsha Hedau, Derek Hoiem, and David Forsyth, will appear in this year’s SIGGRAPH Asia conference.   The authors’ version of the paper can be found at http://kevinkarsch.com/publications/sa11.html, and they have released a video at http://vimeo.com/28962540.  They plan to produce a web-server version of this application in the near future.