Monocular 3D Shape Recovery

Abstract

Recovering the 3D shape of a nonrigid surface from a single viewpoint is known to be both ambiguous and challenging. Resolving the ambiguities typically requires prior knowledge about the most likely deformations that the surface may undergo. Here we extend the Laplacian formalism, which was first introduced in the Graphics  community to regularize  3D meshes, to achieve this goal given  correspondences with  a reference image in which the shape is known.

Our  approach   allows  us   to  reduce  the   dimensionality  of   the  surface reconstruction problem without sacrificing accuracy, thus allowing for real-time implementations.

The algorithm runs in realtime on tablets. It can detect which deformable object appears in the camera stream, then track and reconstruct the corresponding 3D shape as the object deforms over time.

We also use our algorithm to build an augmented reality coloring book App, which runs in realtime on tablets. The App tracks a possibly non-planar drawing page, precisely takes its colors to texture a virtual animated character. This work was done in colloboration with Disney Research Zurich.

Results

Real-Time application on Tablets

Real-Time Demo on PC

Modeling the Impact of a Baseball

3D reconstruction of the baseball during the collision with a bat

On-contact vertices during the impact

Code and Supplementary Material

Code and instructions for running it are available here.

Supplementary material can be downloaded as a zip-file supplementary.zip or viewed on a webpage README.html.

References