Reconstructing Invisible Objects: (Synthesis Lectures on Computer Vision)

The problem of modeling and reconstructing the "invisibles", e.g., specular or transparent objects such as 3D fluid wavefront and gas flows, has attracted much attention in recent years. Successful solutions can benefit numerous applications in oceanology, fluid mechanism and computer graphics as well as lead to new insights towards shape reconstruction algorithms. The problem, however, is inherently difficult for a number of reasons. First such objects do not have their own image. Instead, they borrow appearance from nearby diffuse objects. Second, determining the light path within these objects for shape reconstruction is non-trivial since refraction or reflection non-linearly alters the light paths. Finally, dynamic specular or transparent objects often exhibit spatially and temporally varying distortions that are hard to correct. Existing solutions can be essentially viewed as a special class of correspondence matching algorithms. Often a known pattern such as a checkerboard is positioned near the surface and conceptually one can analyze the corresponding feature points in the observed cameras views and then apply stereo or volumetric reconstruction techniques. In this book, we discuss a broad range of classical solutions as well as the emerging approaches based computational cameras and displays. Classical Approaches. Most previous solutions build upon point-pixel correspondences where a special planar pattern such as a checkerboard is placed near the surface and a single or multiple cameras are used to acquire the distorted pattern for shape reconstruction. A common issue in point-pixel based solutions is ambiguity: a pixel corresponds to a ray from the camera while the specular surface can lie at any position along the ray. We discuss state-of-the-art techniques by adding additional constraints such as smoothness, integrability, multi-view consistency, etc., for resolving this ambiguity. We also discuss in depth the cons and pros of each method. Computational Imaging. An emerging class of solutions aims to use novel computational cameras and displays to directly resolve the point-pixel ambiguity by acquiring ray-ray correspondences. In this book, we discuss five recent approaches: 1) the light path triangulation method by Kutulakos and Steger for recovering static specular objects, 2) the light field probe approaches by Wetzstein et al. and Ye et al. for acquiring both static and dynamic specular objects, 3) the Bokode approach by Ye et al. for reconstructing dynamic fluid wavefronts, and 4) the multi-camera approaches by Atcheson et al. and by Ding et al. for recovering 3D gas flows and wavefronts.