An Object-Based Approach to Image-Based Rendering

This dissertation, "An Object-based Approach to Image-based Rendering" by Zhifeng, Gan, 甘智峰, was obtained from The University of Hong Kong (Pokfulam, Hong Kong) and is being sold pursuant to Creative Commons: Attribution 3.0 Hong Kong License. The content of this dissertation has not been altered in any way. We have altered the formatting in order to facilitate the ease of printing and reading of the dissertation. All rights not granted by the above license are retained by the author. Abstract: Abstract of the thesis entitled An Object-based Approach to Image-based Rendering submitted by Gan Zhifeng for the degree of Doctor of Philosophy at the University of Hong Kong in November 2006 Image-based rendering (IBR) has recently emerged as an attractive alternative to traditional geometry-based techniques for photo-realistic rendering from a collection of densely sampled images and videos. In this thesis, an object-based approach for a class of dynamic image-based representations called plenoptic videos is proposed. The plenoptic video is a simplified dynamic light field, which is obtained by capturing videos at regular locations along a series of line segments. In the object-based approach, objects at large depth differences are segmented into layers for rendering, compression and processing. The rendering quality in large environment can be significantly improved. In addition, by coding the plenoptic video at the object level, desirable functionalities such as scalability of contents, error resilience and interactivity with individual IBR objects can be achieved. A critical step in the object-based approach is to segment the objects in the video ii streams into layers or image-based objects, which is largely done semi-automatically. To reduce the segmentation time for segmenting plenoptic videos, efficient tracking techniques are highly desirable. This thesis proposes an automatic object tracking method based on the level-set method. Our method, which utilizes both local and global features of the image sequences instead of global features exploited in other approach, can achieve better tracking results for objects, especially with non-uniform energy distribution. Due to possible segmentation errors around object boundaries, natural matting with Bayesian approach is also incorporated into our system. Using the alpha map and texture so estimated, it is very convenient to composite the image-based objects onto the background of the original or other plenoptic videos. After the objects in a plenoptic video have been extracted, the depth information for each IBR objects can be estimated separately. An algorithm for rendering and post-processing of plenoptic video with layered depth map is proposed, which yields high quality renderings and can effectively suppress the ghosting and blurring artifacts in conventional rendering approach with a single mean depth. Furthermore, an object-based algorithm is developed for coding the plenoptic videos, which exploits both the temporal and spatial redundancy among video object streams in the videos to achieve higher compression efficiency. Finally, an object-based method to plenoptic video processing is studied. Because of the close relationship between plenoptic videos with traditional videos, many conventional image and video processing algorithms can also be applied in plenoptic video processing. To verify the proposed approach, two plenoptic video systems for capturing different dynamic scenes were constructed. The experimental results of segmentation, rendering, compression and processing are quite satisfactory, and iii demonstrate the usefulness, good quality and flexibility of the proposed object-based approach. _______________________ An abstract of exactly 425 words iv DOI: 10.5353/th_b3873183 Subjects: Image processing - Digital techniques Computer vision