Real-time View Synthesis using Image Domain Warping
From iis-projects
Contents
Date
Personnel
- Pierre Greisen
- Michael Schaffner
- Simon Heinzle
Funding
Partners
Summary
With the steadily increasing frame rates and resolutions, real-time video processing poses significant computational challenges. A recent application that is particularly demanding is video content adaptation: while a growing amount of content is watched on an increasing number of different mobile platforms, most productions are captured with one acquisition system at fixed parameters. Examples for content adaption algorithms are content-aware aspect ratio retargeting, non-linear stereoscopic 3D (S3D) adaption, and S3D to multi-view generation. Getting such view synthesis algorithms to run in real-time in mobile devices was the goal of this project.
The employed framework in this work is the image domain warping (IDW) pipeline. As a first step, an IDW view synthesis algorithm determines an image warping function that is dependent on the display characteristics. The input frames are then transformed to the output frames according to the given warping function using a warping algorithm. The generation of the warping function is application-specific, and is separated from the rendering. For instance, in video retargeting, the warping function retains the aspect ratio of salient (i.e., visually important) parts of the image, while the image distortion is hidden in visually less important regions. In S3D applications, the warping function is derived from the 3D structure of the scene to generate in-between views.
The primary focus of this work has been put on efficient algorithms and hardware architectures for real-time view synthesis using the IDW pipeline. Various hardware architectures have been designed and implemented therefore.
