4.4 3D Multi-View Generation

In this section, we discuss the topic of how to derive a multi-view 3D content from a stereo image pair, as the future glass-free displays require multi-view content. So far, no automatic multi-view generation algorithm can claim to reach a satisfactory visual quality, so some human editing or tuning is necessary during the processing. A few fundamental automatic modules are required in the process:

• Depth extraction from stereo content: stereo matching is a typical approach to estimate disparity from a given stereo image pair. In [6], a large number of efforts in this field have been evaluated.
• New view synthesis: 3D warping is a key approach in depth-based view synthesis, in which the pixels in a reference image are back-projected to 3D spaces, and then reprojected onto the target viewpoint. In this approach, visual errors, such as black-contours appearing, and holes around boundaries, need to be fixed using filtering or in-painting techniques.

  In this process, the major challenges can be summarized as follows:

• Missing information due to occlusion will produce noticeable errors in the newly generated views.
• Flickering artifacts around object boundaries and disoccluded regions in the new views will cause degradation in viewing experiences.
• The viewing condition related aspects, such as display size, light condition, specific autostereoscopic display characteristics and constraints, and so on, should be taken into account in the process.
• The user preference aspects, such as depth range and viewing distance, should also be taken into account.
• Effective user interaction into a multi-view environment is hard to design.
• Pixel-based view representation makes the editing very difficult.

So far there are very few solutions in the market on this topic. In almost all these efforts, the processing is conducted in a view-based mode, thus objects evolving in the time domain across the video frames are seldom considered in the framework, which undermines its capability to reduce flickering artifacts. However, very few efforts consider user interaction as an effective assistance to improve the content quality. It may be worth considering a new concept which is aligned with the human intuition of object-based access and manipulation in the 3D environment. With a patented 3D data format and the associated data derivation and generation scheme, the stereo analysis, object manipulation, and final 3D view rendering are fully decoupled in the process. The data format needs to have a good extensibility that can be converted to and from other multi-view 3DTV formats, such as MVD (multi-view video plus depth). A set of computer vision algorithms need to be deployed to reduce human intervention during the process to the minimum, but this approach needs to have the extensibility to be optimized for various autostereoscopic displays.