3D Convolution Neural Network for Stereoscopic Video Depth Map Generation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for converting 2D films into 3D films are costly and labor-intensive, with depth maps produced by existing techniques often having low resolution and unclear contours, and struggling to address occlusion and invisible parts.
Innovation Solution
A monocular-to-binocular stereoscopic video generation method using a 3D convolution neural network is trained with a large dataset of 2D videos, employing 3D and 2D convolutional layers to generate high-resolution depth maps and handle occlusion, with shot segmentation using fuzzy C-means clustering and stochastic gradient descent for efficient network training.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Extent of automation
If manual depth map creation and depth map rendering algorithm are used to convert 2D films into 3D films, then stereoscopic video can be produced, but production costs and manpower requirements increase significantly
Solution Approach 1:
The system uses automatically learned depth estimation algorithms that self-improve through training on large datasets of paired 2D-3D video sequences. The deep learning model performs depth map generation autonomously without requiring manual depth map creation, thereby reducing both manpower requirements and production costs while maintaining high-quality stereoscopic video output
Solution Approach 2:
The patent replaces the manual mechanical process of depth map creation with an automated deep learning system. The convolutional neural network automatically learns depth estimation from training data, substituting human operators and manual depth map rendering algorithms with an intelligent automated system that reduces production costs and increases efficiency
2Manufacturing precision
If existing depth estimation techniques are used to generate depth maps, then 3D video can be produced, but the depth map resolution is low and contours are unclear
Solution Approach 1:
The patent transitions from traditional 2D convolutional neural networks to a 3D convolutional neural network architecture. This dimensional enhancement allows the model to process temporal information across multiple frames simultaneously, capturing motion patterns and temporal consistency. The 3D convolution operations enable the network to generate higher resolution depth maps with clearer contours by leveraging information from adjacent frames in the temporal dimension
Solution Approach 2:
The system performs preliminary training on large datasets of paired 2D-3D video sequences before deployment. During this pre-training phase, the network learns robust depth estimation patterns and temporal relationships. This preliminary action ensures that when the model is deployed for actual stereoscopic video generation, it can produce high-resolution depth maps with clear contours, effectively solving the resolution and contour clarity issues
3Reliability
If existing depth estimation methods are used, then depth maps can be generated, but occlusion and invisible parts caused by view changes cannot be properly handled
Solution Approach 1:
The 3D convolutional neural network processes continuous sequences of video frames rather than individual frames in isolation. By maintaining temporal continuity and processing multiple frames simultaneously, the model can track objects across frame transitions and properly handle occlusion and invisible parts. The continuous processing approach allows the network to infer missing information from adjacent frames, improving reliability in handling view changes and occlusions
Solution Approach 2:
The system performs extensive pre-training on large datasets containing various occlusion scenarios and view changes. During this preliminary training phase, the network learns to handle occlusion and invisible parts by observing patterns in the training data. This pre-training ensures that when deployed, the model can reliably handle occlusion and view changes without requiring additional processing time during actual operation
Data Source
AI summary
A stereoscopic video generation method based on 3D convolution neural network is disclosed, which is able to convert existing 2D video sources into stereoscopic videos. The method includes preparing the training data, dividing the training video sources into left eye view sequences and right eye view sequences; and then processing the left eye image sequences through shot segmentation via fuzzy C-means clustering method, calculating a mean image of all left eye images, subtracting the mean image from the left eye images, taking the right eye images as a training target; training the obtained 3D convolution neural network through the training data; processing the 2D video sources which need to be converted into stereoscopic videos in the same way as training set, inputting to the trained 3D convolution neural network to obtain the right eye view image sequences of the 2D videos; and finally combining the two to be stereoscopic videos.