Composite Image Rendering via Depth Layer Segmentation

The patent introduces depth information (a fourth dimension beyond RGB) to enable composition of images with different perspectives. By capturing depth maps and performing 3D reconstruction, the system can transform images from different viewpoints into a common 3D space, then render them from a unified perspective, thus resolving the perspective compatibility limitation while maintaining real-time performance

The patent uses 3D point clouds and depth maps as intermediary representations between the source images and the final composite. These intermediaries encode spatial and geometric information that allows perspective transformation and occlusion handling, enabling flexible composition without requiring direct pixel-level operations between mismatched perspectives

The patent segments the 3D reconstruction task into two parts: first capturing depth information from a single RGB-D sensor, then using computational methods to generate virtual viewpoints through 3D model manipulation. This segmentation eliminates the need for multiple physical sensors while achieving the same viewpoint changing capability

The patent creates a 3D copy or digital twin of the scene using depth information from a single sensor. This 3D model can then be rendered from any viewpoint computationally, replacing the need for multiple physical sensors that would otherwise be required to capture all possible viewpoints simultaneously