3D Messaging Depth Data Processing via Segmentation and Packing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing messaging systems face challenges in enhancing user experiences with digital images by efficiently processing and sharing interactive 3D media, particularly in handling image and depth data across various conditions such as changes in scale, noise, lighting, and movement, which is computationally intensive.
Innovation Solution
The development of a messaging system infrastructure that supports the creation and sharing of interactive 3D messages, enabling depth-based media to be shared alongside photo and video messages, with features like motion sensor input, depth data management, and loading of external effects and asset data, allowing for real-time rendering and interaction.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If 3D messages with depth data are shared in messaging systems, then user interaction and engagement are enhanced, but computational intensity and processing complexity increase
Solution Approach 1:
The patent segments the 3D message processing into distinct components: depth data extraction, image processing, and rendering. The depth data is separated from the image data, allowing independent processing and optimization of each component, which reduces overall processing complexity while maintaining interactive capabilities
Solution Approach 2:
The patent transitions from traditional 2D image messaging to 3D message representation by adding depth information as a new dimension. This enables immersive user interaction and spatial understanding while the system manages complexity through structured data organization and selective rendering based on viewer position
2Manufacturing precision
If depth data and image data are processed for 3D messages, then spatial details and geometry are rendered, but processing time and computational resources increase
Solution Approach 1:
The patent applies preliminary processing actions by pre-extracting and pre-processing depth data and image data before message transmission. Depth maps and processed image data are prepared in advance, allowing faster rendering when the message is viewed, thus reducing real-time processing time while maintaining high spatial detail
Solution Approach 2:
The patent applies local quality processing by rendering different levels of detail for different regions of the 3D message based on viewer position and attention. Important spatial regions are rendered with higher precision while less critical areas use lower detail, optimizing processing time while maintaining necessary spatial accuracy
3Adaptability or versatility
If 3D effects and external assets are loaded for messages, then augmented reality experience is enhanced, but system complexity and data management requirements increase
Solution Approach 1:
The patent implements a universal message processing system that can handle multiple types of content (2D images, 3D messages with depth data, and augmented reality effects) through a unified architecture. This multi-functional approach allows the system to manage diverse data types using common processing pipelines, reducing overall system complexity while enhancing adaptability
Solution Approach 2:
The patent introduces intermediary processing layers that act as mediators between the raw 3D data and the final rendered output. These intermediary components (such as depth data processors and effect applicators) simplify the system architecture by breaking down complex processing tasks into manageable stages, making data management more organized and systematic
Data Source
AI summary
The subject technology generates depth data using a machine learning model based at least in part on captured image data from at least one camera of a client device. The subject technology applies, to the captured image data and the generated depth data, a 3D effect based at least in part on an augmented reality content generator. The subject technology generates a depth map using at least the depth data. The subject technology generates a packed depth map based at least in part on the depth map, the generating the packed depth map. The subject technology converts a single channel floating point texture to a raw depth map. The subject technology generates multiple channels based at least in part on the raw depth map. The subject technology generates a segmentation mask based at least on the captured image data. The subject technology performs background inpainting and blurring of the captured image data using at least the segmentation mask to generate background inpainted image data.