3D Message Beautification Using Depth Maps and Background Inpainting
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing technologies face challenges in enhancing user experiences with digital images by efficiently processing and sharing interactive 3D media across messaging systems, particularly in handling changes in image scales, noises, lighting, movement, and geometric distortions, which are computationally intensive.
Innovation Solution
A messaging system infrastructure that supports the creation and sharing of 3D messages, incorporating depth-based media, motion sensor input, and manages the sending and storage of depth data, along with loading external effects and asset data, utilizing augmented reality content generators for real-time image and depth data modifications.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If real-time processing and rendering of 3D media with multiple effects are implemented, then user engagement and experience are enhanced, but computational load and processing time increase
Solution Approach 1:
The system pre-processes and stores depth maps, segmentation masks, and other processing parameters during the capture phase. These pre-computed data structures are then reused during rendering, eliminating the need for real-time computation of these elements and reducing processing time during user interaction.
Solution Approach 2:
The image processing pipeline is divided into distinct segments: capture phase (pre-processing), transmission phase (data exchange), and rendering phase (display). Each segment handles specific tasks independently, allowing optimization of each stage and enabling real-time performance by avoiding redundant computations across the entire pipeline.
2Manufacturing precision
If complex image processing operations such as noise reduction, lighting adjustment, and geometric distortion correction are performed, then image quality is improved, but computational complexity increases
Solution Approach 1:
Complex image processing operations including noise reduction, lighting adjustment, and geometric distortion correction are performed during the capture phase and stored as pre-processed depth maps and segmentation masks. This eliminates the need to re-execute these computationally intensive operations during rendering, reducing real-time computational complexity while maintaining image quality.
3Adaptability or versatility
If depth data and multiple asset types are transmitted and stored, then interactive 3D rendering capabilities are enabled, but data transmission and storage requirements increase
Solution Approach 1:
The system extracts and transmits only the essential data elements needed for 3D rendering: depth maps, segmentation masks, and processing parameters. Non-essential or redundant data is eliminated, reducing transmission and storage requirements while preserving the core interactive 3D rendering capabilities.
Solution Approach 2:
Instead of transmitting and storing multiple complete versions of processed image data at different stages, the system transmits compact source data (depth maps and segmentation masks) that can be efficiently copied and reused across multiple rendering operations, significantly reducing data volume requirements.
Data Source
AI summary
The subject technology applies, to image data and depth data, a 3D effect including at least one beautification operation based on an augmented reality content generator, the 3D effect including a beautification operation, the beautification operation comprising modifying image data, the image data including a region corresponding to a representation of a face, the beautification operation comprising using a machine learning model for at least one of smoothing blemishes or preserving facial skin texture. The subject technology generates a depth map using at least the depth data. The subject technology generates a segmentation mask based at least on the image data. The subject technology performs background inpainting and blurring of the image data using at least the segmentation mask to generate background inpainted image data. The subject technology generates a 3D message based at least in part on the applied 3D effect including the at least one beautification operation.