3D Messaging System With Depth-Aware Inpainting
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing technologies face challenges in enhancing user experiences with digital images by performing image processing operations on various objects and features captured in changing conditions, such as image scales, noises, lighting, movement, or geometric distortion, which can be computationally intensive.
Innovation Solution
The proposed solution involves a messaging system infrastructure that supports the creation and sharing of interactive 3D media, referred to as 3D messages, which include both image and depth data. This system enables motion sensor input, manages depth data, and loads external effects and asset data, allowing for enhanced interactive experiences.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If image processing operations are performed on objects and features captured in changing conditions (image scales, noises, lighting, movement, geometric distortion), then user experience enhancement is improved, but computational intensity increases
Solution Approach 1:
The patent segments the image processing task into two distinct phases: a server-side phase that performs computationally intensive operations on captured images under varying conditions, and a client-side phase that displays the processed results. This segmentation allows the server to handle complex computational demands while clients focus on user interaction and display, thereby enhancing user experience without overloading individual device computational resources.
Solution Approach 2:
The patent introduces a messaging server system as an intermediary between the capturing device and the display device. This intermediary receives captured images, performs the necessary image processing operations to handle changing conditions (scaling, noise reduction, lighting adjustment, movement compensation, geometric distortion correction), and transmits the processed images back to clients for display. The intermediary absorbs the computational intensity while enabling enhanced user experience at the client end.
2Adaptability or versatility
If 3D effects and augmented reality content are applied to images, then interactivity and immersion are improved, but system complexity increases
Solution Approach 1:
The patent implements a universal messaging server system that handles multiple types of content and effects through a single integrated platform. The server can process various image types (2D images, 3D models, videos), apply different augmented reality effects, and support multiple interaction modes (viewing, manipulating, sharing) within one unified system architecture. This multi-functionality enables enhanced interactivity and immersion while managing system complexity through consolidation rather than proliferation of separate systems.
Solution Approach 2:
The patent performs preliminary processing of images and generation of 3D effects on the server side before transmission to client devices. By pre-processing captured images, generating depth maps, creating augmented reality overlays, and preparing interactive 3D content in advance, the system reduces the computational burden on client devices and simplifies the overall system architecture. The heavy lifting is done beforehand at the server level, allowing clients to focus on display and user interaction.
Data Source
AI summary
The subject technology receives, at a client device, a selection of a selectable graphical item from a plurality of selectable graphical items, the selectable graphical item comprising an augmented reality content generator including a 3D effect. The subject technology applies, to image data and depth data, the 3D effect based at least in part on the augmented reality content generator, the applying the 3D effect. The subject technology generates a depth map using at least the depth data, generates a segmentation mask based at least on the image data, and performs background inpainting and blurring of the image data using at least the segmentation mask to generate background inpainted image data. The subject technology generates a 3D message based at least in part on the applied 3D effect.