Neural Network Lighting Estimation for Messaging Augmentations
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing image processing methods for estimating lighting properties in messaging systems are complex and computationally demanding, making them expensive to develop and resource-intensive, especially for mobile devices, and struggle with natural integration of augmentations due to lighting mismatch.
Innovation Solution
A neural network system that estimates lighting properties of original images and adjusts augmentations to match, using 3D models for training data generation and a generative adversarial network to process images, ensuring natural appearance of modifications.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional computer graphic methods are used to estimate lighting properties, then measurement precision is improved, but device complexity and computational demand increase significantly
Solution Approach 1:
The patent replaces traditional mechanical/computational computer graphic methods with a neural network-based system. The neural network is trained offline using synthetic data generated from 3D models and lighting simulations, then deployed as a pre-trained model that performs lighting estimation through simple forward propagation, eliminating complex real-time calculations.
Solution Approach 2:
The patent performs the computationally intensive work in advance by generating synthetic training data from 3D models with known lighting properties, and by pre-training the neural network offline. This preliminary action allows the deployed system to perform lighting estimation with minimal computational resources during actual use.
2Measurement precision
If traditional computer graphic methods are used to estimate lighting properties, then measurement precision is improved, but productivity decreases due to high computational demands
Solution Approach 1:
The patent replaces computationally intensive traditional computer graphic methods with a neural network that performs lighting estimation through efficient forward propagation. The network processes images rapidly once trained, achieving both high precision and productivity.
Solution Approach 2:
The patent uses synthetic images generated from 3D models as training data, creating copies of real-world scenarios with known ground truth lighting properties. This allows the neural network to learn from extensive synthetic data without requiring equivalent computational resources during deployment.
3Ease of operation
If augmentations are added to images without lighting adjustment, then ease of operation is improved, but manufacturing precision worsens due to lighting mismatch
Solution Approach 1:
The patent applies the estimated lighting properties of the original image to the augmentation elements, using the lighting estimation as feedback to adjust augmentation appearance. This ensures augmentations match the scene's lighting conditions while maintaining ease of operation.
Solution Approach 2:
The patent applies lighting adjustments specifically to augmentation elements based on their local lighting conditions estimated from the original image. Each augmentation receives customized lighting parameters matching its position and context in the scene.
Data Source
AI summary
A messaging system performs image processing to estimate lighting properties with neural networks for images provided by users of the messaging system. A method of estimating light properties includes receiving an input image with first lighting properties and processing the input image using a convolutional neural network to generate an estimate of the first lighting properties. The method may further include modifying the input image with an augmentation to generate a modified input image, where the augmentation has second lighting properties, and changing the second lighting properties of the augmentation in the modified input image to the estimate of the first lighting properties.