Neural Object Relighting for Realistic Mobile Image Compositing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing image processing methods for changing lighting properties are complex and computationally demanding, making them expensive and impractical for mobile devices, especially when dealing with multiple lighting sources and unknown lighting conditions.
Innovation Solution
Utilizing neural networks trained with 3D models to render images with different lighting conditions, generating ground truth pairs, and employing generative adversarial networks to adjust lighting properties of objects within images.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If traditional computer graphic methods are used to change lighting properties, then lighting accuracy can be improved, but device complexity and computational requirements increase significantly
Solution Approach 1:
The patent replaces traditional mechanical computer graphic lighting calculation systems with a neural network-based system. The neural network is trained offline using 3D models and ground truth pairs, then deployed as a pre-trained model that performs lighting adjustment through simple tensor operations rather than complex iterative calculations, thereby reducing device complexity while maintaining lighting accuracy.
Solution Approach 2:
The patent performs preliminary training of the neural network offline using extensive 3D models and ground truth data pairs. This preliminary action creates a pre-trained model that encapsulates complex lighting relationships, allowing the mobile device to perform lighting adjustment without executing complex training algorithms, thus reducing computational burden on the end device.
2Manufacturing precision
If traditional computer graphic methods are used to change lighting properties, then lighting accuracy can be improved, but computational cost increases making it impractical for mobile devices
Solution Approach 1:
The patent substitutes energy-intensive traditional computer graphic lighting calculations with a neural network that performs lighting adjustment through efficient tensor operations. The computationally expensive training phase is moved offline, leaving only lightweight inference operations for mobile devices, significantly reducing energy consumption while maintaining lighting accuracy.
Solution Approach 2:
The patent performs the computationally intensive neural network training in advance offline using powerful servers. The pre-trained model is then deployed to mobile devices, which only need to execute efficient forward propagation operations, thereby avoiding the high energy cost of training on mobile devices while achieving accurate lighting adjustment.
3Reliability
If multiple lighting sources with different color properties are simulated, then lighting realism can be improved, but system complexity and computational requirements increase
Solution Approach 1:
The patent uses parameter changes by representing complex multi-source lighting conditions as variations in input image characteristics (brightness, color temperature, shadow patterns). The neural network learns to map these parameter variations to appropriate lighting adjustments, simplifying the representation of multiple lighting sources while maintaining realism.
Solution Approach 2:
The patent creates a universal neural network model that handles multiple lighting conditions and scenarios through a single unified architecture. The model is trained on diverse ground truth pairs covering various lighting situations, enabling it to generalize across different lighting sources and configurations without requiring separate specialized systems for each scenario.
Data Source
AI summary
A messaging system performs image processing to relight objects with neural networks for images provided by users of the messaging system. A method of relighting objects with neural networks includes receiving an input image with first lighting properties comprising an object with second lighting properties and processing the input image using a convolutional neural network to generate an output image with the first lighting properties and comprising the object with third lighting properties, where the convolutional neural network is trained to modify the second lighting properties to be consistent with lighting conditions indicated by the first lighting properties to generate the third lighting properties. The method further includes modifying the second lighting properties of the object to generate the object with modified second lighting properties and blending the third lighting properties with the modified second lighting properties to generate a modified output image comprising the object with fourth lighting properties.