Gravity Aligned Image Rotation for AR Lighting Estimation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing Augmented Reality (AR) systems face challenges in accurately aligning images captured by mobile devices, leading to improper lighting estimation and tracking, especially when devices are held in orientations other than portrait mode, which affects the realism and accuracy of AR experiences.
Innovation Solution
A system that uses a combination of Inertial Measurement Units (IMUs) and tracking stacks to detect device orientation and movement, rotating images to ensure gravity-aligned input for neural networks, thereby correcting for non-standard orientations and generating realistic lighting and tracking estimates.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If images are processed in non-portrait orientations, then device versatility is improved, but lighting estimation accuracy deteriorates
Solution Approach 1:
The system performs preliminary rotation of images to portrait orientation before processing them through the neural network for lighting estimation. This preliminary action ensures that the neural network receives input in its expected orientation, maintaining accurate lighting estimation while still supporting multiple device orientations through the rotation transformation.
Solution Approach 2:
Image rotation serves as an intermediary transformation between the captured image orientation and the neural network's expected input orientation. By rotating images to a standard portrait orientation before processing, the system bridges the gap between diverse capture orientations and the fixed expectations of the lighting estimation model.
2Measurement precision
If images are rotated to gravity-aligned orientation, then lighting estimation accuracy is improved, but processing time increases
Solution Approach 1:
The rotation operation is performed as a preliminary action before the image is processed by the neural network. By completing the rotation beforehand, the subsequent neural network processing can proceed efficiently without needing to handle rotated orientations, thus minimizing overall processing time while ensuring accurate lighting estimation.
3Device complexity
If tracking is performed on non-gravity-aligned images, then system simplicity is maintained, but tracking accuracy deteriorates
Solution Approach 1:
The system performs preliminary rotation of images to gravity-aligned orientation before tracking. This preliminary transformation ensures that tracking algorithms operate on properly oriented data, improving tracking accuracy while maintaining relatively simple tracking implementations that expect portrait-oriented input.
Data Source
AI summary
Systems, methods, and computer program products are described that include obtaining, at a processor, a first image from an image capture device onboard a computing device, detecting, using the processor and at least one sensor, a device orientation of the computing device associated with capture of the first image, determining, based on the device orientation and a tracking stack associated with the computing device, a rotation angle in which to rotate the first image, rotating the first image to the rotation angle to generate a second image, and generating neural network based estimates associated with the first image and the second image.