Extrapolating HDR Panoramas from LDR Images via ML Lighting Estimation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current algorithms fail to generate realistic 360-degree high dynamic range (HDR) panoramas from low dynamic range (LDR) narrow field of view (FOV) images, especially when inserting shiny or mirror objects, as they lack sufficient environmental information and struggle to accurately represent light sources, resulting in blurry and low-contrast renderings.
Innovation Solution
A data-driven approach using machine learning techniques, such as neural networks, to estimate lighting parameters and generate HDR panoramas by aggregating image and lighting features, allowing user control over lighting settings, and employing generative adversarial networks to produce detailed and realistic environment maps with accurate light source intensities.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If conventional algorithms are used to generate panoramas from LDR narrow FOV images, then the processing is computationally simpler, but the output quality is blurry and lacks realistic lighting representation
Solution Approach 1:
The patent introduces an environment map as an intermediary element that captures lighting and environmental information. This environment map serves as a mediator between the input LDR image and the final HDR panorama, enabling realistic lighting representation without requiring complex direct computation between the input and output images.
Solution Approach 2:
The patent segments the panorama generation process into distinct components: extracting lighting parameters from the input image, generating an environment map with HDR values, and composing the final panorama. This segmentation allows each component to be optimized independently, improving overall quality while managing computational complexity.
2Loss of information
If the input image provides only 10% or less of environment information, then the data processing is simpler, but the ability to hallucinate accurate lighting environment is insufficient
Solution Approach 1:
The patent performs preliminary extraction of lighting parameters (such as light source positions, intensities, and colors) from the input LDR image before generating the environment map. This preliminary action captures the limited environmental information available and uses it as a foundation for hallucinating the complete lighting environment, thereby reducing information loss.
Solution Approach 2:
The patent creates an environment map that copies and extrapolates the lighting information from the limited input image. By copying the available lighting parameters and using them to generate a comprehensive HDR environment map, the system recovers the missing 90%+ of environmental information while maintaining lighting accuracy.
3Illumination intensity
If HDR panoramas are generated from LDR images, then the dynamic range is improved, but the computational complexity and processing requirements increase
Solution Approach 1:
The patent changes the dynamic range parameter of the environment map from LDR to HDR, allowing the representation of a wider range of lighting intensities. By applying parameter changes specifically to the environment map rather than the entire processing pipeline, the system achieves HDR output while minimizing the computational energy required for the transformation.
Data Source
AI summary
In some examples, a computing system accesses a field of view (FOV) image that has a field of view less than 360 degrees and has low dynamic range (LDR) values. The computing system estimates lighting parameters from a scene depicted in the FOV image and generates a lighting image based on the lighting parameters. The computing system further generates lighting features generated the lighting image and image features generated from the FOV image. These features are aggregated into aggregated features and a machine learning model is applied to the image features and the aggregated features to generate a panorama image having high dynamic range (HDR) values.