Neural Network Light Source Estimation for 3D Object Realism

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Current methods for synthesizing virtual or augmented reality objects with 3D images struggle to accurately simulate light sources, affecting the natural appearance of these objects, as they often rely on pre-defined light sources or fail to account for indirect light sources.

Innovation Solution

A processor-implemented method using a trained neural network to detect objects and estimate light source information, including position, intensity, and radiation direction, regardless of whether the light source is present in the input image, by analyzing object regions and shadow information.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If pre-defined light sources are used for rendering, then the rendering process is simple and fast, but the realism and natural appearance of virtual objects deteriorate

Engineering Contradiction:

Improverendering speedVSAvoidrealism of virtual object

Core Design Contradiction:

ProductivityVSManufacturing precision

Solution Approach 1:

The system performs preliminary action by detecting objects and estimating light source information (position, intensity, radiation direction) from the input image before rendering the virtual object. This pre-analysis of the environment allows the rendering process to use accurate light source data, improving realism without significantly increasing rendering time.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system introduces an intermediary component (light source estimation module) that bridges the gap between simple pre-defined light sources and complex real-world lighting. This module analyzes shadows, shading, and object properties to infer light source characteristics, providing realistic lighting data without requiring direct measurement or complex physical simulations.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Manufacturing precision

If complex light source estimation methods are used to improve realism, then the natural appearance of virtual objects improves, but the computational complexity and processing time increase

Engineering Contradiction:

Improverealism of virtual objectVSAvoidprocessing complexity

Core Design Contradiction:

Manufacturing precisionVSDevice complexity

Solution Approach 1:

The system extracts only the essential light source information (position, intensity, radiation direction) needed for realistic rendering, rather than attempting to model all aspects of complex lighting scenarios. This selective extraction maintains realism while reducing computational complexity by focusing on the most impactful light source parameters.

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

The system performs partial action by estimating light source information only for detected objects in the scene rather than analyzing the entire image or all possible light sources. This approach achieves realistic rendering for virtual objects while limiting computational complexity to only the necessary processing steps.

Inventive Principle:

Principle #16Partial or excessive action

3Measurement precision

If light sources not visible in the image are considered, then the accuracy of light source estimation improves, but the difficulty of detecting and measuring light source properties increases

Engineering Contradiction:

Improvelight source estimation accuracyVSAvoidlight source detection difficulty

Core Design Contradiction:

Measurement precisionVSDifficulty of detecting and measuring

Solution Approach 1:

The system converts the challenge of invisible light sources into a benefit by using the shadows and shading patterns they create on visible objects as indirect evidence. Instead of directly detecting the light source, the system analyzes the effects (shadows, highlights, shading gradients) that invisible light sources produce, thereby accurately estimating light source properties without direct observation.

Inventive Principle:

Principle #22Blessing in disguise (Convert harm into benefit)

Solution Approach 2:

The system applies inversion by reversing the detection approach: instead of trying to detect the light source directly, it detects the effects of the light source (shadows, shading on objects) and infers the light source properties from these effects. This indirect detection method improves accuracy for invisible light sources while managing detection difficulty.

Inventive Principle:

Principle #13The other way round (Inversion)

Data Source

PatentUS11663796B2Method and apparatus with light source information output

Publication Date: 2023.05.30 SAMSUNG ELECTRONICS CO LTD

AI summary

A processor-implemented light source information output method includes: receiving an input image; detecting, using a trained neural network, at least one object in the input image; estimating, using the trained neural network, light source information of a light source corresponding to the at least one object; and outputting the light source information.