Neural Network Rendering Layer for Realistic Image Editing

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Conventional digital image editing systems struggle to generate realistic modified images that accurately reflect target properties such as material changes, object modifications, and illumination environment adjustments, often relying on simplifying assumptions that introduce inaccuracies and limitations.

Innovation Solution

A neural network with a rendering layer is trained to explicitly model the image formation process, predicting intrinsic properties like material properties, surface orientations, and illumination environments, allowing for accurate modification and synthesis of digital images without making assumptions about geometric shapes or materials.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Ease of manufacture

If conventional digital editing systems apply filters or add objects to digital images, then basic image modification capability is achieved, but the ability to generate realistic appearance models reflecting environment (illumination, material properties) deteriorates

Engineering Contradiction:

Improveimage modification capabilityVSAvoidrealistic appearance model accuracy

Core Design Contradiction:

Ease of manufactureVSReliability

Solution Approach 1:

The patent introduces an intermediary physical properties representation layer between the input digital image and the output modified image. This intermediary layer explicitly models material properties, surface orientations, and illumination environments, allowing the system to maintain both ease of modification and realistic appearance by operating through this physical property mediator rather than directly manipulating pixels

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The patent changes the parameter space from direct pixel manipulation to physical property parameter manipulation. By representing images in terms of material properties, surface orientations, and illumination parameters, the system enables realistic modifications through parameter changes while maintaining accuracy in the appearance model

Inventive Principle:

Principle #35Parameter changes

2Device complexity

If digital image decomposition systems make simplifying assumptions about geometry, material properties, or lighting conditions, then decomposition complexity is reduced, but accuracy and applicability to various circumstances deteriorates

Engineering Contradiction:

Improvedecomposition complexityVSAvoidphysical property identification accuracy

Core Design Contradiction:

Device complexityVSMeasurement precision

Solution Approach 1:

The patent segments the image decomposition task into distinct physical property components: material properties, surface orientations, and illumination environments. By separating these properties into independent representational layers, the system reduces decomposition complexity while maintaining accuracy, as each property can be analyzed and modified independently without requiring simplifying assumptions about their interactions

Inventive Principle:

Principle #1Segmentation

3Extent of automation

If machine learning systems represent properties in latent feature space, then image editing capability is achieved, but the ability to easily manipulate physical properties deteriorates

Engineering Contradiction:

Improveimage editing automationVSAvoidphysical property manipulation ease

Core Design Contradiction:

Extent of automationVSEase of operation

Solution Approach 1:

The patent introduces a physical property representation layer as an intermediary between the latent feature space and the user interface. This intermediary layer translates abstract latent features into interpretable physical properties (material, surface orientation, illumination), enabling automated image editing while maintaining ease of physical property manipulation through intuitive parameter control

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The patent transforms the operation space from manipulating abstract latent features to manipulating concrete physical property parameters. By representing image properties in terms of physical parameters rather than latent dimensions, the system maintains automation capability while dramatically improving the ease of operating with and understanding physical property modifications

Inventive Principle:

Principle #35Parameter changes

Data Source

PatentUS10430978B2Editing digital images utilizing a neural network with an in-network rendering layer

Publication Date: 2019.10.01 ADOBE INC

AI summary

The present disclosure includes methods and systems for generating modified digital images utilizing a neural network that includes a rendering layer. In particular, the disclosed systems and methods can train a neural network to decompose an input digital image into intrinsic physical properties (e.g., such as material, illumination, and shape). Moreover, the systems and methods can substitute one of the intrinsic physical properties for a target property (e.g., a modified material, illumination, or shape). The systems and methods can utilize a rendering layer trained to synthesize a digital image to generate a modified digital image based on the target property and the remaining (unsubstituted) intrinsic physical properties. Systems and methods can increase the accuracy of modified digital images by generating modified digital images that realistically reflect a confluence of intrinsic physical properties of an input digital image and target (i.e., modified) properties.