Human Inpainting Models for Scene-Based Object Editing

Solution Overview

Problem

Conventional image editing systems are inflexible and inefficient, requiring significant user interaction to edit digital images, as they operate on a pixel level and lack the ability to anticipate and prepare for edits, necessitating meticulous processes and multiple steps to modify objects.

Innovation Solution

A scene-based image editing system that utilizes machine learning models to pre-process digital images, segment objects, generate object masks and content fills, and anticipate user interactions, allowing for intuitive, object-aware modifications that maintain real-world conditions without requiring extensive user input.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Ease of operation

If conventional image editing systems operate on pixel level, then editing precision is maintained, but user interaction complexity and time consumption increase significantly

Engineering Contradiction:

Improveediting efficiencyVSAvoidtime for user interactions

Core Design Contradiction:

Ease of operationVSLoss of time

Solution Approach 1:

The system performs preliminary actions by pre-processing the digital image to identify objects, generate object masks, and create content fills before the user actually requests editing. This anticipation of user needs allows the system to have editing components ready, significantly reducing the time required when the user initiates an edit operation.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system segments the digital image into distinct objects and generates separate object masks for each identified object. This segmentation allows the system to treat objects as cohesive units rather than individual pixels, enabling more efficient editing operations where entire objects can be modified, moved, or replaced with a single user action.

Inventive Principle:

Principle #1Segmentation

2Adaptability or versatility

If the system pre-processes images to anticipate edits, then editing flexibility improves, but computational complexity and processing time increase

Engineering Contradiction:

Improveediting flexibilityVSAvoidsystem complexity

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The system performs preliminary actions by pre-processing the digital image to identify objects, generate object masks, and create content fills before the user actually requests editing. This anticipation of user needs allows the system to have editing components ready, significantly reducing the time required when the user initiates an edit operation.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system introduces intermediary components including object detection modules, mask generation networks, and content fill creation mechanisms that act as mediators between the raw image and the final edited output. These intermediaries break down the complex editing task into manageable stages, improving flexibility while organizing system complexity into modular, manageable components.

Inventive Principle:

Principle #24Intermediary (Mediator)

3Productivity

If objects are treated as cohesive units, then editing efficiency increases, but precision in modifying specific parts of objects decreases

Engineering Contradiction:

Improveediting speedVSAvoidedit precision

Core Design Contradiction:

ProductivityVSManufacturing precision

Solution Approach 1:

The system segments the digital image into distinct objects and generates separate object masks for each identified object. This segmentation allows the system to treat objects as cohesive units rather than individual pixels, enabling more efficient editing operations where entire objects can be modified, moved, or replaced with a single user action.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The system applies local quality by generating specific object masks that isolate particular regions or parts of objects when needed. This allows the editing operation to be applied with varying precision - sometimes to entire objects for efficiency, and sometimes to specific parts of objects when the user requires more granular control, thus maintaining both productivity and precision as needed.

Inventive Principle:

Principle #3Local quality

Data Source

PatentUS20250217946A1Generating a modified digital image utilizing a human inpainting model

Publication Date: 2025.07.03 ADOBE INC

AI summary

The present disclosure relates to systems, methods, and non-transitory computer-readable media that modify digital images via scene-based editing using image understanding facilitated by artificial intelligence. For example, in one or more embodiments the disclosed systems utilize generative machine learning models to create modified digital images portraying human subjects. In particular, the disclosed systems generate modified digital images by performing infill modifications to complete a digital image or human inpainting for portions of a digital image that portrays a human. Moreover, in some embodiments, the disclosed systems perform reposing of subjects portrayed within a digital image to generate modified digital images. In addition, the disclosed systems in some embodiments perform facial expression transfer and facial expression animations to generate modified digital images or animations.