Under-Display Camera Image Restoration Using Spatially Variant PSF Training

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Training deep learning networks for under-display camera (UDC) image restoration is challenging due to the difficulty in capturing and labeling real image pairs, which often result in subtle differences and are costly, and existing simulations assume constant blur, failing to account for varying display layouts.

Innovation Solution

Generate synthetic ground truth images using a spatially-variant point spread function based on an optical transmission model of the display, creating training images that simulate UDC captures, allowing effective training of a machine learning model to invert blur caused by the display.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Measurement precision

If real image pairs are used for training, then training data accuracy is improved, but data collection cost and time increase

Engineering Contradiction:

Improvetraining data accuracyVSAvoiddata collection time

Core Design Contradiction:

Measurement precisionVSLoss of time

Solution Approach 1:

The patent uses synthetic images generated by rendering ground truth images through a point spread function that models the display's optical characteristics. This copying approach creates realistic training data pairs without requiring actual UDC captures, thereby reducing data collection time and cost while maintaining training accuracy.

Inventive Principle:

Principle #26Copying

Solution Approach 2:

The patent pre-computes the point spread function based on the display's optical transmission model before training. This preliminary action allows for efficient generation of synthetic training data without needing to capture and process real images during the training process, significantly reducing the time and complexity of data collection.

Inventive Principle:

Principle #10Preliminary action

2Productivity

If existing simulation methods are used, then training speed is improved, but accuracy deteriorates due to constant blur assumption

Engineering Contradiction:

Improvetraining speedVSAvoidtraining data accuracy

Core Design Contradiction:

ProductivityVSMeasurement precision

Solution Approach 1:

The patent replaces the static constant blur assumption with a dynamic point spread function that varies across different spatial locations. This spatially-variant PSF accurately models the real optical behavior of UDC systems, where blur characteristics change depending on the position and content of the image, thereby improving training data accuracy while maintaining computational efficiency.

Inventive Principle:

Principle #15Dynamics

Solution Approach 2:

The patent changes the parameters of the point spread function from a constant value to spatially-variant parameters based on the display's optical transmission model. This allows the simulation to accurately represent the varying blur characteristics across different regions of the image, improving both training accuracy and the realism of the synthetic data.

Inventive Principle:

Principle #35Parameter changes

3Measurement precision

If display layout is accounted for in simulation, then training accuracy is improved, but device complexity increases

Engineering Contradiction:

Improvetraining data accuracyVSAvoidsimulation complexity

Core Design Contradiction:

Measurement precisionVSDevice complexity

Solution Approach 1:

The patent replaces complex physical optical modeling with a computational approach using a point spread function derived from an optical transmission model. This substitution simplifies the simulation process by using mathematical operations on image data rather than complex physical simulations, reducing device complexity while maintaining high training accuracy.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Applied Scientific Principles

This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.

Function Achieved in This Case

This approach enables timely, cost-effective training of machine learning models for UDC image restoration, adaptable to different display layouts, reducing blur and improving image quality without real data requirements.

Implementation Method 1

Due to optical diffraction and noise generated by having a display in front of a camera lens, image restoration is performed in order to improve the quality of images captured by an under-display camera

Methodology Applied

Scientific EffectOptical diffraction: Diffraction

Data Source

PatentUS12511867B2Machine learning model training using synthetic data for under-display camera (UDC) image restoration

Publication Date: 2025.12.30 SAMSUNG ELECTRONICS CO LTD

AI summary

A method includes obtaining an under-display camera (UDC) image captured using a camera located under a display. The method also includes processing, using at least one processing device of an electronic device, the UDC image based on a machine learning model to restore the UDC image. The method further includes displaying or storing the restored image corresponding to the UDC image. The machine learning model is trained using (i) a ground truth image and (ii) a synthetic image generated using the ground truth image and a point spread function that is based on an optical transmission model of the display.