Image Warping for HMDs Using Depth Offset Maps

Solution Overview

Problem

Head-mountable devices (HMDs) cause user discomfort due to spatial offset between the camera and display planes, leading to mismatched visual perceptions, and per-pixel rendering is computationally expensive, resulting in high power consumption and latency.

Innovation Solution

An electronic device warps input images based on depth information and estimated eye-to-display distance to align the displayed image with the user's visual perception, using a distance warp map to set operational parameters and generate a warped image, reducing computational resources needed.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Manufacturing precision

If per-pixel rendering is used to correct spatial offset, then visual perception alignment is improved, but computational cost and power consumption increase

Engineering Contradiction:

Improvevisual perception alignmentVSAvoidpower consumption

Core Design Contradiction:

Manufacturing precisionVSUse of energy by moving object

Solution Approach 1:

The patent segments the image processing task by dividing the image into multiple depth layers based on depth information. Instead of processing every pixel uniformly, the system processes different regions at different levels of detail according to their depth, reducing the overall computational burden while maintaining alignment accuracy for critical regions.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent applies partial correction by selectively warping only the portions of the image that require alignment adjustment based on depth information, rather than applying per-pixel rendering to the entire image. This partial action approach reduces computational cost while achieving sufficient visual perception alignment.

Inventive Principle:

Principle #16Partial or excessive action

2Manufacturing precision

If per-pixel rendering is used to correct spatial offset, then visual perception alignment is improved, but processing time and latency increase

Engineering Contradiction:

Improvevisual perception alignmentVSAvoidprocessing latency

Core Design Contradiction:

Manufacturing precisionVSLoss of time

Solution Approach 1:

The patent segments the image processing task by dividing the image into multiple depth layers based on depth information. Instead of processing every pixel uniformly, the system processes different regions at different levels of detail according to their depth, reducing the overall computational burden while maintaining alignment accuracy for critical regions.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent applies partial correction by selectively warping only the portions of the image that require alignment adjustment based on depth information, rather than applying per-pixel rendering to the entire image. This partial action approach reduces computational cost while achieving sufficient visual perception alignment.

Inventive Principle:

Principle #16Partial or excessive action

3Ease of operation

If spatial offset correction is applied, then user comfort is improved, but device complexity increases

Engineering Contradiction:

Improveuser comfortVSAvoidprocessing complexity

Core Design Contradiction:

Ease of operationVSDevice complexity

Solution Approach 1:

The patent performs preliminary actions by pre-calculating depth information and preparing warp maps before the actual image display. The system obtains depth information and determines warp maps in advance, which are then applied during image processing, reducing the complexity of real-time processing while maintaining user comfort.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent introduces an intermediary approach by using depth information as a mediator between the captured image and the display. Instead of directly applying complex per-pixel rendering, the system uses depth data to generate warp maps that serve as intermediaries, simplifying the overall processing pipeline while achieving spatial offset correction.

Inventive Principle:

Principle #24Intermediary (Mediator)

Data Source

PatentUS20230419439A1Warping an input image based on depth and offset information

Publication Date: 2023.12.28 APPLE INC

AI summary

Various implementations disclosed herein include a method performed at an electronic device including one or more processors, a non-transitory memory, an image sensor, and a display device. The method includes obtaining, via the image sensor, an input image that includes an object. The method includes obtaining depth information characterizing the object, wherein the depth information characterizes a first distance between the image sensor and a portion of the object. The method includes determining a distance warp map for the input image based on a function of the depth information and a first offset value characterizing an estimated distance between eyes of a user and the display device. The method includes setting an operational parameter for the electronic device based on the distance warp map and generating, by the electronic device set to the operational parameter, a warped image from the input image.