HMD Reprojection Imaging With Occlusion Sampling for Low-Latency View Updates

Solution Overview

Problem

Existing image display systems, particularly head-mounted displays, face challenges in achieving high responsiveness and image quality due to increased processing loads and transfer delays when displaying images with changing viewpoints, leading to discomfort and motion sickness.

Innovation Solution

An image data generation device sets a reference viewpoint and generates a reference image, acquires additional sampling data for occluded parts, and transmits this data to a display device for reprojection, allowing for efficient image transformation to match the latest viewpoint.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Manufacturing precision

If higher resolution and complex calculations are used to achieve more realistic image representation, then image quality is improved, but image processing load increases

Engineering Contradiction:

Improveimage qualityVSAvoidimage processing load

Core Design Contradiction:

Manufacturing precisionVSDevice complexity

Solution Approach 1:

The patent divides the image into a reference image (containing visible surfaces from the reference viewpoint) and additional sampling data (containing color information for occluded parts). This segmentation allows the display device to process only the necessary portions of the image at high resolution, rather than processing the entire image at full complexity, thus improving image quality while managing processing load.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent extracts only the essential information needed for image generation: the reference image and additional sampling data for occluded parts. By taking out only the necessary data elements and transmitting them to the display device, the system reduces the overall data transfer burden while maintaining the quality of the final rendered image.

Inventive Principle:

Principle #2Taking out (Extraction)

2Manufacturing precision

If image data is transferred from a separate device, then image generation capability is improved, but transfer time increases

Engineering Contradiction:

Improveimage generation capabilityVSAvoidtransfer time

Core Design Contradiction:

Manufacturing precisionVSLoss of time

Solution Approach 1:

The patent extracts only the essential information needed for image generation: the reference image and additional sampling data for occluded parts. By taking out only the necessary data elements and transmitting them to the display device, the system reduces the overall data transfer burden while maintaining the quality of the final rendered image.

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

The image data generation device performs preliminary processing to create the reference image and additional sampling data in advance. This preliminary action allows the display device to receive pre-processed data that requires less processing power, thereby reducing transfer time while maintaining image generation capability.

Inventive Principle:

Principle #10Preliminary action

3Speed

If high frame rate display is implemented for responsiveness, then user experience is improved, but processing load increases

Engineering Contradiction:

ImproveresponsivenessVSAvoidprocessing load

Core Design Contradiction:

SpeedVSDevice complexity

Solution Approach 1:

The patent divides the image into a reference image (containing visible surfaces from the reference viewpoint) and additional sampling data (containing color information for occluded parts). This segmentation allows the display device to process only the necessary portions of the image at high resolution, rather than processing the entire image at full complexity, thus improving image quality while managing processing load.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent applies partial processing by generating additional sampling data only for occluded parts rather than processing the entire image. This partial action reduces the overall processing load while still achieving the necessary image quality, enabling high frame rate display without excessive processing requirements.

Inventive Principle:

Principle #16Partial or excessive action

Data Source

PatentUS12526391B2Image data generation device, display device, image display system, image data generation method, image display method, and data structure of image data

Publication Date: 2026.01.13 SONY INTERACTIVE ENTERTAINMENT LLC

AI summary

A viewpoint setting unit of an image data generation device sets a reference viewpoint on a basis of position and orientation information regarding a head-mounted display, and a reference image drawing unit generates a reference image in a field of view corresponding to the reference viewpoint. An additional data generation unit acquires color information regarding an occluded part not represented in the reference image from a different viewpoint as additional sampling data. A reprojection unit of the head-mounted display transforms the reference image into an image from a latest viewpoint and determines a pixel value of the hidden part using the additional sampling data.