Event Data Motion Prediction for XR Frame Generation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The video see-through method in cross reality (XR) devices suffers from delays in processing, leading to phenomena like VR sickness and difficulties in tracking moving objects, due to the need for calculating motion vectors at frame rates that do not allow for detailed motion prediction and increased communication traffic.
Innovation Solution
An image capturing device with an event data obtaining unit and a processing unit that detects changes in pixel values at a higher frequency than the frame period, allowing for accurate prediction of future frames by generating predictive frame data with detailed motion information, reducing processing delays and communication traffic.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If image capturing is performed at a high frame rate to take detailed motion into consideration, then prediction accuracy is improved, but processing speed requirement increases and communication traffic increases
Solution Approach 1:
The patent segments the motion detection process into two parts: (1) coarse motion detection at frame rate using traditional motion vectors, and (2) fine motion detection at higher frequency using event data from pixels that detect changes between frames. This segmentation allows detailed motion capture without requiring full high-frame-rate processing of all image data.
Solution Approach 2:
The patent introduces event data as an intermediary between full frame capture and motion detection. Event data contains only the necessary information about pixel changes and motion events, serving as a compressed intermediary representation that enables high-frequency motion analysis without the full data burden of high-frame-rate imaging.
2Measurement precision
If image capturing is performed at a high frame rate to take detailed motion into consideration, then prediction accuracy is improved, but communication traffic from image sensor to processing device increases
Solution Approach 1:
The patent extracts only the essential motion information from the image data stream. Instead of transmitting full high-frame-rate image data, the system extracts event data containing pixel change information, motion vectors, and timestamps. This extraction reduces communication traffic to only the necessary motion-related data while preserving prediction accuracy.
Solution Approach 2:
The patent performs preliminary processing at the sensor level by detecting pixel changes and generating event data before transmission. Motion detection and event identification occur preliminarily in the imaging device itself, so that only processed motion events rather than raw high-volume image data need to be communicated to the processing device.
3Device complexity
If motion vector calculation is performed at frame rate, then processing load is reduced, but detailed motion between frames cannot be considered
Solution Approach 1:
The patent implements dynamic motion detection frequency adjustment. The system detects motion events at variable frequencies based on scene activity - using event data to identify when detailed motion occurs and processing at high frequency only when necessary, rather than uniformly at high frame rate. This dynamic approach balances processing load with motion detection accuracy.
Data Source
Figure 1~2
Figure 3
Figure 4
AI summary
A processing device includes: first obtaining means configured to obtain first frame data that is frame data of an image of an object at a predetermined interval; second obtaining means configured to obtain event data that is obtainable at an interval shorter than the predetermined interval and is a detection result of a change in a pixel value of the object; detection means configured to detect motion of the object at an interval shorter than the predetermined interval by using the event data; and generation means configured to generate second frame data from the first frame data and the motion of the object, the second frame data corresponding to a time later than a time to which the first frame data corresponds.