Light Field Display Using Device Pose for Dynamic Refocusing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional image capture devices lose directional light information, limiting the ability to recover depth and provide personalized views of light field data, which restricts user interaction and experience in visualizing light field images.
Innovation Solution
A method and apparatus that utilize user device pose to dynamically adjust viewpoint and refocusing parameters, allowing natural user interactions to control 4D to 2D re-projection and refocusing of light field images, using sensing devices like IMUs and cameras to estimate device position and orientation, and adapt the focal stack sampling interval based on scene layout.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If conventional image capture devices project a three-dimensional scene onto a two-dimensional sensor, then the device structure is simple and easy to manufacture, but directional light information is lost and depth cannot be recovered
Solution Approach 1:
The image sensor is divided into multiple regions, each corresponding to a specific directional range of incoming light. By segmenting the sensor into spatial zones, the device can capture directional light information from different angles simultaneously, resolving the contradiction between information loss and device complexity
Solution Approach 2:
The patent transitions from conventional 2D image capture to 4D light field capture by adding two spatial dimensions (u, v) to the traditional (x, y) sensor coordinates. This dimensional expansion allows the sensor to record not only position but also directional information of light rays, enabling depth recovery while maintaining a relatively simple device structure
2Loss of information
If light-field capture devices measure a four-dimensional light-field by capturing light from different viewpoints, then directional light information is preserved, but the device complexity increases significantly
Solution Approach 1:
A single image sensor performs multiple functions: it captures both conventional 2D images and 4D light field data by utilizing different regions of the sensor. This multi-functionality eliminates the need for separate devices for different imaging modes, reducing overall system complexity while preserving directional light information
Solution Approach 2:
The sensor array automatically captures light from multiple directions simultaneously without requiring mechanical movement or complex optical components. The system uses the inherent spatial arrangement of sensor elements to perform light field measurement, making the device self-sufficient and reducing external complexity
3Ease of operation
If user interaction is implemented through mouse pointing or touching a tactile screen, then the interface is simple to implement, but the user experience is limited and interaction is not intuitive
Solution Approach 1:
The system provides real-time feedback by dynamically adjusting the displayed image's focus and viewpoint based on the user's device orientation and position. As users move or rotate their devices, the light field rendering adapts immediately, creating an intuitive and responsive interaction experience that enhances both ease of operation and adaptability
Solution Approach 2:
The patent replaces manual mechanical interaction (mouse pointing, touching) with automatic sensor-based tracking of device pose. Inertial sensors and cameras detect user intent through device movement, substituting mechanical control with automated sensing and processing, thereby improving user experience while maintaining operational simplicity
4Adaptability or versatility
If multiple users view light field content with different viewpoints, then personalized viewing is achieved, but additional storage is required for multiple focal stacks
Solution Approach 1:
Instead of storing multiple static focal stacks for different users, the system dynamically generates personalized viewpoints in real-time based on each user's device pose. The light field data is processed on-the-fly to render customized images, eliminating the need for additional storage while maintaining adaptability for multiple users
Solution Approach 2:
The system changes rendering parameters (viewpoint, focus distance, depth of field) dynamically based on user input rather than storing multiple complete image sets. By adjusting these parameters in real-time from a single light field capture, the system achieves personalized viewing without increasing storage requirements
Data Source
Figure 1~3
Figure 2(a)~2(b)
Figure 4~6
AI summary
A method for displaying at least one light field based image (51) on a user's device (52) is disclosed. Such a method comprises displaying the image focused according to at least one focusing parameter determined as a function of a pose of the user's device (52). A sensor embedded in the users' device, which may be a handheld device, may estimate the pose.