Gaze-Driven Autofocus Cameras for Sharp MR Passthrough
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing mixed-reality (MR) systems using fixed-focus cameras in head-mounted devices (HMDs) suffer from blurry passthrough images when users look at objects outside the camera's depth of field, leading to motion sickness and an impractical solution like infinite depth of field being too costly.
Innovation Solution
Equipping HMDs with autofocus cameras and using eye-tracking to determine the user's gaze and adjust focal distance dynamically, incorporating depth information and machine-learning techniques to ensure sharp passthrough images.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If fixed-focus cameras are used in HMDs, then device complexity is reduced, but image sharpness deteriorates when users look at objects outside the camera's depth of field
Solution Approach 1:
The patent applies the Dynamics principle by transitioning from fixed-focus cameras to autofocus cameras that can dynamically adjust their focal distance. The autofocus mechanism allows the camera to change its focus point based on the user's gaze direction, enabling sharp images of objects at varying distances while maintaining reasonable device complexity through automated control algorithms.
Solution Approach 2:
The patent implements Parameter changes by modifying the focal distance parameter of the camera based on eye-tracking data. When the user's gaze indicates interest in a specific direction, the system adjusts the camera's focal length to match the distance of objects in that direction, thereby maintaining image sharpness across different viewing scenarios without requiring a completely different camera system.
2Device complexity
If autofocus cameras are used without eye-tracking, then device complexity is reduced, but adaptability deteriorates because the autofocus system cannot determine user focus points
Solution Approach 1:
The patent applies the Feedback principle by using eye-tracking data as input to guide the autofocus mechanism. The eye-tracking system continuously monitors the user's gaze direction and provides feedback signals to the autofocus controller, which then adjusts the focal distance accordingly. This closed-loop feedback system enables the camera to adapt to user viewing preferences without requiring complex manual intervention or pre-programming.
Solution Approach 2:
The patent implements Self-service by enabling the autofocus system to automatically determine and adjust focus points based on user gaze behavior. Rather than requiring manual focus adjustment or complex scene analysis algorithms, the system uses the naturally occurring eye-tracking data to self-regulate the focal distance, making the autofocus operation intuitive and responsive to user needs.
3Manufacturing precision
If dynamic autofocus adjustment is implemented, then image sharpness is improved, but device complexity increases due to additional sensors and control mechanisms
Solution Approach 1:
The patent applies the Universality principle by designing the eye-tracking system to serve multiple functions: it provides gaze direction data for autofocus control, enables passthrough rendering from the correct viewpoint, and supports overall scene understanding. By making the eye-tracking sensor multi-functional, the patent avoids adding dedicated autofocus sensors, thereby reducing the net increase in device complexity while maintaining image sharpness through dynamic focus adjustment.
Data Source
AI summary
Particular embodiments are directed to passthrough image generation for a mixed-reality experience. A device may determine, using an eye-tracking system of a head-mounted device, eye-tracking data associated with a user of the head-mounted device. The device may determine, based on the eye-tracking data, a desired scene depth for the user. The device may instruct a first autofocus camera of the head-mounted device to adjust a first focus distance based on the desired scene depth and capture a first image of a real-world environment of the user. The device may generate a first passthrough image based on the first image. The device may display the first passthrough image to a first eye of the user via a first display of the head-mounted device.