Expanding XR Field of View via Side Camera Depth Layering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Extended reality (XR) systems, such as mixed reality (MR) and augmented reality (AR), have a limited field of view (FoV) due to the limitations of forward-facing cameras, which restricts the immersive experience and natural viewing perspective.
Innovation Solution
The use of side cameras with wide FoV, such as fisheye cameras, is leveraged to expand the FoV by generating additional image layers with depth information, allowing for a more immersive and natural view by projecting actual scene content into the extended regions, creating a virtual camera effect with a shallow depth of field.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Area of stationary object
If forward-facing cameras are used to capture images for XR systems, then stereo view and depth perception are provided, but the field of view is limited
Solution Approach 1:
The patent combines images from multiple camera systems (forward-facing cameras and side cameras) to create an expanded field of view. The side camera images are integrated with the forward-facing camera images through image processing techniques, merging the wide-angle peripheral view with the stereo depth information to resolve the contradiction between limited FoV and system complexity.
Solution Approach 2:
The side cameras, which could be used solely for peripheral detection, are made multi-functional by using them to expand the field of view while maintaining their original detection capabilities. This allows the same hardware to serve multiple purposes: providing both the expanded peripheral view and the depth information needed for stereo perception.
2Area of stationary object
If side cameras with wide FoV are used to expand the field of view, then immersive experience is improved, but image distortion and depth accuracy deteriorate
Solution Approach 1:
The patent applies different processing techniques to different regions of the image. The central region from forward-facing cameras maintains high depth accuracy, while the peripheral regions from side cameras are processed with undistortion algorithms. This local differentiation allows the system to maintain depth accuracy where needed while expanding the overall field of view.
Solution Approach 2:
The patent uses depth mapping and image processing algorithms as intermediaries to transfer and adapt depth information from the forward-facing cameras to the side camera images. This intermediary processing layer corrects the depth inaccuracies introduced by the wide-angle side cameras while preserving their expanded field of view benefit.
3Area of stationary object
If depth information is extended to create expanded FoV layers, then field of view is expanded, but processing complexity and computational load increase
Solution Approach 1:
The patent performs preliminary depth estimation and layer creation from the forward-facing camera images before integrating the side camera images. By pre-processing the depth information and creating the expanded FoV layers in advance, the system reduces the computational complexity of the final image integration step.
Solution Approach 2:
The patent divides the final expanded FoV image into multiple layers: the central stereo region from forward-facing cameras and the peripheral extended regions from side cameras. This segmentation allows independent processing of each region with appropriate algorithms, reducing overall processing complexity compared to processing the entire image uniformly.
Data Source
AI summary
Methods and apparatus for providing an expanded field of view (FoV) using multiple cameras of a device. A device may include front-facing cameras that capture views of an environment, as well as side cameras that capture wider-angle views. Images from the front and side cameras may be processed to provide a stereo view of the environment with an expanded FoV. Images from the front cameras are processed to provide stereo images, and depth information is generated from the stereo images. The depth information is expanded into a second layer, and a median depth is set for a third layer. The side camera images are undistorted, pixel information from the side camera images is then reprojected into the second and third layers, and the second and third layers are blurred.