Deep Neural Network Style Transfer for Mixed Reality Passthrough
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current mixed-reality systems require multiple cameras to generate passthrough visualizations, leading to increased weight, cost, and battery usage, while failing to optimize the viewing experience with enhanced data.
Innovation Solution
A deep neural network (DNN) is used to transition the style of images from one camera type to another, allowing a single camera to capture and process multiple types of data, such as thermal and low-light images, by learning and modifying image styles to match different camera perspectives.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If multiple cameras are used to generate passthrough visualizations, then the viewing experience is enhanced with diverse data, but the weight, cost, and battery usage increase
Solution Approach 1:
The patent uses a deep neural network to learn the imaging characteristics of different camera types and generate synthetic images that copy the appearance and data characteristics of thermal, low-light, or other specialized camera outputs. A single visible light camera captures images, and the DNN processes these images to create multiple style variations, effectively replacing multiple physical cameras with one camera plus computational processing.
2Reliability
If multiple cameras are used to generate passthrough visualizations, then the viewing experience is enhanced with diverse data, but the device cost increases
Solution Approach 1:
The deep neural network serves as a universal processor that can generate multiple types of image data (thermal, low-light, enhanced visible light) from a single camera input. This multi-functional approach allows the system to provide diverse data streams without requiring multiple specialized camera hardware components, thereby reducing device cost while maintaining enhanced viewing experience quality.
3Reliability
If multiple cameras are used to generate passthrough visualizations, then the viewing experience is enhanced with diverse data, but the battery consumption increases
Solution Approach 1:
The system captures images with a single visible light camera and uses a deep neural network to computationally generate synthetic thermal and low-light images by copying and transforming the visual information. This approach consumes less battery power than operating multiple power-hungry specialized cameras simultaneously, as the DNN processing is more energy-efficient than multiple camera sensors and processing pipelines.
4Weight of moving object
If a single camera is used, then the device weight, cost, and battery usage are reduced, but the ability to capture multiple types of data is limited
Solution Approach 1:
The deep neural network dynamically changes the parameters and characteristics of captured images through style transfer and transformation. By adjusting computational parameters rather than physical camera parameters, the system can generate thermal-like, low-light-like, and enhanced visible light images from a single camera, thereby achieving multi-type data capture capability with reduced device weight.
Data Source
AI summary
Modifications are performed to cause a style of an image to match a different style. A first image is accessed, where the first image has the first style. A second image is also accessed, where the second image has a second style. Subsequent to a deep neural network (DNN) learning these styles, a copy of the first image is fed as input to the DNN. The DNN modifies the first image copy by transitioning the first image copy from being of the first style to subsequently being of the second style. As a consequence, a modified style of the transitioned first image copy bilaterally matches the second style.