Mixed Reality Wearables With Cloud Neural Network Offloading
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing VR and AR systems are limited by size, power requirements, latency, and lack of robust connectivity, making them unsuitable for critical operations like emergency response and military applications, and they fail to effectively balance local and cloud resources for processing and storage.
Innovation Solution
A distributed computing and networking system for mixed reality systems that includes a head-wearable device with a camera and inertial measurement unit, connected to remote servers for image processing and neural network training, enabling robust, portable, and highly-connected wearable computing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If local computing resources are used for VR/AR processing, then processing speed is improved, but device size and power requirements increase
Solution Approach 1:
The computing workload is segmented into two parts: image capture and initial processing are performed locally on the head-wearable device, while complex neural network training and heavy computational tasks are performed remotely on cloud servers. This segmentation allows the wearable device to remain small and portable while still achieving high processing speeds through distributed computing.
Solution Approach 2:
A wireless communication interface acts as an intermediary between the head-wearable device and remote servers. The device captures images locally, transmits them via wireless communication to remote servers for processing, and receives results back. This intermediary approach enables high-capability processing without requiring large local computing resources.
2Speed
If local computing resources are used for VR/AR processing, then processing speed is improved, but power consumption increases
Solution Approach 1:
Energy-intensive neural network training and complex computational tasks are segmented and executed remotely on cloud servers rather than locally on the wearable device. The device only performs lightweight image capture and data transmission, significantly reducing its power consumption while maintaining high processing speeds through remote computing resources.
Solution Approach 2:
Wireless communication serves as an intermediary that enables the device to offload heavy computational tasks to remote servers. The device captures images locally using minimal power, transmits them via wireless communication, and receives processed results without requiring substantial local processing power or battery capacity.
3Adaptability or versatility
If pass-through video is used in VR systems, then user awareness of environment is improved, but latency and image fidelity decrease
Solution Approach 1:
The patent uses neural networks as an intermediary to process and enhance pass-through video feeds. The neural networks compensate for latency and improve image fidelity by predicting and correcting temporal distortions, allowing users to maintain awareness of their environment with acceptable performance in critical applications.
Solution Approach 2:
The system implements feedback mechanisms where neural networks continuously analyze pass-through video feeds and adjust processing in real-time to compensate for latency. This feedback loop ensures that the visual information presented to users remains synchronized with their physical movements, maintaining situational awareness despite the inherent delays in pass-through video transmission.
4Productivity
If distributed computing is used for image processing, then processing capability is improved, but system complexity increases
Solution Approach 1:
A standardized wireless communication interface serves as an intermediary that simplifies the connection between the head-wearable device and remote servers. The interface handles all complex data transmission, authentication, and protocol management automatically, allowing the device to leverage distributed computing capabilities without exposing the user or system architect to the underlying complexity.
Solution Approach 2:
The system implements self-service mechanisms where the head-wearable device automatically manages its own data transmission and processing tasks through integrated neural networks and communication interfaces. The device autonomously determines when to offload tasks to remote servers and how to receive and process results, reducing the need for complex manual configuration and system management.
Data Source
AI summary
Disclosed herein are systems and methods for distributed computing and/or networking for mixed reality systems. A method may include capturing an image via a camera of a head-wearable device. Inertial data may be captured via an inertial measurement unit of the head-wearable device. A position of the head-wearable device can be estimated based on the image and the inertial data via one or more processors of the head-wearable device. The image can be transmitted to a remote server. A neural network can be trained based on the image via the remote server. A trained neural network can be transmitted to the head-wearable device.