On-Sensor Federated Learning for Private AR Model Adaptation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Machine learning models trained on inadequate data due to privacy concerns and lack of raw image sharing lead to poor prediction accuracy, especially in systems like virtual and augmented reality headsets.
Innovation Solution
Implementing smart sensors with computing resources for local adaptation and a system on chip (SoC) to perform incremental training, combined with encrypted data transmission to ensure privacy and security, allowing local user adaptation and global model improvement.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If raw images are shared for training machine learning models, then training data quantity is improved, but data privacy and security are worsened
Solution Approach 1:
The patent segments the training process into local and remote components. Local smart sensors perform incremental training on raw images to extract features, while only encrypted feature data is transmitted to the server for global model updates. This segmentation allows sufficient training data to be processed locally without exposing sensitive raw images to external servers.
Solution Approach 2:
The patent introduces encryption as an intermediary mechanism between the local smart sensor and the server. Encrypted feature data is transmitted to the server, which processes the encrypted data and returns updated model parameters. This intermediary layer enables data exchange while maintaining privacy and security.
2Object-affected harmful factors
If machine learning models are trained on inadequate data due to privacy constraints, then data privacy is maintained, but prediction accuracy is worsened
Solution Approach 1:
The patent divides the training function between local smart sensors and the server. Local sensors accumulate sufficient training data and perform incremental training to maintain privacy, while the server aggregates results from multiple devices to improve overall model accuracy. This segmentation resolves the contradiction by enabling both privacy preservation and accuracy improvement.
Solution Approach 2:
The patent implements a feedback loop where the server receives encrypted feature data from multiple smart sensors, updates the global model, and sends back updated parameters. This feedback mechanism allows continuous improvement of prediction accuracy while maintaining data privacy through encrypted local processing.
3Adaptability or versatility
If local adaptation of machine learning models is performed, then user-specific customization is improved, but device complexity is worsened
Solution Approach 1:
The patent segments computing tasks between the smart sensor and the server. The smart sensor handles local feature extraction and incremental training with a simplified architecture, while the server performs complex global model updates. This segmentation enables user customization without overburdening the device complexity.
Solution Approach 2:
The patent applies partial action by having smart sensors perform only the necessary local processing (feature extraction and incremental training) rather than complete model training. The server handles the computationally intensive global model updates, allowing local devices to achieve customization with reduced complexity.
Data Source
AI summary
An artificial reality system performs local user adaptation of machine learning models and global improvement of the machine learning models while ensuring data security and privacy. An artificial reality headset uses smart sensors and a system on chip (SoC). The smart sensor extracts features from sensor data, encrypts them and transmits the encrypted features to the SoC of the headset. The SoC provides the encrypted features to a server, for example, a server in a cloud platform. The server updates a base model with collected features and sends the updated base model parameters to headsets. The smart sensors of a headset receiving the updated model parameters, balance the model parameters with locally stored feature data, for example, user calibration features to perform user customization of the machine learning model.