Onboard Shadow Validation of Vehicle ML Models at Fleet Scale
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The challenge of efficiently tuning and improving machine learning models for diverse vehicle fleets, considering situational applicability and scalability, poses a significant burden on backend servers due to the vast amount of data and unique vehicle experiences, leading to fragmented deployment strategies and slow model improvement.
Innovation Solution
A vehicle-based AI/ML Ops Pipeline that leverages onboard, edge, and cloud computing to validate newly trained models through shadow execution and real-time benchmarking against performance expectations, allowing for personalized and global improvements without continuous manual oversight.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If machine learning models are continuously tuned and improved for each individual vehicle using vast amounts of vehicle data, then model performance and adaptability improve, but the burden on backend servers becomes massive and scalability deteriorates
Solution Approach 1:
The patent segments the model validation process by implementing shadow execution hardware that runs parallel to production hardware. This allows model validation to occur independently on dedicated hardware resources rather than burdening backend servers with validation computations for each vehicle, thereby maintaining model performance while improving scalability.
Solution Approach 2:
The patent introduces a new dimension of validation by comparing model outputs across different hardware platforms (production vs. shadow execution hardware). This dimensional approach enables validation of model portability and performance without increasing computational burden on existing servers, resolving the contradiction between reliability and productivity.
2Adaptability or versatility
If machine learning models are tuned for global applicability across diverse vehicle fleets, then model versatility improves, but the ability to address individual vehicle circumstances deteriorates
Solution Approach 1:
The patent applies local quality by enabling individual vehicles to have customized model configurations while maintaining a global validation framework. Each vehicle can tune models for its specific circumstances (local) while the shadow execution hardware ensures these local adaptations meet global performance standards, thus resolving the contradiction between versatility and reliability.
3Measurement precision
If manual processes are used to marshal and distill massive amounts of vehicle data, then data processing accuracy improves, but the time and resources required become virtually impossible to complete
Solution Approach 1:
The patent implements self-service by enabling the system to automatically validate models through shadow execution hardware without requiring manual data marshaling and distillation. The hardware automatically compares model outputs against expected results, maintaining accuracy while eliminating the time-consuming manual processes that would be impossible to complete at scale.
4Productivity
If new machine learning models are deployed without validation, then deployment speed improves, but model reliability and performance deteriorates
Solution Approach 1:
The patent applies preliminary action by performing model validation in advance through shadow execution hardware before full deployment. This preliminary validation ensures model reliability is established beforehand, allowing rapid deployment without compromising performance, thus resolving the contradiction between productivity and reliability.
Data Source
AI summary
A vehicle system receives indication of a newly trained machine learning model designated for validation. The system load a copy of the model into shadow execution hardware, capable of background execution of the model and subscribes to one or more data topics to which input data for the model, gathered by a vehicle data gathering process, is published. The system executes the model in the background as the vehicle travels, using data published to the data topics and benchmarks output from the model to determine whether the model outperforms a prior version of the model, that represents the model prior to the model being newly trained, based on relative performance of both models compared to performance expectations defined in a configuration file stored by the vehicle. Also, the system, responsive to the model outperforming the prior version of the model based on the performance expectations defined by the configuration file, validates the model as suitable for deployment.