On-Device ML Model Evaluation Across Variable Client Hardware
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Federated learning techniques often result in sub-optimal performance of on-device ML models due to varying device characteristics across different client devices, leading to unnecessary waste of computational resources.
Innovation Solution
Evaluating on-device ML models based on performance measures such as latency, memory consumption, and CPU usage, and adjusting their activation or sparsity levels based on these measures to optimize performance for individual devices.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Area of stationary object
If a single on-device ML model is shared across multiple client devices with different device characteristics, then device independence and simplicity are improved, but model performance and resource efficiency deteriorate
Solution Approach 1:
The patent implements device-specific model configurations where the ML model is customized for each client device based on its characteristics. The system determines device characteristics (processor type, memory capacity, storage capacity) and configures the ML model accordingly, allowing each device to run an optimized version of the model tailored to its local capabilities rather than a universal one-size-fits-all approach.
2Productivity
If device-specific ML model optimization is implemented, then model performance is improved, but device complexity and evaluation requirements increase
Solution Approach 1:
The patent performs preliminary evaluation of the ML model on each client device before full deployment. The system evaluates the model's performance and resource consumption on the specific device, using this pre-assessment data to determine the appropriate configuration. This preliminary action prevents unnecessary complexity by only activating device-specific optimizations when they are actually beneficial.
Solution Approach 2:
The system implements a feedback mechanism where the ML model's performance and resource usage are continuously monitored on each device. Based on this feedback, the system automatically adjusts model configuration parameters such as model size, complexity, and resource allocation. This closed-loop control simplifies the overall system by using automated feedback-driven adjustments rather than manual configuration complexity.
Data Source
AI summary
Implementations disclosed herein are directed to systems and methods for evaluating on-device machine learning (ML) model(s) based on performance measure(s) of client device(s) and/or the on-device ML model(s). The client device(s) can include on-device memory that stores the on-device ML model(s) and a plurality of testing instances for the on-device ML model(s). When certain condition(s) are satisfied, the client device(s) can process, using the on-device ML model(s), the plurality of testing instances to generate the performance measure(s). The performance measure(s) can include, for example, latency measure(s), memory consumption measure(s), CPU usage measure(s), ML model measure(s) (e.g., precision and/or recall), and/or other measures. In some implementations, the on-device ML model(s) can be activated (or kept active) for use locally at the client device(s) based on the performance measure(s). In other implementations, the on-device ML model(s) can be sparsified based on the performance measure(s).