Edge Node Model Performance Estimation via Proxy and Synthetic Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In machine learning, determining the performance of a trained model without access to ground truth or true output is challenging, especially when the enormous size of training data prevents it from being carried to edge nodes with limited memory and bandwidth.
Innovation Solution
The method involves receiving a trained model and generator models at an edge node, using the trained model to produce labeled testing data, training a proxy model on this data, and then using the proxy model on synthetic training data generated by the generator models to estimate the performance of the trained model.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If training data is carried to edge nodes for performance assessment, then measurement precision is improved, but device complexity and resource requirements worsen due to limited memory and bandwidth
Solution Approach 1:
The patent extracts only the essential statistical properties and distribution characteristics of the training data, rather than transferring the entire training dataset to edge nodes. This extraction approach maintains the ability to assess model performance while dramatically reducing the resource burden on edge devices.
Solution Approach 2:
The patent creates synthetic training data that replicates the statistical properties and distribution of the original training data. This synthetic copy allows performance assessment at edge nodes without requiring access to the actual large-scale training data, thus resolving the contradiction between measurement precision and device complexity.
2Measurement precision
If ground truth is used for performance measurement, then measurement precision is improved, but loss of information worsens when ground truth is unavailable in production environments
Solution Approach 1:
The patent introduces synthetic training data as an intermediary that bridges the gap between unavailable ground truth and the need for performance measurement. The synthetic data preserves the statistical properties necessary for accurate assessment while being generatable in environments where original ground truth is inaccessible.
Solution Approach 2:
The patent changes the measurement approach from relying on original ground truth labels to using synthetic labels generated from models trained on synthetic training data. This parameter change enables continuous performance monitoring in production environments where ground truth is typically unavailable.
3Reliability
If full training data is transferred to edge nodes, then reliability of performance assessment is improved, but loss of time and bandwidth increases
Solution Approach 1:
The patent extracts only the critical statistical properties and distributional characteristics from the training data, transferring minimal information to edge nodes. This extraction maintains assessment reliability while reducing transfer time and bandwidth consumption to negligible levels.
Solution Approach 2:
The patent creates compact synthetic training data that replicates the essential properties of the full training dataset. This synthetic copy can be generated locally or transferred efficiently, maintaining reliability without the time and bandwidth costs of transferring the original large-scale training data.
Data Source
AI summary
Techniques for determining system performance without ground truth include receiving a trained model and one or more generator models, the trained model having been trained on training data. The trained model is used on testing data to produce labeled testing data, and the labeled testing data are used to train a proxy model. The one or more generator models are used to produce synthetic training data that are representative of the training data. The proxy model is used on the synthetic training data to produce predictions, and performance of the trained model is determined based on the predictions by the proxy model.