Predictive Model Compression Using Synthetic Label Transfer
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Complex predictive models are opaque, brittle, and storage-inefficient, making them difficult to understand, update, and deploy, especially in resource-constrained environments, while conventional compression techniques compromise predictive accuracy and require excessive labeled data.
Innovation Solution
A method involving the generation of an unlabeled simulated data set, prediction by a complex model to create a labeled data set, and training a neural network model to replicate the complex model's predictions with incremental updates, allowing for a compressed, accurate, and explainable model.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Volume of stationary object
If conventional compression techniques are applied to complex predictive models, then model size is reduced, but predictive accuracy deteriorates
Solution Approach 1:
The patent creates a compressed model that copies the predictive behavior of the complex model by training it on synthetic data generated from the complex model's predictions. The compressed model learns to replicate the complex model's input-output relationships without inheriting its complexity, thereby maintaining accuracy while reducing size.
Solution Approach 2:
The patent performs preliminary actions by generating synthetic training data using the complex model before training the compressed model. This pre-computation of labeled synthetic data enables the compressed model to learn effective predictions without requiring access to the complex model during deployment, resolving the accuracy-size tradeoff.
2Volume of stationary object
If conventional compression techniques are applied to complex predictive models, then model size is reduced, but extensive labeled data is required
Solution Approach 1:
The patent uses the complex model to generate synthetic labeled data that copies the statistical properties and prediction patterns of real data. This synthetic data serves as a substitute for extensive real labeled data, enabling compressed model training without requiring large amounts of manually labeled real-world data.
Solution Approach 2:
The complex model serves itself by generating the training data needed for the compressed model. The complex model's predictions on synthetic inputs create the labeled dataset automatically, eliminating the need for external data annotation processes and reducing dependency on extensive real labeled data.
3Measurement precision
If complex predictive models are used, then predictive accuracy is improved, but model interpretability deteriorates
Solution Approach 1:
The patent segments the complex model's functionality into a simplified compressed model that retains only the essential prediction capabilities. By separating the complex internal transformations from the input-output relationship, the compressed model achieves interpretability while maintaining accuracy through its streamlined architecture.
Solution Approach 2:
The compressed model copies only the necessary predictive behavior from the complex model, discarding the complex intermediate representations. This selective copying creates a simpler model that is easier to interpret while preserving the essential prediction accuracy needed for the application.
Data Source
AI summary
A computer-implemented method for improving compression of predictive models includes generating an unlabeled simulated data set by expanding an initial data set, and generating a labeled data set by predicting the unlabeled, simulated data set using a complex model to output a plurality of labels. The method also includes training a relatively simple neural network using the labeled data set.