Predictive Model Compression Using Simulated Data Distillation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Complex predictive models are opaque, brittle, and storage inefficient, making them difficult to understand, update, and deploy, especially in resource-constrained environments, while conventional compression techniques compromise predictive accuracy and require excessive labeled data.
Innovation Solution
A method involving the generation of an unlabeled simulated data set, labeling it using a complex model, and training a neural network model to produce accurate predictions, allowing for incremental updates and efficient storage without significant degradation in predictive accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a complex predictive model is used to achieve high predictive accuracy, then the model can make sophisticated predictions, but the model becomes opaque and difficult to understand
Solution Approach 1:
The patent creates a simplified copy (compressed model) that replicates the predictive behavior of the complex model. The compressed model is trained to mimic the input-output relationships of the complex model, capturing its essential predictive capabilities while eliminating unnecessary complexity and improving interpretability.
2Measurement precision
If a complex predictive model is used to achieve sophisticated predictions, then the model can handle multiple transformations and sub-models, but the model requires excessive storage space
Solution Approach 1:
The patent creates a compressed copy of the complex model that retains predictive accuracy while dramatically reducing storage requirements. The compressed model uses fewer parameters and a simplified architecture, eliminating the need to store multiple sub-models and transformation layers while maintaining the ability to make accurate predictions.
3Measurement precision
If a complex predictive model is used to achieve accurate predictions, then the model can make sophisticated predictions, but the model becomes brittle and difficult to update
Solution Approach 1:
The patent separates the complex model into two distinct components: a fixed complex model that captures the primary predictive relationships, and a simplified compressed model that can be independently updated. This segmentation allows the compressed model to be retrained and updated without affecting the complex model, improving adaptability and ease of maintenance.
4Quantity of substance
If conventional compression techniques are used to reduce model size, then storage efficiency improves, but predictive accuracy degrades and extensive labeled data is required
Solution Approach 1:
The patent creates a compressed copy of the complex model that is trained specifically to replicate the complex model's predictions. By using the complex model's predictions on unlabeled data to create training labels for the compressed model, the approach achieves accurate compression without requiring extensive external labeled data or sacrificing predictive accuracy.
Data Source
AI summary
A computer-implemented method for improving compression of predictive models includes generating an unlabeled simulated data set by expanding an initial data set, and generating a labeled data set by predicting the unlabeled, simulated data set using a complex model to output a plurality of labels. The method also includes training a relatively simple neural network using the labeled data set.