Neural Network Model Compression via PU Classifier Data Selection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural network model compression technologies face challenges in reducing data transmission while maintaining accuracy, as they either require massive training data, leading to poor user experience, or use limited data resulting in unsatisfactory compression results.
Innovation Solution
A method involving a server that obtains a neural network model and training data, uses a positive-unlabeled classifier to select extended data with similar properties and distributions, and employs knowledge distillation to train a second neural network model, reducing data transmission while ensuring accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If massive training data is provided to achieve good network convergence result, then the compression accuracy is improved, but the data transmission time is increased and user experience deteriorates
Solution Approach 1:
The patent extracts only the essential and representative data from the massive training dataset through PU learning-based selection. Instead of transmitting all training data, the system identifies and selects a subset of data that captures the key characteristics and distribution patterns, thereby reducing transmission time while preserving compression accuracy.
Solution Approach 2:
The patent changes the parameter of data selection from exhaustive selection to selective sampling based on PU learning. By transforming the selection criteria and using unsupervised learning to identify representative samples, the system reduces the volume of transmitted data while maintaining the essential information needed for accurate model compression.
2Loss of time
If only a small amount of training data is used to compress the model, then the data transmission is reduced, but the compressed neural network model cannot achieve satisfactory result
Solution Approach 1:
The patent introduces PU learning as an intermediary mechanism between the full training dataset and the compressed model. This intermediary selection process identifies and extracts representative data samples that serve as an effective bridge, allowing the system to use a small amount of transmitted data while still achieving compression accuracy comparable to using massive datasets.
Solution Approach 2:
The patent creates a representative copy or subset of the training data that captures the essential characteristics and distribution of the full dataset. Through PU learning-based selection, the system generates a condensed version of the training data that preserves the key information needed for model compression, eliminating the need to transmit the complete original dataset.
3Measurement precision
If the first neural network model is used as teacher network model in KD method, then the knowledge transfer accuracy is improved, but the computational complexity is increased
Solution Approach 1:
The patent applies partial action by using the teacher network model only for knowledge distillation on the selected representative dataset rather than on the entire training dataset. This partial application of the teacher model reduces computational complexity while maintaining knowledge transfer accuracy, as the essential knowledge can be captured from a smaller, carefully selected data subset.
Data Source
AI summary
This application provides a neural network model compression method in the field of artificial intelligence. The method includes: obtaining, by a server, a first neural network model and training data of the first neural network that are uploaded by user equipment; obtaining a PU classifier based on the training data of the first neural network and unlabeled data stored in the server; selecting, by using the PU classifier, extended data from the unlabeled data stored in the server, where the extended data has a property and distribution similar to a property and distribution of the training data of the first neural network model; and training a second neural network model by using a knowledge distillation (KD) method based on the extended data, where the first neural network model is used as a teacher network model and the second neural network model is used as a student network model.