Learner Model Mimics Reference Model via Vector Distance Updates
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep learning models are computationally intensive, limiting their application on devices with limited computational power, such as mobile phones, and existing solutions do not effectively mimic the behavior of more complex reference models with simpler learner models.
Innovation Solution
An artificial intelligence system that uses a structurally simpler learner model to mimic the behavior of a more complex reference model by iteratively updating parameters based on the difference between reference and learner vectors, determining convergence, and utilizing a Gradient Descent algorithm to adjust parameters.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a structurally more complicated reference model is used, then the accuracy and performance of the model are improved, but the computational overhead and resource requirements increase
Solution Approach 1:
The patent creates a learner model that copies the behavioral output of the reference model rather than replicating its complex internal structure. The learner model has a simpler architecture but is trained to produce similar outputs, thereby achieving comparable accuracy with reduced computational overhead.
Solution Approach 2:
The patent transforms the complex parameters and weights of the reference model into a different parameter space suitable for the simpler learner model. By changing the parameter representation and using knowledge distillation techniques, the simpler model can achieve similar performance without requiring the same computational resources.
2Use of energy by moving object
If a structurally simpler learner model is used, then the computational overhead is reduced, but the ability to effectively mimic the behavior of the complex reference model deteriorates
Solution Approach 1:
The patent employs feedback mechanisms during the training process where the learner model's outputs are continuously compared with the reference model's outputs. This feedback loop allows the simpler model to iteratively improve its behavior mimicry by adjusting its parameters based on the performance gap, thereby enhancing reliability without increasing structural complexity.
3Ease of manufacture
If the learner model structure is simplified, then the ease of deployment on resource-constrained devices is improved, but the training complexity and convergence difficulty increase
Solution Approach 1:
The patent performs preliminary actions by pre-training the complex reference model on abundant data before using it to train the simpler learner model. This two-stage approach where the reference model is first established and then used as a teacher for the learner model simplifies the overall training process and improves convergence, making deployment on resource-constrained devices more feasible.
Data Source
AI summary
A method for using a structurally more complicated reference model to train a structurally simpler learner model includes: obtaining a trained reference model at least including N reference blocks and a learner model at least including N learner blocks respectively corresponding to the N reference blocks; training the learner model by conducting an iterative operation; determining whether the learner model is convergent; and in response to that the learner model is convergent, stopping the iterative operation to assign the learner model as a trained learner model. The iterative operation includes inputting a sample data set into the trained reference model and the learner model; for each of the N learner blocks: determining a distance between a learner vector of the learner block and a reference vector of the reference block, and updating parameters in the learner block based on the determined distance.