Deep Ensemble Distillation via Iterative Retrospective Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep ensembles consume excessive computational resources, limiting their adoption and deployment, and existing knowledge distillation techniques either sacrifice inferencing accuracy or fail to support uncertainty quantification when reducing their size.
Innovation Solution
An iterative knowledge distillation method that trains a new neural network in each iteration using a retrospective loss function based on previously trained networks, ensuring non-redundancy and minimizing the size of the condensed deep ensemble while maintaining accuracy and uncertainty quantification capabilities.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Use of energy by moving object
If knowledge distillation is implemented to reduce computational resources, then computational footprint is reduced, but uncertainty quantification capability is lost
Solution Approach 1:
The patent segments the ensemble into multiple smaller sub-ensembles rather than collapsing it into a single network. Each sub-ensemble maintains enough complexity to perform uncertainty quantification while being smaller than the full ensemble, thus resolving the contradiction between reducing computational resources and preserving uncertainty quantification capability
Solution Approach 2:
The patent implements a nested structure where multiple levels of ensembles are created hierarchically. The condensed ensemble contains sub-ensembles that are further condensed, allowing the system to maintain uncertainty quantification at multiple scales while progressively reducing the overall computational footprint
2Reliability
If a condensed deep ensemble with fixed size is used, then uncertainty quantification is maintained, but inferencing accuracy trades off against computational footprint
Solution Approach 1:
The patent transforms the fixed-size condensed ensemble into a dynamic, adaptive structure. The ensemble size and composition can be adjusted based on the specific task requirements, allowing optimization of the trade-off between uncertainty quantification capability and computational footprint for different application scenarios
Solution Approach 2:
The patent changes key parameters of the ensemble structure, including the number of sub-ensembles, the size of each sub-ensemble, and the distillation coefficients. These parameter adjustments enable fine-tuning of the balance between maintaining uncertainty quantification and reducing computational complexity
3Use of energy by moving object
If a single neural network is used for distillation, then computational resources are reduced, but uncertainty quantification capability is lost
Solution Approach 1:
Instead of distilling to a single network, the patent segments the distillation process into multiple sub-ensembles. Each sub-ensemble processes different aspects of the knowledge from the teacher ensemble, collectively preserving uncertainty information while reducing the total computational burden compared to the full ensemble
Solution Approach 2:
The patent introduces intermediate condensed sub-ensembles as mediators between the full teacher ensemble and the final deployment model. These intermediaries preserve and transfer uncertainty information through the distillation process, preventing information loss while enabling computational efficiency
Data Source
AI summary
Systems/techniques that facilitate improved distillation of deep ensembles are provided. In various embodiments, a system can access a deep learning ensemble configured to perform an inferencing task. In various aspects, the system can iteratively distill the deep learning ensemble into a smaller deep learning ensemble configured to perform the inferencing task, wherein a current distillation iteration can involve training a new neural network of the smaller deep learning ensemble via a loss function that is based on one or more neural networks of the smaller deep learning ensemble which were trained during one or more previous distillation iterations.