Nested Neural Network Sub-models for Computational Resource Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep Neural Networks (DNNs) require significant computational resources for training and inference, which can be a challenge when computational budgets change or are reduced.
Innovation Solution
The use of a nested architecture for Neural Network models, comprising smaller 'sub-models' or 'sub-networks' that are shallower and/or narrower than the main model, along with a scalable methodology combining sub-model sampling and gradient accumulation to reduce computational costs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a large pre-trained language model is used, then performance on downstream tasks is improved, but computational resources and training costs increase significantly
Solution Approach 1:
The patent divides a large pre-trained language model into multiple smaller sub-models with different depths or configurations. Each sub-model can be independently trained and deployed, allowing the system to select appropriate model sizes based on available computational resources and performance requirements, thus resolving the contradiction between model performance and computational cost.
Solution Approach 2:
The patent employs techniques to dynamically adjust model parameters such as depth, width, or number of layers during training and inference. By varying these parameters, the system can optimize the balance between model capacity and computational resource consumption, achieving high performance when resources are abundant while maintaining functionality when resources are constrained.
2Use of energy by moving object
If model size is reduced to decrease computational cost, then training resources are reduced, but model performance deteriorates
Solution Approach 1:
The patent performs preliminary training on a large model to learn rich representations, then uses this pre-trained knowledge to initialize and train multiple smaller sub-models. This preliminary action allows smaller models to achieve better performance with less training cost, as they inherit knowledge from the larger model rather than learning from scratch.
Solution Approach 2:
The patent creates a nested architecture where smaller sub-models are embedded within or derived from a larger model structure. The smaller models share parameters or architectural components with the larger model, allowing them to maintain performance while requiring fewer computational resources for training and inference.
3Adaptability or versatility
If multiple sub-models are trained to handle different computational budgets, then adaptability is improved, but training complexity increases
Solution Approach 1:
The patent designs sub-models with universal architectures that can serve multiple functions across different computational budgets. The same base architecture is reused across different model sizes, allowing the system to handle various deployment scenarios while maintaining consistent training procedures and reducing overall training complexity.
Data Source
AI summary
Methods and processors for using a Neural Network (NN) are disclosed. The method includes, during a first iteration, determining a first sub-network of the NN and training the first sub-network. The method includes during a second iteration a second sub-network of the NN and training the second sub-network. The method includes during an inference iteration of the NN, selecting a target sub-network amongst the first sub-network and the second sub-network, and generating an inference output by employing only the target sub-network of the NN on inference data for reducing computational resources of the processor for generating the inference output.