Composed Neural Networks for Low-Data Task Adaptation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large foundation models, such as LLMs, are challenging and expensive to augment with new skills due to their monolithic structure, especially when only a small number of training examples are available, and fine-tuning can degrade existing capabilities.
Innovation Solution
Compose a base neural network with smaller augmenting neural networks through cross-attention, training only the learned transformation and cross-attention mechanism on limited task data, preserving existing capabilities and adding minimal parameters.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If fine-tuning is applied to a large base neural network to perform a new task, then the model can learn the new task, but the existing capabilities of the model are degraded and the computational cost is prohibitively expensive
Solution Approach 1:
The model is segmented into a base neural network and separate augmenting neural networks. The base network maintains existing capabilities while frozen, and augmenting networks are added specifically for new tasks. This segmentation prevents degradation of existing capabilities while enabling new task performance.
Solution Approach 2:
The base neural network serves multiple functions: it maintains its original capabilities while also serving as a foundation for new tasks through the augmenting networks. The cross-attention mechanism enables the base network to be universally applicable across different tasks without retraining.
2Adaptability or versatility
If fine-tuning is applied to a large base neural network to perform a new task, then the model can learn the new task, but the computational cost and data requirements become prohibitively expensive
Solution Approach 1:
The learning capability for new tasks is extracted from the base network and placed into separate augmenting neural networks. Only these smaller augmenting networks are trained on new tasks with limited data, significantly reducing computational cost while maintaining adaptability.
Solution Approach 2:
Instead of training the entire base network, only the necessary augmenting components are trained. This partial action approach uses minimal computational resources and data while achieving the desired adaptability for new tasks.
3Device complexity
If the base neural network structure is kept monolithic, then the model is simple to implement, but it is challenging and expensive to augment with new skills
Solution Approach 1:
The model structure transitions from a static monolithic design to a dynamic composable architecture. The base network remains simple, but augmenting networks can be dynamically added, removed, or modified based on task requirements, enabling flexible skill acquisition.
Solution Approach 2:
Augmenting neural networks are nested within the base network architecture through cross-attention mechanisms. This nesting allows the simple base structure to contain additional functional layers that provide new capabilities without disrupting the underlying simplicity.
Data Source
AI summary
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for composing machine learning models to perform new tasks.