Sparse High-Rank Adapters for Fast AI Model Switching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Adapting large generative AI models to new applications is computationally intensive, and existing adapter methods like LoRA incur high latencies and concept loss when rapidly switching or merging multiple adapters.
Innovation Solution
The introduction of sparse high-rank adapters (SHIRA) allows for rapid adapter switching and multi-adapter fusion without overwriting original weights, utilizing a small subset of pre-trained model weights and sparse masks to train and merge adapters efficiently.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If existing adapter methods like LoRA are used to adapt large generative AI models, then model adaptation is achieved, but high computational intensity and high latencies occur during rapid adapter switching
Solution Approach 1:
The patent segments the adapter weights into sparse subsets that can be independently stored and rapidly switched. Instead of loading entire adapter matrices, only sparse weight subsets corresponding to non-zero mask positions are transferred, enabling fast adapter switching while maintaining full adaptation capability.
Solution Approach 2:
The patent applies local quality by using sparse masks to identify and process only specific positions (non-zero elements) within the adapter weight matrices. This allows the system to focus computational resources only on relevant weight positions, reducing overall computational intensity and switching latency.
2Adaptability or versatility
If multiple adapters are merged using existing methods, then multi-adapter functionality is achieved, but concept loss occurs due to overwriting original weights
Solution Approach 1:
The patent extracts only the necessary sparse weight subsets from each adapter based on their respective masks. When merging multiple adapters, only the extracted sparse components are combined, leaving the original base model weights intact and preventing concept loss while achieving multi-adapter functionality.
Solution Approach 2:
The patent creates sparse copies of adapter weights that map to specific positions in the base model. Multiple adapters can be merged by combining these sparse copies at their respective mask positions without overwriting the original base model, preserving all concepts while enabling multi-adapter functionality.
3Adaptability or versatility
If dense adapter weights are used for model fine-tuning, then comprehensive model adaptation is achieved, but large memory footprint results
Solution Approach 1:
The patent employs sparse adapters that are analogous to porous materials - they have a structured emptiness (zero positions) that reduces their effective volume. The sparse mask creates a porous structure in the weight matrices, allowing the system to maintain full adaptation capability while occupying significantly less memory space by storing only non-zero elements.
Data Source
AI summary
A processor-implemented method includes receiving an artificial neural network having a number of pre-trained weights. The method also includes training a subset of the number of pre-trained weights to obtain trained sparse adapter weights for obtaining a fine-tuned version of the artificial neural network. The subset of the number of pre-trained weights includes base model weights of a base model for the artificial neural network. The subset of the number of pre-trained weights is selected with a sparse mask of a sparse adapter. The method may also include replacing the subset of the number of pre-trained weights with the trained sparse adapter weights.