Vision-Language Model Pruning for Inference Speed
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current vision-language models are inefficient due to high computational costs, making them unsuitable for deployment in resource-constrained environments, and conventional pruning methods can damage the representativity of embeddings and misalign modalities in the embedding space.
Innovation Solution
A progressive pruning process is applied to the embedding neural network, selectively pruning neurons with the least influence on downstream tasks and fine-tuning the model multiple times to maintain alignment across modalities, reducing the number of parameters and improving inference speed.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the vision-language model is trained with more parameters to improve capability, then the model capability is improved, but the computational cost increases
Solution Approach 1:
The patent extracts and removes redundant or less important neurons from the vision-language model through progressive pruning. By identifying and eliminating unnecessary computational elements, the model achieves reduced computational cost while maintaining essential capabilities through iterative pruning and fine-tuning cycles.
Solution Approach 2:
The patent changes the parameter configuration of the model by dynamically adjusting the number of active neurons through progressive pruning. The model transitions from a dense initial state to a sparser optimized state, modifying parameters iteratively to balance capability and computational efficiency.
2Device complexity
If conventional pruning methods are applied to reduce model size, then the number of parameters is reduced, but the representativity of embeddings is damaged and modalities become misaligned
Solution Approach 1:
The patent performs preliminary fine-tuning before and between pruning stages to prepare the model for structural changes. By pre-adjusting parameters and performing iterative fine-tuning, the model maintains embedding representativity throughout the pruning process, preventing modality misalignment before it occurs.
Solution Approach 2:
The patent implements feedback mechanisms through iterative pruning and fine-tuning cycles. The model's performance and embedding quality are continuously monitored, and fine-tuning adjustments are made based on feedback from each pruning stage, ensuring that representativity is maintained while reducing parameter count.
3Productivity
If the model is pruned to reduce computational cost, then the inference speed is improved, but the cross-modal retrieval accuracy may deteriorate
Solution Approach 1:
The patent applies dynamic pruning strategies where the model structure is adaptively adjusted through multiple stages. Rather than static pruning, the model undergoes iterative refinement where pruning decisions are made dynamically based on performance metrics, allowing inference speed to improve while maintaining retrieval accuracy through adaptive structural changes.
Solution Approach 2:
The patent applies partial pruning in progressive stages rather than removing excessive neurons at once. By implementing gradual, controlled pruning with intermediate fine-tuning, the model achieves speed improvements through parameter reduction while preventing accuracy deterioration by maintaining sufficient neural capacity at each stage.
Data Source
AI summary
Systems and methods for reducing inference time of vision-language models, as well as for multimodal search, are described herein. Embodiments are configured to obtain an embedding neural network. The embedding neural network is pretrained to embed inputs from a plurality of modalities into a multimodal embedding space. Embodiments are further configured to perform a first progressive pruning stage, where the first progressive pruning stage includes a first pruning of the embedding neural network and a first fine-tuning of the embedding neural network. Embodiments then perform a second progressive pruning stage based on an output of the first progressive pruning stage, where the second progressive pruning stage includes a second pruning of the embedding neural network and a second fine-tuning of the embedding neural network.