Multimodal Large Model Deployment via Visual-Linguistic Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Multimodal large models face challenges in practical deployment due to their large number of parameters and complex computational requirements, leading to inefficient hardware resource utilization and increased inference time.
Innovation Solution
The method involves splitting a multimodal large model into a visual part and a linguistic part, determining corresponding static graph models for each part, and deploying the model based on these static graph models, optimizing hardware resource utilization and inference speed.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a multimodal large model is deployed as a whole, then the model can maintain complete functionality, but hardware resource utilization becomes inefficient and inference time increases
Solution Approach 1:
The patent divides the multimodal large model into multiple sub-models corresponding to different processing stages (e.g., embedding layer, encoder layers, decoder layers). Each sub-model is deployed as an independent static graph model, allowing parallel execution and optimized resource allocation. This segmentation resolves the contradiction by improving inference speed through parallel processing while maintaining efficient hardware resource utilization through distributed deployment.
2Device complexity
If the model is split into multiple parts, then hardware resource utilization improves, but the complexity of model deployment increases
Solution Approach 1:
The patent performs graph optimization and converts each sub-model into a static graph model in advance, before actual deployment. This preliminary action includes optimizing the computational graph, fusing operations, and preparing the models for efficient execution. By doing this preparation work beforehand, the system achieves efficient hardware resource utilization during deployment while the complexity is managed during the offline optimization phase rather than during runtime deployment.
3Speed
If static graph models are used for each part, then inference speed improves, but memory consumption increases
Solution Approach 1:
The patent divides the large model into multiple smaller sub-models, each converted to a static graph model. While each static graph model consumes memory, the segmented approach allows for optimized memory management where each sub-model's memory footprint is smaller and can be managed independently. The overall system achieves better memory efficiency compared to loading the entire model at once, while still benefiting from the speed improvements of static graph execution for each segment.
Data Source
AI summary
Provided is a method of deploying a multimodal large model, an electronic device and a storage medium, relating to field of artificial intelligence technology, and in particular, to fields of deep learning and model deployment. The method includes: splitting a first multimodal large model into a visual part and a linguistic part; determining a first static graph model corresponding to the visual part and a second static graph model corresponding to the linguistic part; and deploying the first multimodal large model based on the first static graph model and the second static graph model.