Multi-modal Feature Correction Sub-model for Efficient Parameter Adjustment
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current multi-modal deep learning models face inefficiencies in parameter adjustment and storage burdens due to the need for complete fine-tuning and separate model parameter savings for each downstream task.
Innovation Solution
A method involving a target processing model comprising a multi-modal pre-trained sub-model and a multi-modal feature correction sub-model, where the multi-modal feature correction sub-model is trained with fixed parameters of the pre-trained model, allowing for efficient parameter adjustment and reduced storage needs by saving only the correction sub-model parameters for different tasks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the pre-trained model is completely fine-tuned for each downstream task, then the model can achieve high performance on specific tasks, but the parameter adjustment efficiency becomes low and the storage burden increases
Solution Approach 1:
The model parameters are segmented into two independent parts: shared pre-trained parameters and task-specific correction parameters. This segmentation allows the pre-trained model to be reused across multiple tasks while only the correction parameters need to be adjusted for each specific downstream task, thereby improving parameter adjustment efficiency while maintaining model performance.
Solution Approach 2:
The pre-trained model parameters are designed to be universal and applicable to multiple downstream tasks. By freezing these shared parameters and only training task-specific correction parameters, the system achieves multi-functionality where a single pre-trained model can serve multiple purposes without requiring complete fine-tuning for each task.
2Reliability
If a separate set of model parameters is saved for each downstream task, then the model can be optimized for each task, but the storage burden becomes serious as the number of models increases
Solution Approach 1:
The task-specific parameters are extracted from the complete model parameters. Instead of storing entire model copies for each task, only the correction parameters (the difference between task-specific and pre-trained parameters) are saved. This dramatically reduces storage requirements while preserving task-specific optimization capabilities.
Solution Approach 2:
Multiple task-specific models are merged into a unified structure consisting of a single shared pre-trained model combined with multiple small correction parameter sets. This merging approach eliminates redundant storage of identical pre-trained parameters across multiple tasks, reducing overall storage burden while maintaining the ability to perform different downstream tasks.
Data Source
AI summary
Embodiments of the disclosure disclose a method, apparatus, electronic device and storage medium for multi-modal data processing, wherein the method includes: acquiring data of original modality; and processing the data of the original modality by a target processing model to determine data of target modality corresponding to the data of the original modality; wherein the target processing model comprises a multi-modal pre-trained sub-model and a multi-modal feature correction sub-model; a training process of the target processing model comprises training the multi-modal feature correction sub-model with parameters of the multi-modal pre-training sub-model fixed.