Natural Language Description Generation via Multi-Model Feature Fusion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for generating natural language descriptions of media objects, such as videos, using encoder-decoder architectures result in low accuracy, making them difficult to apply widely in practical applications.
Innovation Solution
The method involves processing a media object using multiple natural language description models to obtain first feature vectors, fusing these vectors to create a second feature vector, and then generating a target natural language description, which integrates different types of information to more accurately represent the media object.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a single encoder-decoder network is used to generate natural language description, then the device complexity is low, but the description accuracy is low
Solution Approach 1:
The patent divides the description generation task into multiple specialized networks, each responsible for extracting specific feature types (visual features, audio features, text features). This segmentation allows each network to focus on particular aspects of the media object, improving overall description accuracy while maintaining manageable complexity through modular architecture
Solution Approach 2:
The patent combines multiple types of feature vectors (visual, audio, text) into a fused comprehensive feature representation. By integrating heterogeneous feature types from different modalities, the system creates a composite feature set that captures richer semantic information, thereby improving description accuracy
2Measurement precision
If multiple natural language description models are used to process media objects, then the description accuracy is improved, but the computational cost increases
Solution Approach 1:
The patent extracts only the necessary feature vectors from each specialized network rather than processing complete data streams. By selecting and extracting specific visual, audio, and text features that are most relevant to description generation, the system reduces unnecessary computational operations while maintaining high description accuracy
Solution Approach 2:
The patent merges the feature vectors from multiple specialized networks into a single fused feature representation. This consolidation integrates information from visual, audio, and text modalities into one comprehensive input for the description generation network, improving accuracy while avoiding redundant processing of separate feature streams
Data Source
Figure 1~2
Figure 3~4
Figure 5~6
AI summary
A method and a device for generating natural language description information. Wherein, the method comprises: respectively processing media resources through a plurality of natural language description models to obtain first feature vectors output by the natural language description models, the first feature vectors output by the nature language description models correspond to different feature types (S 102); fusing the plurality of first feature vectors to obtain a second feature vector (S 104); and generating natural language description information according to the second feature vector (S 106). The method solves the technical problem that the natural language description information generated for the media resources is low in the accuracy of the media resource description in the related art.