Multi-modal Voice Recognition for Conversation Summarization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current voice recognition systems struggle to accurately identify and summarize important keywords in multi-speaker conversations, leading to the need for manual summarization, as they fail to effectively select and enhance recognition performance for key words.
Innovation Solution
A multi-modal voice recognition system that combines audio and video signals to extract feature and importance vectors, using a neural network to generate an acoustic model and output a recognition result by masking unimportant words, thereby focusing on key words and improving summarization performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional voice recognition systems are used to transcribe all conversation content, then complete transcripts can be generated, but the systems fail to accurately identify and summarize important keywords
Solution Approach 1:
The system extracts and separates important keywords from the complete conversation transcript by using an importance scoring mechanism. The neural network identifies and extracts only the salient words that contribute most to the conversation meaning, separating them from less important content to enable automatic summarization without manual intervention
Solution Approach 2:
An importance vector serves as an intermediary between the audio feature extraction and the final keyword identification. This vector acts as a mediator that weights and prioritizes different words in the transcript, enabling the system to automatically distinguish important keywords from less important content
2Reliability
If the system processes complete conversation transcripts, then all content is captured, but recognition performance for important keywords is not enhanced
Solution Approach 1:
The system applies different processing quality to different parts of the transcript by assigning importance scores to individual words. High-importance words receive enhanced processing and attention in the neural network, while less important words are processed with lower priority, creating local quality differentiation that improves keyword recognition
Solution Approach 2:
The system changes the parameter of word importance by introducing an importance vector that modifies the weight of each word in the transcript. This parameter change allows the neural network to focus computational resources on words with higher importance values, enhancing recognition reliability for key information
3Measurement precision
If multi-modal processing is implemented to extract importance vectors from audio and video signals, then keyword identification accuracy improves, but system complexity increases
Solution Approach 1:
The system segments the complex multi-modal processing into distinct functional modules: audio feature extraction, video feature extraction, importance vector generation, and neural network processing. This segmentation allows each component to handle a specific aspect of the task, improving measurement precision while making the overall system complexity more manageable through modular design
Data Source
AI summary
Disclosed herein is a voice recognition system with an enhanced summarization function according to the present invention. The voice recognition system include: an audio feature extractor configured to extract a voice feature from an audio signal to generate a feature vector; a salience extractor configured to extract a importance of speech from at least one of the audio signal or a video signal to generate an importance vector; and a neural network configured to output a recognition result based on the feature vector and the importance vector, in which the recognition result is output by masking some.