Large-Model Cross-Modal Question Answering with Speech Pause Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional interaction methods in human-computer interaction are limited to single modalities, such as text input or simple visual feedback, failing to meet diverse user needs in complex scenarios, necessitating cross-modal processing for improved interaction.
Innovation Solution
A method for processing cross-modal question answering using a large model that includes activity detection on speech input, text recognition during pauses, and utilization of a pre-trained speech question answering system for response processing, incorporating a speech encoder and cross-attention large language model for enhanced interaction.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If traditional single-modality interaction methods are used, then system complexity is reduced, but interaction effectiveness and user experience deteriorate
Solution Approach 1:
The patent merges speech and text modalities into a unified cross-modal question answering system. The speech input is converted to text through speech recognition, then both the converted text and original speech features are processed together by the large language model to generate answers, achieving multi-modality integration while maintaining manageable system complexity through standardized processing pipelines.
Solution Approach 2:
The system implements multi-functionality by handling both speech and text inputs through a unified architecture. The same large language model processes both modalities after appropriate conversion, allowing the system to adapt to different input types (speech, text) and provide consistent question answering capabilities across multiple interaction modes.
2Adaptability or versatility
If cross-modal processing is implemented, then interaction versatility improves, but processing time and system complexity increase
Solution Approach 1:
The system performs preliminary speech recognition and text conversion before the main question answering processing. By converting speech to text in advance and preparing both modalities' features beforehand, the system reduces the processing time required during the actual interaction, as the large language model receives pre-processed inputs rather than having to handle raw speech in real-time.
3Measurement precision
If speech and text modalities are aligned through end-to-end processing, then response accuracy improves, but computational complexity increases
Solution Approach 1:
The system introduces text as an intermediary representation that bridges speech and the large language model processing. Speech inputs are converted to text through speech recognition, creating a common representation space that the large language model can process uniformly. This intermediary approach enables accurate cross-modal alignment while leveraging the existing capabilities of text-based language models, avoiding the need for complex direct speech-to-answer architectures.
Data Source
AI summary
A method for processing cross-modal question answering based on large model, an apparatus, and a storage medium are suggested, which relates to the field of artificial intelligence technologies such as speech interaction processing, large models, machine learning and natural language processing. The specific implementation includes: performing an activity detection on a target speech input by a user; in response to detecting a pause in the inputting of the target speech, obtaining a first text corresponding to a first input speech before the moment of the pause in the target speech; performing a text response processing using a pre-trained speech question answering processing system based on the first text and the first input speech.