Voice Dialogue Platform Heartbeat Audio Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional man-machine interaction systems face issues with timeliness of responses and unreasonable sentence segmentation in full-duplex dialogue, leading to irrelevant answers and poor user experience due to network delays and discontinuous audio uploads.
Innovation Solution
Implementing a method that uses heartbeat events to determine if a user's speech is short, marking start and end times of audio inputs, and combining recognition results to form complete sentences, with feedback time marks to ensure accurate and timely responses, and setting time offsets to handle various interaction scenarios.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If continuous broadcasting of responses is implemented at the client end, then full-duplex interaction capability is achieved, but response timeliness deteriorates due to network and processing delays
Solution Approach 1:
The system performs preliminary actions by pre-processing audio segments and preparing recognition results before complete sentences are formed. The server processes audio segments as they arrive and prepares potential responses in advance, so when a complete sentence is identified, the response can be generated immediately without waiting for the entire sentence to be fully received and processed.
Solution Approach 2:
The patent segments audio input into smaller units that can be processed incrementally. By dividing the continuous audio stream into segments and processing them separately, the system can identify complete sentences earlier in the stream and generate responses without waiting for the entire audio input to be received, thereby improving response timeliness while maintaining full-duplex capability.
2Productivity
If sentence segmentation is performed based solely on acoustic features, then processing speed is improved, but sentence segmentation accuracy deteriorates in different speaking scenarios
Solution Approach 1:
The system dynamically changes segmentation parameters based on the speaking scenario. Different acoustic models and segmentation thresholds are applied depending on whether the user is speaking quickly, slowly, or with interruptions. This allows the system to maintain high processing speed while adapting segmentation accuracy to match the specific speaking characteristics of each scenario.
Solution Approach 2:
The sentence segmentation process is made dynamic rather than static. The system continuously adapts segmentation parameters based on real-time analysis of speaking patterns, pause durations, and contextual information. This dynamic approach allows the system to maintain high processing speed while accurately segmenting sentences across various speaking scenarios by adjusting to the current speaking rhythm and patterns.
3Ease of operation
If discontinuous audio upload is allowed, then user speaking flexibility is improved, but server's ability to determine actual interval between sentences deteriorates
Solution Approach 1:
The system implements feedback mechanisms where the server sends acknowledgments and status information back to the client about received audio segments. This feedback loop allows the server to communicate timing information and segment boundaries back to the client, enabling both parties to synchronize their understanding of the dialogue flow and sentence intervals even when audio upload is discontinuous.
Solution Approach 2:
The patent introduces an intermediary timing mechanism that mediates between the client's discontinuous audio uploads and the server's processing. This intermediary layer maintains a record of upload timestamps and segment boundaries, allowing the server to accurately determine actual intervals between sentences by referencing this intermediary timing information rather than relying solely on raw upload timing.
Data Source
Figure 1~2
Figure 3
Figure 4~5
AI summary
An embodiment of the present invention provides a method of man-machine interaction, including: receiving first audio uploaded by a user through a client end, marking a start time and an end time of the first audio, and generating a first recognition result of the first audio using an audio decoder; determining whether the first audio is a short speech based on the start time and end time thereof, and in case of a short speech, generating a second recognition result of the second audio using the audio decoder upon receiving the second audio uploaded by the client end within a preset heartbeat protection time range,; sending at least the first recognition result and the second recognition result to a language prediction model; and if it is determined that a combination of the recognition results constitutes a sentence, generating an answering instruction corresponding to the sentence, and sending the answering instruction together with a feedback time mark of the answering instruction to the client end. Unreasonable sentence segmentation in a full-duplex dialogue scenario and redundant replies in the dialogue can thereby be avoided.