Low Latency Audio Mixing in Multi-Point Video Conferencing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current video conferencing systems suffer from high audio latency, exceeding 200 ms, which disrupts natural conversation and synchronized activities like music playing, due to factors like frame buffering, codec compression, and echo cancellation algorithms, making real-time audio transmission challenging.
Innovation Solution
The method involves capturing and transmitting lossless digital audio streams with frame sizes ranging from 0.25 ms to 3 ms, using a single frame buffer, and bypassing echo cancellation algorithms, allowing for low-latency audio mixing and playback through a multipoint control unit, employing Pulse-Code Modulation (PCM) and Real-time Transport Protocol (RTP) to minimize latency and maintain high audio quality.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If audio is transmitted using traditional frame buffering and codec compression, then audio quality is maintained, but latency increases to over 200 ms
Solution Approach 1:
The patent extracts and removes the audio codec compression component from the traditional audio transmission chain. By transmitting raw PCM audio data without codec compression, the system eliminates the algorithmic delay (3-7 ms) and avoids the need for complex decoding operations at the receiving end, thereby reducing latency while maintaining audio quality
Solution Approach 2:
The patent segments the audio transmission process into smaller, more frequent frames. Instead of using traditional 20 ms frames with three-frame buffering (creating 40-60 ms delay), the system uses 10 ms frames with reduced buffering, cutting the buffering latency approximately in half and enabling more responsive real-time audio transmission
2Reliability
If three-frame buffering is used in the audio codec, then glitches are avoided, but latency increases by 40 - 60 ms
Solution Approach 1:
The patent dynamically adjusts the buffering strategy by reducing from three-frame buffering to a single-frame buffer. This dynamic reduction in buffer depth decreases latency from 40-60 ms to approximately 10-20 ms while maintaining sufficient buffering to prevent audio glitches, achieving a balance between reliability and real-time performance
Solution Approach 2:
The system prepares audio data in advance by organizing it into 10 ms frames before transmission. This preliminary organization allows the receiving end to process and play back audio with minimal additional delay, reducing the need for extensive buffering at the receiver and thereby reducing overall latency
3Quantity of substance
If lossy compression is applied to maximize compression, then bandwidth is reduced, but audio quality deteriorates
Solution Approach 1:
The patent uses inexpensive, efficient compression algorithms like Huffmann encoding or Speex that provide sufficient compression for bandwidth efficiency without the quality degradation of complex lossy codecs. These simpler compression methods process audio faster with less algorithmic delay, maintaining both bandwidth efficiency and audio quality for real-time applications
4Object-affected harmful factors
If echo cancellation algorithm is implemented, then echo is avoided, but latency increases by 60 ms - several hundred ms
Solution Approach 1:
The patent removes the echo cancellation algorithm from the audio processing chain. By eliminating this component, the system reduces latency by 60 ms to several hundred ms, accepting that echo may occur in certain acoustic environments. This extraction prioritizes real-time performance over echo suppression
Data Source
Figure 1~2
Figure 3
AI summary
A real time audio multi-point videoconferencing system comprising at least two videoconferencing endpoints in communication with a multipoint control unit, and a method of providing the real time audio. The method comprising capturing input lossless digital audio streams on each of the at least two videoconferencing endpoints, transmitting the input lossless digital audio streams from each of the at least two videoconferencing endpoints to the MCU using a frame size in the range 0.25 ms - 3 ms, mixing, using the MCU, the input lossless digital audio streams into a plurality of output lossless digital audio streams, transmitting a different one of the plurality of output lossless digital audio streams to each of the at least two videoconferencing endpoints using a frame size in the range 0.25 ms - 3 ms, and playback the plurality of output lossless digital audio streams on each of the at least two videoconferencing endpoints.