AR Language Translation Using Gaze and Multimodal Speaker Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing automatic interpretation technologies face challenges in distinguishing desired voice from environmental and voice noise, and accurately interpreting multiple speakers in real-world situations without user inconvenience.
Innovation Solution
A gaze-based augmented interpretation system that utilizes image and voice information to detect and interpret voices within a user's visual field, generating results in the user's native language and displaying them on a screen or outputting synthesized voice, using multimodal models and user information databases.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If voice section detection using only voice information is used, then the system operates automatically without user input, but the interpretation performance is limited due to inability to distinguish desired voice from environmental noise
Solution Approach 1:
The patent combines image information and voice information into a unified processing framework. The image processing unit detects face regions and identifies speakers visually, while the voice processing unit extracts speech sections. By merging these modalities through the situation predictor and word extractor, the system achieves accurate voice detection without requiring manual button input, resolving the contradiction between automatic operation and detection accuracy.
Solution Approach 2:
The situation predictor acts as an intermediary that receives both image and voice information, analyzes the context, and generates situation predictions that guide the word extractor. This intermediary processing layer enables the system to distinguish desired speech from environmental noise by understanding the situational context, thereby improving voice detection accuracy while maintaining automatic operation.
2Loss of information
If interpretation results are provided for all detected speakers, then comprehensive information is available, but the system becomes complex and difficult to manage
Solution Approach 1:
The patent applies local quality by differentiating the processing level for each detected speaker based on situation predictions. The word extractor uses the predicted situation to selectively extract relevant words from specific speakers rather than uniformly processing all speakers. This allows the system to provide comprehensive information when needed while simplifying processing in specific contexts, reducing system management complexity.
Solution Approach 2:
The system dynamically adjusts which speakers are interpreted based on the predicted situation. The situation predictor continuously updates the understanding of the current context, and the word extractor adapts its selection of speakers to interpret accordingly. This dynamic adaptation allows the system to manage complexity by focusing computational resources on relevant speakers while maintaining the capability to provide comprehensive information when the situation requires it.
3Loss of information
If the system processes multiple speakers simultaneously, then all voices are interpreted, but the accuracy of distinguishing individual voices decreases due to voice noise and overlap
Solution Approach 1:
The patent segments the processing of multiple speakers by first detecting individual face regions in the image, then associating each face with a speaker identity. The voice processing is similarly segmented by extracting speech sections for each identified speaker based on situation predictions. This segmentation approach allows the system to process multiple speakers simultaneously while maintaining accurate individual voice distinction through separate processing streams for each speaker.
Data Source
AI summary
The present invention relates to an automatic interpretation method and system for converting only voice of a speaker into a target language. The present invention may significantly improve performance of automatic interpretation with a foreigner to be communicated with, even in a high-noise environment in which multiple speakers utter at the same time by utilizing voice and image information input to a smart device in a complex manner. In addition, the present invention may determine a situation based on text information and image information existing around a user, and reflect the situation information together with multimodal information to an interpretation engine in real time. In addition, the present invention may significantly improve user convenience of an automatic interpretation system by directly augmenting and displaying an interpreted sentence directly next to a speaker image or generating a synthesized sound by distinguishing the interpreted sentence from other speeches.