Visual Language Understanding for Display-Referenced Voice Commands
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing natural language processing systems struggle to accurately interpret user inputs that refer to currently or previously displayed content, leading to inefficiencies in user interactions.
Innovation Solution
A system that utilizes automatic speech recognition (ASR) and natural language understanding (NLU) to process user inputs in conjunction with visual embeddings of displayed content, enabling precise identification of user intent and actions related to displayed images or videos.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If natural language processing systems use traditional text or audio input processing, then the system can process user inputs, but it cannot accurately interpret references to displayed content
Solution Approach 1:
The patent merges audio processing components (microphone, ASR module) with visual processing components (display processor, VLM module) into a unified system. The display processor generates visual embeddings from displayed content and feeds them to the VLM, which processes both visual and audio inputs together to accurately interpret user references to displayed content, resolving the contradiction between accuracy and complexity through integrated architecture
Solution Approach 2:
The patent introduces visual embeddings as an intermediary representation that bridges the gap between displayed content and user input processing. The display processor converts visual content into embeddings that serve as mediators for the VLM to understand and interpret user references to displayed content, enabling accurate interpretation without direct complex visual processing in the audio path
2Productivity
If the system processes user inputs without visual context, then processing is simpler and faster, but interaction efficiency and accuracy deteriorate
Solution Approach 1:
The display processor performs preliminary action by continuously generating visual embeddings from displayed content and storing them in a buffer before user input occurs. This pre-computed visual context is immediately available when audio input is received, eliminating the need for time-consuming real-time visual processing during user interaction, thus improving productivity without significant time loss
Solution Approach 2:
The system creates a simplified representation (copy) of the visual content in the form of visual embeddings that capture essential information about displayed content. These embeddings serve as compressed copies that the VLM can process efficiently without analyzing the full visual content in real-time, enabling fast and efficient interaction
3Reliability
If the system integrates visual processing with audio processing, then accuracy of interpreting displayed content references improves, but computational resources and processing complexity increase
Solution Approach 1:
The patent segments the processing system into distinct functional modules: a display processor that handles visual content and generates embeddings, and a VLM that handles audio input and integrates visual context. This segmentation allows each component to process its specific data type efficiently, reducing overall computational resource consumption while maintaining high accuracy in interpreting content references through specialized processing
4Measurement precision
If the system uses traditional speech recognition without visual context, then processing is faster and simpler, but the ability to understand references to displayed content is limited
Solution Approach 1:
The patent adds a new dimension to the processing system by introducing visual embeddings as an additional input dimension to the VLM. Instead of processing only audio inputs, the system combines audio inputs with visual embedding data, creating a multi-dimensional input space that enables precise identification of user intent related to displayed content while maintaining manageable system complexity through structured integration
Data Source
AI summary
Techniques for performing an action with respect to displayed content are described. A natural language interpretation corresponding to a received spoken user input may be determined. Prior to receiving the spoken user input, content may be displayed to the user from which the spoken user input was received. The natural language interpretation may represent a request to perform an action with respect to a portion of the content currently being displayed. Content identifiers corresponding to content being displayed, may be determined, and embedding data representing at least one feature of the content may be determined using the content identifiers. The natural language interpretation and the embedding data may be processed to determine that the spoken user input relates to a first portion of the displayed content instead of a second portion of the displayed content. Based on the determination, an action responsive to the spoken user input may be performed.