Multimodal Interface Selection for Accurate Split-Screen Voice Control
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing speech control technologies for intelligent terminals with multiple display interfaces in the same space are inflexible, often controlling the wrong interface and leading to poor user experience due to inability to determine which interface the user intends to interact with.
Innovation Solution
An interface control method utilizing collaborative decision-making based on multimode information such as sound source location, line-of-sight tracking, and speech semantics to accurately control the intended interface.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If speech control is used to control the head unit interface, then driving safety is ensured, but user experience is poor due to inability to accurately control the intended interface
Solution Approach 1:
The patent introduces line-of-sight tracking information and sound source location information as intermediary elements to bridge the gap between speech instructions and the intended target interface. The line-of-sight tracking module captures where the driver is looking, and the sound source location module determines where the speech originates, using these intermediaries to accurately identify which interface the driver intends to control among multiple interfaces
Solution Approach 2:
The patent adds spatial dimension information to the speech control system. By incorporating line-of-sight direction and sound source location in three-dimensional space, the system can distinguish between multiple interfaces on different screens or positions, transforming a two-dimensional interface selection problem into a three-dimensional spatial localization problem
2Adaptability or versatility
If multiple display interfaces are provided in the same space, then functional versatility is improved, but interface identification accuracy deteriorates due to inability to determine user intent
Solution Approach 1:
The patent applies local quality by making each display interface have distinct spatial characteristics. Each interface is associated with specific spatial information including its position on the screen, the direction it faces, and its location in three-dimensional space. This allows the system to identify which interface the user intends to control based on the local spatial context rather than treating all interfaces uniformly
3Ease of operation
If touch operation is used to control the head unit interface, then ease of operation is improved, but driving safety deteriorates due to requiring user attention and time
Solution Approach 1:
The patent merges multiple control modalities into a unified speech-based control system. By combining speech recognition with line-of-sight tracking and sound source location, the system creates a single integrated control method that provides both the ease of operation of speech input and the safety of hands-free operation, eliminating the need to choose between touch operation convenience and driving safety
Data Source
Figure 1
Figure 2
Figure 3
AI summary
This application provides an interface control method. The method includes: obtaining a speech instruction of a user and a sound source location of the user; obtaining line-of-sight information of the user; determining a target window on an interface based on the sound source location and the line-of-sight information; and controlling the target window based on the speech instruction. According to the interface control method in this application, collaborative decision-making is performed with reference to multimode information such as sound source information, line-of-sight tracking information, speech semantic information, and priorities thereof, so that page content in a plurality of windows on the interface is quickly and accurately controlled, to improve user experience.