Multimodal Touch Selection via Audio Coarse-Fine Input
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Mobile computing devices with touch screens face challenges in accurately selecting smaller elements due to limited touch resolution and the presence of multiple layers at a given location, making it difficult to precisely choose elements, especially when fingers are not precise enough.
Innovation Solution
An apparatus comprising a processor and memory that receives coarse-grained and fine-grained indications, allowing users to select multimodal elements on a touch screen by combining touch gestures and audio signals to execute commands on specific portions of electronic documents.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If touch screen interface is used for element selection, then device portability and ease of use are improved, but selection accuracy deteriorates due to limited touch resolution and multiple layers at given location
Solution Approach 1:
The patent introduces audio signals as an intermediary modality to bridge the gap between coarse touch input and fine element selection. The audio signal acts as a mediator that provides precise identification of target elements, complementing the imprecise touch gesture and enabling accurate selection without improving touch screen resolution
Solution Approach 2:
The patent transitions from a single-dimensional touch interface to a multi-dimensional interaction system by incorporating audio signals as an additional dimension. This allows the system to differentiate between multiple elements at the same screen location by adding temporal and acoustic dimensions to the selection process
2Quantity of substance
If smaller elements are displayed on touch screen, then information density is improved, but selection difficulty increases due to insufficient touch resolution
Solution Approach 1:
The patent replaces the mechanical touch-based selection system with an audio-based identification system. Instead of relying on mechanical precision of finger movement to select small elements, the system uses audio signals to identify and select elements, substituting the mechanical selection mechanism with an acoustic recognition mechanism
3Area of stationary object
If multiple layers of elements are placed at given location, then screen space utilization is improved, but selection precision deteriorates because touch point cannot distinguish between layers
Solution Approach 1:
The patent segments the selection process into two distinct phases: coarse localization through touch and fine identification through audio. This segmentation allows multiple layers to coexist at the same location while providing distinct audio identifiers for each layer, enabling precise selection without reducing screen space utilization
Data Source
AI summary
For selecting multimodal elements, a method is disclosed that includes receiving a coarse-grained indication indicating a coarse-grained portion of an electronic document, receiving an audio signal comprising a fine-grained indication indicating a fine-grained portion of the coarse-grained portion and a command to be performed on the fine-grained portion, and executing the command on the fine-grained portion in response to receiving the audio signal.