Dynamic Speech Navigation via View Hierarchy Mapping
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing speech-based user interfaces for electronic devices are limited to pre-defined commands and require updates to screen elements, restricting dynamic interaction with viewable screens.
Innovation Solution
An electronic device and method that enables screen-specific user interfacing by converting speech input into text sequences, comparing them with text representations of viewable screen elements, and selecting corresponding elements, allowing for dynamic action triggering without pre-programmed commands.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If pre-defined speech commands are used to control screen elements, then the system has a fixed and manageable command set, but it cannot dynamically interact with new or updated screen elements without manual updates to the command list
Solution Approach 1:
The patent introduces a view hierarchy file as an intermediary layer between the speech recognition system and the screen elements. This file contains text representations of all interactive elements on the screen, allowing the system to dynamically map speech input to current screen elements without hardcoding commands. The view hierarchy acts as a mediator that automatically bridges user speech and UI elements, resolving the contradiction between adaptability and complexity.
Solution Approach 2:
The system enables self-service by automatically generating the mapping between speech commands and screen elements through the view hierarchy file. When the screen content changes, the view hierarchy is updated automatically, and the speech system adapts without manual intervention. The system serves itself by dynamically creating command mappings based on the current screen state, eliminating the need for developers to manually update command lists.
2Reliability
If speech commands are updated to match new screen elements, then the system remains synchronized with the UI, but this requires manual maintenance and increases development effort
Solution Approach 1:
The view hierarchy file enables the system to automatically maintain synchronization between speech commands and screen elements. The text representations in the view hierarchy file are generated automatically from the UI framework, ensuring that speech commands always correspond to current screen elements without manual updates. This self-updating mechanism maintains reliability while eliminating development effort.
Solution Approach 2:
The patent uses text representations from the view hierarchy file as copies of screen element identifiers. These text copies are used by the speech recognition system to match user input with screen elements, ensuring synchronization without direct coupling between the speech system and UI code. Any changes to screen elements automatically update the text copies, maintaining reliability without manual intervention.
3Adaptability or versatility
If a comprehensive list of speech commands is maintained for all possible screens, then all functions are accessible via speech, but the command list becomes large and difficult to manage
Solution Approach 1:
The patent segments the comprehensive command list into individual text representations stored in a view hierarchy file. Instead of maintaining one large command list, the system divides commands into screen-specific text representations that are automatically organized by the UI framework. This segmentation allows the system to access only relevant commands for the current screen, reducing management complexity while maintaining comprehensive coverage.
Solution Approach 2:
The system transitions from a static command list to a dynamic structure where available speech commands are determined at runtime based on the current screen state. The view hierarchy file is generated dynamically for each screen, providing only the commands relevant to that specific screen context. This dynamic approach maintains comprehensive coverage while automatically managing complexity by showing only necessary commands at any given moment.
Data Source
AI summary
Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for enabling screen-specific user interfacing with elements of viewable screens presented by an electronic device are disclosed. In one aspect, a method includes the actions of identifying a character sequence representing a first input that is received while displaying a viewable screen having at least one selectable viewable element. The actions further include accessing an electronic file that provides a text representation of one or more of the at least one selectable viewable element. The actions further include comparing the character sequence to the text representation. The actions further include selecting, within the viewable screen, a selectable viewable element whose text representation matches the character sequence. The actions further include triggering any action linked to the selecting the selectable viewable element.