Voice-Controlled UI Navigation via Screen Component Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing mobile terminal systems face challenges in simplifying user interactions, particularly for complex functions, as users often need to manually operate devices to perform tasks, which can be cumbersome and inefficient.
Innovation Solution
The system utilizes screen component information of installed applications to automatically perform user-requested functions through voice commands, generating realistic user input events by analyzing UI elements and their functions, allowing devices to execute tasks without manual operation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If users manually operate the device to perform tasks, then the device can execute functions, but the cognitive and operational burden on users increases
Solution Approach 1:
The patent replaces manual mechanical operations (touching, clicking, navigating) with voice-based acoustic input. The voice recognition system converts spoken commands into digital signals that directly control application functions, eliminating the need for manual UI interaction and reducing operational burden on users.
Solution Approach 2:
The patent introduces a voice recognition system as an intermediary between the user and the device functions. This mediator translates natural language voice commands into structured control signals, bridging the gap between user intent and device execution without requiring direct manual interaction with complex interfaces.
2Ease of operation
If the system uses voice commands to perform functions, then user interaction is simplified, but the system requires complex processing of voice input and screen component analysis
Solution Approach 1:
The patent segments the complex task of voice-based function execution into distinct modules: voice input acquisition, voice recognition processing, screen component information analysis, matching logic, and execution control. Each module handles a specific aspect of the process, making the overall complex system manageable through functional decomposition.
Solution Approach 2:
The patent creates a universal voice command processing framework that can handle multiple different functions across various applications through a single integrated system. The screen component information database and matching mechanism provide multi-functional capability, allowing the same voice interface to control diverse operations without requiring application-specific customization.
3Extent of automation
If the system analyzes screen component information to generate realistic user input events, then automated function execution is enabled, but the processing time and computational resources increase
Solution Approach 1:
The patent performs preliminary actions by pre-analyzing and storing screen component information in a structured database format before voice commands are issued. The system maintains an updated representation of UI elements, their functions, and hierarchical relationships in advance, so that when a voice command arrives, the matching and execution process can proceed quickly without performing full screen analysis in real-time.
Data Source
Figure 1A
Figure 1B
Figure 1C
AI summary
Provided is a device including a display, an audio inputter, and a controller. The display displays at least one screen page of an application that is being executed. The audio inputter receives a voice command of a user. The controller performs an operation corresponding to the voice command by using screen page transition information for transition between application screen pages corresponding to the voice command, which is obtained from information about user interface (Ul) elements included in the application screen pages of the application. Each of the Ul elements performs a predetermined function when selected by the user.