Voice-Driven UI Graph Control for Apps Without Open APIs
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing electronic apparatuses face limitations in controlling applications without an open API, requiring manual operations for UI updates, leading to functional gaps when UIs change.
Innovation Solution
An electronic apparatus with a memory and processor that stores UI graphs and edges, enabling natural language understanding of user voices to identify actions, update UIs, and perform actions based on identified sequences, even in the absence of an open API.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Extent of automation
If voice control is implemented for applications without open API, then automation capability is improved, but system complexity increases due to need for UI graph construction and maintenance
Solution Approach 1:
The system performs preliminary actions by automatically constructing UI graphs during application installation or initial execution, and pre-defining action sequences for common operations. This allows the system to have control capabilities ready before actual voice commands are issued, reducing real-time processing complexity while maintaining high automation.
Solution Approach 2:
The UI graph serves as an intermediary data structure that mediates between voice commands and application operations. It abstracts the complex application interface into a structured format with nodes representing UI elements and edges representing possible actions, making the system manageable while enabling automation for applications without open APIs.
2Reliability
If manual operations are required for UI updates, then system reliability is maintained, but productivity decreases due to additional manual intervention
Solution Approach 1:
The system performs self-service by automatically detecting UI changes through screenshot analysis and comparing them against the stored UI graph. When changes are detected, the system automatically updates the UI graph structure and re-generates action sequences, eliminating the need for manual intervention while maintaining system reliability through automated validation.
Solution Approach 2:
The system implements feedback mechanisms where the actual UI state is continuously monitored and compared with the expected state in the UI graph. This feedback loop enables automatic detection of UI updates and triggers appropriate updates to the action sequences, ensuring the system remains synchronized with the application without manual input.
3Ease of operation
If fixed operations are assigned to UI manipulation commands, then ease of operation is improved, but adaptability decreases when UI updates occur
Solution Approach 1:
The system applies dynamics by making action sequences flexible rather than fixed. Action sequences are stored as structured data that can be automatically regenerated when UI changes are detected. This allows the system to maintain simple voice commands for ease of operation while adapting the underlying action sequences to match current UI states, resolving the contradiction between simplicity and adaptability.
Data Source
AI summary
An electronic apparatus is provided. The electronic apparatus includes a memory configured to store a plurality of nodes corresponding to a plurality of user interface (UI) types, for each application, a display, and a processor connected with the memory and the display and controls the electronic apparatus, wherein the processor is configured to identify a first UI graph corresponding to the target application, acquire information on a target node that will perform the user command among a plurality of first nodes included in the first UI graph based on the user command and the at least one parameter, identify the current node corresponding to a UI displayed through the display, identify an action sequence from the current node to the target node based on the information on the current node and the target node, and perform an action corresponding to the user voice based on the action sequence.