Natural Language to UI Action Mapping
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Mobile applications present challenges in creating a standard test environment for tracking tags, especially with the rise of voice interaction and natural language inputs, making it difficult to determine intended actions and involved UI elements.
Innovation Solution
A computer system that renders a user interface, receives natural language inputs specifying element descriptions and action types, identifies matching UI elements, and maps these inputs into executable code, allowing for the execution and verification of actions, with secondary actions implemented if initial ones do not function as expected.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If mobile applications use voice interaction and natural language inputs, then user interaction capability is improved, but determining intended actions and involved UI elements becomes more difficult
Solution Approach 1:
The system presents multiple possible interpretations of the natural language input to the user and receives feedback to select the correct interpretation. This feedback loop resolves the ambiguity in determining intended actions and UI elements by allowing the user to confirm or correct the system's understanding.
Solution Approach 2:
The system introduces an intermediary step between receiving natural language input and executing actions - it generates multiple possible interpretations as intermediate results before final execution. This intermediary layer allows for disambiguation and selection of the correct interpretation.
2Reliability
If multiple tags from various parties are embedded in webpages, then tracking capability is improved, but managing and auditing tags becomes more complex
Solution Approach 1:
The system provides a universal testing framework that can handle multiple types of tags from various parties through a single unified interface. This multi-functional approach allows auditors to manage and test different tag types consistently without needing separate processes for each tag provider.
Solution Approach 2:
The system changes the parameters of tag testing by allowing dynamic configuration of test conditions, input data, and expected outcomes. This enables flexible management of complex tag environments where tags may have different parameters, formats, and testing requirements.
Data Source
AI summary
Embodiments are directed to methods and systems for deriving automation instructions. In one scenario, a computer system derives automation instructions by performing the following: rendering a user interface (UI) based on information from an information source and receiving natural language inputs from a user, where the natural language inputs specify an element description and an action type for UI elements rendered in the UI. The method also includes identifying UI elements in the UI that match the element descriptions in the natural language input and whose actions are performable according to their specified action type, and mapping the natural language inputs into executable code using information that corresponds to the identified UI elements.