Automated Dialogue Script Discovery via GUI Trace Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current technologies face challenges in automatically accessing and integrating human-oriented interfaces for computer-implemented systems, particularly in retrieving content from web-based services without extensive programming, and in providing a seamless robotic interaction experience.
Innovation Solution
The development of an automated system that discovers and configures interfaces by analyzing content, probing for input combinations, and simulating human-computer interactions to retrieve content, using techniques such as crawling and machine learning to identify required information elements and create templates for automated dialogue systems.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Extent of automation
If automated web crawling and content extraction techniques are used, then automated access to web content is improved, but the system cannot effectively access content through human-oriented interfaces that require interaction
Solution Approach 1:
The patent introduces a robotic interface agent that acts as an intermediary between automated systems and human-oriented interfaces. This agent simulates human interactions by performing actions such as filling forms, clicking buttons, and navigating interfaces, thereby enabling automated access to content behind interactive interfaces without requiring the interfaces to be specifically designed for machine access.
2Extent of automation
If screen scraping and robotic interaction techniques are used, then automated interface access is improved, but extensive programming is required for each interface
Solution Approach 1:
The system employs self-service mechanisms where the robotic interface agent automatically discovers interface elements, learns interaction patterns, and adapts to different interfaces without requiring extensive pre-programming. The agent can autonomously perform tasks such as identifying form fields, understanding button functions, and navigating interface structures, thereby reducing the programming burden while maintaining automated interaction capabilities.
3Ease of operation
If conventional GUI-based interfaces are used, then user interaction is improved, but automated access requires extensive programming
Solution Approach 1:
The robotic interface agent serves as a mediator that translates automated system requests into human-oriented interface interactions. It maintains the ease of operation for human users while enabling automated access by simulating human actions such as typing, clicking, and navigating, thereby bridging the gap between conventional GUI design and automated system needs without requiring interface redesign.
4Extent of automation
If web services with structured interfaces are used, then automated access is improved, but most computer-implemented systems only provide human-oriented interfaces
Solution Approach 1:
Instead of requiring systems to provide structured web service interfaces for automated access, the patent inverts the approach by enabling automated access to human-oriented interfaces through robotic agents. This allows existing systems with only human-oriented interfaces to be accessed automatically without requiring them to implement structured interfaces, thereby maintaining interface availability while achieving automation.
Data Source
AI summary
A method for configuring an automated dialogue system uses traces of interactions via a graphical user interface (GUI) for an application. Each trace includes interactions in the context of a plurality of presentations of the GUI. Elements of one or more presentations of the GUI are identified, and templates are associated with portions of the trace. Each template has one or more defined inputs and a defined output. For each template of the plurality of templates, the portions of the traces are processed to automatically configure the template by specifying a procedure for providing values of inputs to the template via the GUI and obtaining a value of an output. The automated dialogue system is configured with the configured templates, thereby avoiding manual configuration of the dialogue system.