Web Page Analysis for Automated Assistant Navigation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The existing development paradigm for automated assistants requires vendors to create third-party agents compatible with their API, which is resource-intensive and duplicative, as vendors often already have interactive web pages and mobile applications, leading to inefficiencies in scaling interactions with users.
Innovation Solution
Techniques are described to analyze existing interactive web sites, allowing automated assistants to automatically engage with them by abstracting, validating, and generalizing actions, enabling users to interact through spoken or typed dialog sessions without requiring direct visual interaction or the vendor to implement a third-party agent, using universal identifier matrices to adapt to changes in web page layouts and identify similar user states for successful action replication.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If vendors create third-party agents compatible with automated assistant APIs, then the automated assistant can directly interact with vendor services, but the development process becomes resource-intensive and difficult to scale
Solution Approach 1:
The patent introduces a web page analysis system as an intermediary between the automated assistant and vendor web pages. This system automatically analyzes vendor web pages, extracts interactive elements, and generates actionable data structures, eliminating the need for vendors to manually create third-party agents while maintaining reliable interaction
Solution Approach 2:
The vendor web pages automatically provide their own interaction data through the analysis system. The web pages self-describe their interactive elements and workflows, allowing the automated assistant to understand and interact with them without requiring external agent development
2Adaptability or versatility
If vendors create custom third-party agents, then direct API interaction is enabled, but duplication of effort occurs since vendors already have interactive web interfaces
Solution Approach 1:
The patent creates a universal analysis system that can process any vendor web page regardless of the specific vendor or service. This single system serves multiple functions: analyzing web page structure, extracting interactive elements, generating action data, and enabling automated assistant integration across diverse vendor platforms
Solution Approach 2:
Instead of having vendors create custom agents, the system creates a standardized representation (copy) of the vendor web page's interactive elements. This copy captures the essential interaction data in a universal format that the automated assistant can use without requiring the original vendor-specific implementation
3Measurement precision
If automated assistants require visual interaction with web pages, then precise navigation is possible, but user convenience decreases due to the need for direct visual engagement
Solution Approach 1:
The patent replaces the mechanical visual interaction system with an automated analysis system. Instead of requiring users to visually navigate web pages and manually interact with elements, the system automatically analyzes the web page structure, identifies interactive elements, and executes actions based on extracted data, substituting automated processing for manual visual engagement
Data Source
AI summary
Implementations are described herein for analyzing existing interactive web sites to facilitate automatic engagement with those web sites, e.g., by automated assistants or via other user interfaces, with minimal effort from the hosts of those websites. For example, in various implementations, techniques described herein may be used to abstract, validate, maintain, generalize, extend and/or distribute individual actions and “traces” of actions that are useable to navigate through various interactive websites. Additionally, techniques are described herein for leveraging these actions and/or traces to automate aspects of interaction with a third party website. For example, in some implementations, techniques described herein may enable users to engage with an automated assistant (via a spoken or typed dialog session) to interact with the third party website without requiring the user to visually interact with the third party web site directly and without requiring the third party to implement their own third party agent.