DOM Navigator Agent for Low-Bandwidth Browser Collaboration
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing screen-sharing technologies face challenges such as performance degradation due to low-bandwidth connectivity, compatibility issues across diverse systems, and exposure of sensitive information, particularly in collaborative sessions involving computer use agents like chatbots and virtual assistants.
Innovation Solution
A collaborative session is initiated between a user application and an AI agent using a bi-directional, synchronous communication channel that manipulates the document object model (DOM) of the active browser page, reducing data transmission by sharing only necessary elements, and utilizing a multi-modal large language model (LLM) to generate and execute DOM actions, thereby optimizing bandwidth and enhancing security.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional screen-sharing is used for collaborative sessions, then users can share their screen with AI agents, but performance degrades due to low-bandwidth connectivity and high data transmission requirements
Solution Approach 1:
The patent extracts and transmits only the essential DOM elements and their state changes rather than the entire screen image. This selective extraction of necessary information significantly reduces bandwidth consumption while maintaining collaborative session functionality.
Solution Approach 2:
The screen content is segmented into discrete DOM elements that can be individually identified, transmitted, and reconstructed. This segmentation allows the system to send only the specific elements that changed, rather than transmitting the complete screen, thereby improving performance on low-bandwidth connections.
2Adaptability or versatility
If screen-sharing is implemented across diverse server and client systems, then collaboration is enabled, but compatibility issues arise due to diverse operating systems and software version differences
Solution Approach 1:
The patent uses DOM elements as a universal interface that works across different operating systems and browsers. The DOM is a standardized web technology that provides consistent behavior across diverse platforms, enabling the collaborative session system to function reliably without requiring platform-specific implementations.
3Loss of information
If complete screen-capture is transmitted during screen-sharing, then full visual information is available to AI agents, but sensitive information may be inadvertently exposed due to user error or insufficient security
Solution Approach 1:
The system extracts only the specific DOM elements relevant to the collaborative task rather than transmitting the complete screen capture. This selective extraction inherently filters out sensitive information that is not part of the current interaction context, reducing security risks while maintaining functional completeness.
4Ease of operation
If traditional screen-sharing protocols are used, then established communication methods are employed, but additional installation of web browser plugins or user application extensions is required
Solution Approach 1:
The system leverages the existing DOM structure and standard web technologies that are already present in modern browsers. By using these self-sufficient, built-in resources, the patent eliminates the need for additional plugins or extensions, simplifying the setup process while maintaining full functionality.
Data Source
AI summary
A method includes initiating a collaborative session with a user application. In the collaborative session, a user request, a document object model (DOM) of an active browser page of the user application, and a screen-capture of the active browser page are retrieved, and a sequence of actions is iteratively performed. A context window of a large language model (LLM) with is populated with the user request, the DOM, the screen-capture, a first action of the sequence of actions, and a corresponding action tool. The LLM configures the corresponding action tool using the context window. The active browser page is updated using the configured corresponding action tool. An updated DOM and an updated screen-capture corresponding to the updated active browser page are obtained to continue the iterations.