Interactive GUI Data Streams With Local Low-Latency Response
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems for remote operation of interactive programs in safety-critical environments, such as aircraft cockpits, suffer from high latency due to user interactions being communicated back and forth between remote systems, leading to poor responsiveness.
Innovation Solution
A system that executes a program on a remote system, generates a data stream with primary and supplemental information for a graphical user interface (GUI), and modifies the GUI response locally based on user interactions without requiring immediate communication with the remote system, using embedded supplemental data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If user interactions are communicated back and forth between remote systems, then security and safety are maintained, but latency increases and responsiveness deteriorates
Solution Approach 1:
The system embeds supplemental information (such as interactive element responses, hints, or pre-computed data) directly into the data stream before the user completes their interaction. This allows the local controller to prepare and display responses in advance, reducing the time needed to process and transmit user actions back to the remote system and then back to the display.
Solution Approach 2:
The patent introduces an intermediary mechanism where supplemental information is embedded in the data stream as a mediator between the remote system and the local display. This intermediary layer allows the system to provide enhanced responsiveness by incorporating pre-processed or anticipatory information without requiring immediate round-trip communication for every user action.
2Extent of automation
If the system waits for remote system response before displaying GUI changes, then centralized control is maintained, but user experience responsiveness deteriorates
Solution Approach 1:
The system segments the control function by allowing the local controller to independently handle certain GUI modification tasks using embedded supplemental information, while the remote system maintains centralized control over the core program execution. This segmentation enables local responsiveness for common interactions while preserving centralized authority for critical operations.
Solution Approach 2:
The system implements partial automation by allowing the local controller to execute certain GUI updates autonomously based on embedded supplemental information, without requiring complete remote system approval for every change. This partial action approach maintains user experience responsiveness while the remote system retains ultimate control through selective overrides.
Data Source
AI summary
Systems and methods are provided that allow for providing interactive functionality to streamed data. The systems include a remote system configured to execute a program, generate a data stream that includes primary information indicative of a visual representation of a graphical user interface (GUI) of the program, and transmit the data stream. The system includes an aircraft that includes a controller configured to receive the data stream from the remote system, display the visual representation of the GUI on a display system based on the primary information, detect an interaction by a user with an interactive element of the GUI represented on the display system, and modify the visual representation of the GUI on the display system in response to detecting the interaction such that a modified visual representation of the GUI is displayed on the display system using supplemental information embedded in the data stream.