UI Remoting via Video Encoding for Latency Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing user interface remoting technologies face challenges with latency and bandwidth limitations, making it difficult to provide a responsive rich user experience, especially due to the need for transferring large amounts of data across client devices and servers.
Innovation Solution
The User Interface Remoting and Optimization System (UIROS) employs video streaming techniques, specifically using generic video standards like H.264, to efficiently remote user interfaces by exploiting frame coherency, reducing data transfer by only sending updates of moved or new pixel information, and leveraging existing hardware decoders in client devices without the need for specialized drivers or hardware.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional user interface remoting technologies are used to transfer screen output data, then the user interface can be displayed on remote devices, but latency and bandwidth limitations occur due to transferring large amounts of data
Solution Approach 1:
The patent extracts only the changed portions of the frame buffer (dirty regions) instead of transmitting the entire frame buffer. The system identifies regions that have changed since the last transmission and sends only those specific regions to the client device, dramatically reducing data transfer volume and latency.
Solution Approach 2:
The frame buffer is divided into multiple regions or tiles, and only the segments that have changed are identified and transmitted. This segmentation allows the system to process and transmit data in manageable chunks rather than as a single large data stream.
2Reliability
If traditional user interface remoting technologies transfer complete frame buffer data, then all pixel information is transmitted, but bandwidth limitations are encountered due to large data volume
Solution Approach 1:
The system extracts only the essential changed information from the frame buffer and transmits only that data. By using dirty region detection, the patent identifies and extracts only the portions of the screen that have changed, eliminating the need to transmit redundant unchanged data.
Solution Approach 2:
Instead of transmitting complete frame buffer data, the system performs partial action by transmitting only the necessary changed regions. This partial transmission approach maintains image quality while significantly reducing bandwidth consumption.
3Productivity
If video encoding techniques are used to compress user interface data, then data transfer efficiency is improved, but complexity of encoding and decoding processes increases
Solution Approach 1:
The patent uses a simplified copying approach where changed regions are identified and copied directly to the transmission buffer without complex compression algorithms. The system creates a copy of only the changed pixel data and transmits it, leveraging the client device's existing hardware capabilities for efficient rendering.
Solution Approach 2:
The system leverages the client device's own hardware decoding capabilities rather than implementing complex encoding on the server side. The server prepares the data in a format that the client's hardware can efficiently process, shifting the computational burden to the client device.
4Productivity
If specialized drivers or hardware are required for video decoding, then decoding performance is improved, but adaptability to different client devices is reduced
Solution Approach 1:
The patent makes the system universal by designing it to work with any client device that has standard video decoding capabilities. The server prepares data in a standardized format that can be decoded by any device with common video hardware, eliminating the need for specialized drivers or proprietary hardware on the client side.
Solution Approach 2:
The system uses a standardized video encoding format as an intermediary between the server and diverse client devices. This intermediary format acts as a universal language that different devices can decode using their existing hardware capabilities, bridging the gap between different device architectures.
Data Source
AI summary
Methods, techniques, and systems for user interface remoting using video streaming techniques are provided. Example embodiments provide User Interface Remoting and Optimization System (“UIROS”), which enables the efficient remoting of pixel-oriented user interfaces on behalf of their guests using generic video streaming techniques, such as H.264, to send compressed user interface image information in the form of video frame encoded bitstreams. In one embodiment, the UIROS comprises server side support including a UI remoting server, a video encoder, and rendering support and client side support including a UI remoting client, a video decoder, and a display. These components cooperate to implement optimized UI remoting that is bandwidth efficient, low latency and CPU efficient.