UI Remoting via H.264 Video Encoding

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing user interface remoting technologies face challenges with latency and bandwidth limitations, making it difficult to provide a responsive rich user experience, especially due to limitations in client devices and network connectivity.

Innovation Solution

The User Interface Remoting and Optimization System (UIROS) employs video streaming techniques, such as the H.264 standard, to efficiently remote user interfaces by exploiting frame coherency, reducing the need for exhaustive motion estimation and utilizing generic video decoders, thus minimizing data transfer and leveraging existing hardware accelerators in client devices.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Loss of time

If traditional user interface remoting technologies are used, then user interface can be remotely accessed, but latency and bandwidth limitations make it difficult to provide responsive rich user experience

Engineering Contradiction:

Improveuser interface latencyVSAvoidresponsive user experience quality

Core Design Contradiction:

Loss of timeVSProductivity

Solution Approach 1:

The patent segments the user interface into multiple regions or frames and transmits only the changed portions rather than the entire interface. This is achieved through techniques like region-based encoding, differential encoding, or selective frame transmission, which reduces the amount of data transferred and consequently decreases latency while maintaining responsive user experience.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent implements periodic compression and encoding of user interface data at optimized intervals rather than continuously transmitting all data. By using periodic action with adaptive timing based on network conditions and UI changes, the system reduces bandwidth consumption and latency while maintaining acceptable update rates for responsive interaction.

Inventive Principle:

Principle #19Periodic action

2Speed

If more data is transferred quickly to client device, then user interface updates are more frequent, but bandwidth limitations are encountered

Engineering Contradiction:

Improvedata transfer speedVSAvoiddata transfer volume

Core Design Contradiction:

SpeedVSQuantity of substance

Solution Approach 1:

The patent extracts only the essential changed data from the complete user interface state and transmits merely that portion to the client device. Techniques include transmitting only delta changes between frames, extracting only modified regions, or sending compressed representations of changes, which significantly reduces data volume while maintaining update speed and responsiveness.

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

The patent dynamically adjusts encoding parameters such as compression level, frame rate, and data precision based on network bandwidth conditions and UI change characteristics. By changing these parameters adaptively, the system optimizes the balance between data transfer speed and volume, ensuring fast updates without excessive bandwidth consumption.

Inventive Principle:

Principle #35Parameter changes

3Productivity

If specialized hardware or device drivers are used for optimized remoting, then remoting efficiency improves, but device complexity and cost increase

Engineering Contradiction:

Improveremoting efficiencyVSAvoidhardware and driver requirements

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The patent designs a universal remoting system that can operate on standard client devices with common hardware and software components. By using universally supported compression algorithms, standard network protocols, and generic processing capabilities, the system achieves optimized remoting efficiency without requiring specialized hardware or complex device-specific drivers, thereby reducing device complexity and cost.

Inventive Principle:

Principle #6Universality (Multi-functionality)

Solution Approach 2:

The patent replaces specialized hardware-based remoting mechanisms with software-based compression and encoding algorithms that run on standard processors. By substituting mechanical/specialized hardware solutions with software algorithms, the system achieves similar or better efficiency while eliminating the need for specialized hardware components and complex device drivers.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Data Source

PatentUS9213556B2Application directed user interface remoting using video encoding techniques

Publication Date: 2015.12.15 OMNISSA LLC

AI summary

Methods, techniques, and systems for user interface remoting using video streaming techniques are provided. Example embodiments provide User Interface Remoting and Optimization System (“UIROS”), which enables the efficient remoting of user interfaces on behalf of their guests using generic video streaming techniques, such as H.264, to send compressed user interface image information in the form of video frame encoded bitstreams. With application cooperation, pixels are explicitly cached on the server using lookahead methods to lower latency in remoting the user interface for certain operations. In one embodiment, the UIROS comprises server side support including a UI remoting server, a video encoder, and rendering support and client side support including a UI remoting client, a video decoder, and a display. These components cooperate to implement optimized UI remoting that is bandwidth efficient, low latency and CPU efficient.