Remote UI Frame Encoding via Motion Hints and Delta Data

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing remoting methods for thin clients are inefficient in processing rich user interfaces, leading to suboptimal network bandwidth usage and client processing resources, especially when handling complex graphics updates.

Innovation Solution

The technology encodes graphical data for real-time remoting by compositing user interfaces into bitmap data on a frame-by-frame basis, using pixel accurate delta encoding, motion hints, and caching operations to minimize network payload and CPU load, determining the minimum set of commands needed to transform the current frame into a new frame.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If bitmapped graphics for user interface frames are transmitted using basic remoting methods, then the user interface can be rendered at the client, but network bandwidth consumption and client processing resources increase significantly

Engineering Contradiction:

Improveuser interface rendering capabilityVSAvoidnetwork bandwidth consumption

Core Design Contradiction:

ReliabilityVSLoss of energy

Solution Approach 1:

The patent segments the user interface frame into multiple layers (background layer, foreground layer, video layer, audio layer, icon layer, text layer) and transmits only the difference data for each layer rather than the complete frame. This segmentation allows the system to send minimal data updates while maintaining complete rendering capability at the client side.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent extracts only the essential difference information between frames using delta encoding, removing redundant data that would otherwise need to be transmitted. By extracting and transmitting only the minimal necessary data (such as motion vectors and difference pixels), the system maintains rendering reliability while dramatically reducing network bandwidth consumption.

Inventive Principle:

Principle #2Taking out (Extraction)

2Manufacturing precision

If complete frame data is transmitted to update user interface, then rendering accuracy is maintained, but network payload size and transmission cost increase

Engineering Contradiction:

Improveframe rendering accuracyVSAvoidnetwork payload size

Core Design Contradiction:

Manufacturing precisionVSQuantity of substance

Solution Approach 1:

The patent applies different encoding strategies to different layers of the user interface based on their specific characteristics. For example, the background layer uses one encoding approach while the foreground layer uses another, optimizing the balance between rendering accuracy and payload size for each layer's specific requirements rather than using a uniform approach.

Inventive Principle:

Principle #3Local quality

Solution Approach 2:

The patent transmits only the partial difference data necessary to update frames rather than complete frame data. By sending minimal essential information (difference pixels and motion vectors) and relying on client-side reconstruction, the system achieves sufficient rendering accuracy while dramatically reducing payload size.

Inventive Principle:

Principle #16Partial or excessive action

3Adaptability or versatility

If richly rendered user interface with advanced graphics is provided, then user experience is enhanced, but remoting efficiency and network performance deteriorate

Engineering Contradiction:

Improveuser interface richnessVSAvoidremoting efficiency

Core Design Contradiction:

Adaptability or versatilityVSProductivity

Solution Approach 1:

The patent implements a nested layer structure where multiple UI elements (icons, text, video, audio) are nested within the frame hierarchy. Each layer can be independently encoded and transmitted, allowing rich multimedia content to be delivered efficiently through incremental updates rather than requiring transmission of complete high-resolution data for all elements simultaneously.

Inventive Principle:

Principle #7Nested doll (Nesting)

Data Source

PatentUS8127233B2Remote user interface updates using difference and motion encoding

Publication Date: 2012.02.28 MICROSOFT TECHNOLOGY LICENSING LLC

AI summary

Frames of user interface graphical data can be remotely rendered at a client during a remote session with a server by providing graphical data commands to the client. The commands include motion commands derived from objects that change position between a current frame and a new frame and delta commands derived from differences between the frames. The delta commands can be generated from a frame update after applying motion commands or without applying motion commands. A server identifies moving objects having a first position in the current frame and a second position in the new frame, generates motion hints for the moving objects, and reduces the motion hints based on collision detection, motion verification and other factors. Motion commands are generated for the reduced set of motion hints and applied to a copy of the current frame at the server. Differences between the modified current frame and the new frame are then encoded as delta commands. The server then sends the motion commands and delta commands to the client. The client receives and applies the commands to the current frame to render the new frame.