Screen Sharing Encoder Object Motion Smoothing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Screen sharing technologies face challenges in maintaining smooth object motion at low picture rates, leading to jerky mouse pointer movements and difficulty in controlling remotely operated pointers, due to the resource-intensive nature of high picture resolution and the complexity of encoding and decoding processes.
Innovation Solution
The method involves signaling object motion parameters and pixel masks from the encoder to the decoder, allowing the reconstruction of intermediate pictures to smooth out object motion, thereby maintaining a low bitrate without compromising video quality.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If the picture rate is lowered to reduce bitrate and complexity, then bandwidth consumption and computational complexity are reduced, but object motion becomes jerky and difficult to follow
Solution Approach 1:
The patent segments the screen content by identifying and tracking specific objects (such as mouse pointers, windows, text) separately from the background. This allows independent processing of moving objects, enabling smooth motion representation for these objects even when the overall picture rate is low. The segmentation is achieved through object detection and tracking algorithms that isolate foreground elements from the static or slowly changing background.
Solution Approach 2:
The patent adds temporal dimension to object representation by generating intermediate frames specifically for moving objects. Instead of relying solely on discrete low-rate keyframes, the system creates additional temporal samples for objects in motion by interpolating their positions and appearances between keyframes. This dimensional enhancement allows smooth object motion to be perceived without increasing the base picture rate.
2Device complexity
If the picture resolution is reduced to lower resource consumption, then processing resources are saved, but the quality of screen sharing deteriorates
Solution Approach 1:
The patent applies local quality enhancement by processing moving objects with higher fidelity than the overall video stream. Specifically, objects such as mouse pointers, window borders, and text are detected and tracked with high precision, and their representations are maintained or enhanced at higher resolutions or with more detailed information than the compressed background. This selective quality allocation ensures that critical interactive elements remain clear and recognizable without requiring the entire video stream to be high resolution.
3Ease of operation
If intermediate pictures are generated to smooth object motion, then object motion smoothness is improved, but encoding and decoding complexity increases
Solution Approach 1:
The patent performs preliminary action by detecting and tracking objects during the encoding phase, and preparing motion information and object parameters in advance. The encoder identifies moving objects, determines their trajectories, and generates motion compensation data that can be used by the decoder to reconstruct smooth object motion. This preliminary processing during encoding reduces the computational burden on the decoder, as the difficult object detection and tracking work has already been performed and the results are packaged in a usable format.
Solution Approach 2:
The patent uses copying by creating simplified representations of moving objects that can be replicated and positioned in intermediate frames. Instead of fully encoding each intermediate frame, the system copies object parameters (position, size, shape, appearance) from reference frames and applies transformations based on motion information. This copying approach significantly reduces decoding complexity compared to full frame decoding, as it only requires parameter manipulation rather than complex pixel-level processing.
Data Source
AI summary
The invention relates to an encoder and a decoder and methods therein for supporting screen sharing, where the screen sharing comprises encoding pictures F representing pixels of a shared screen, providing said pictures to a decoder where they are decoded. The method performed by the encoder comprises deriving information related to an object which is moved on the shared screen between two pictures Fn and Fn+1. At least part of the derived information is related to an intermediate position of the object, on the shared screen, between the two pictures Fn and Fn+1. The method further comprises providing said information to the decoder, thereby enabling the decoder to generate intermediate pictures, FIMD, in addition to the pictures, F, based on the provided information, wherein the object is generated in different positions in said intermediate pictures, FIMD, thus reconstructing the movement of the object between two provided pictures Fn and Fn+1.