Video Overlay Encoding With No-Display Frames and Motion Vectors
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing video encoding technologies face challenges in efficiently rendering overlays in augmented reality applications when computational resources are limited, leading to synchronization issues and potential overlay disappearance due to insufficient processing power.
Innovation Solution
The method involves capturing an initial image with an overlay, determining its new position in a subsequent image, encoding the subsequent image as a no-display frame, and generating a third frame with temporally predicted and skip-macroblocks referencing previous frames, reducing the need for real-time re-rendering.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If the overlay is re-rendered every time the camera moves or the object moves, then the overlay position accuracy is maintained, but the processing power requirement increases and synchronization issues occur
Solution Approach 1:
The patent applies preliminary action by pre-calculating motion vectors between frames and preparing no-display frames in advance. Instead of rendering the overlay in real-time for every frame, the system pre-computes the necessary transformation parameters and uses them to efficiently update overlay positions, thereby maintaining accuracy while reducing processing power requirements during actual display.
Solution Approach 2:
The patent uses copying by utilizing no-display frames that contain pre-rendered overlay information. These frames are copied and reused across multiple display frames, allowing the overlay to maintain its position accuracy without requiring complete re-rendering. The motion vectors enable efficient copying of overlay data from one frame to another with minimal computational overhead.
2Reliability
If the overlay is re-rendered frequently to maintain position accuracy, then the overlay remains synchronized with the scene, but the encoding overhead increases and bitrate control becomes more difficult
Solution Approach 1:
The system performs preliminary rendering of overlays in no-display frames before they are needed for display. This allows the overlay to be pre-positioned and pre-rendered with accurate synchronization information, so that when the actual display frame is generated, the overlay can be efficiently inserted without requiring complex real-time rendering, thus maintaining synchronization while improving encoding efficiency.
Solution Approach 2:
The patent introduces no-display frames as intermediary elements between the captured image frames and the final encoded video stream. These intermediary frames carry the pre-computed overlay information and motion vectors, acting as a mediator that decouples the overlay rendering process from the main encoding pipeline. This intermediary structure maintains overlay synchronization while significantly reducing the encoding overhead for display frames.
3Manufacturing precision
If computational resources are allocated to real-time overlay rendering, then overlay position accuracy is maintained, but other processing tasks such as object detection and tracking are compromised
Solution Approach 1:
The patent applies preliminary action by performing overlay rendering in advance during no-display frames, which are processed separately from the main video encoding pipeline. This allows overlay computation to be performed when computational resources are available without interfering with real-time object detection and tracking tasks. The pre-computed overlay information is then efficiently applied to display frames, maintaining position accuracy while preserving processing power for other critical tasks.
Solution Approach 2:
The patent segments the video processing pipeline into separate components: no-display frames for overlay computation, display frames for actual output, and main encoding pipeline for video compression. This segmentation allows overlay rendering to be performed in isolation during no-display frames, preventing it from blocking object detection and tracking operations. The segmented architecture maintains overlay accuracy while ensuring that other processing tasks can utilize available computational resources effectively.
Data Source
AI summary
A method of encoding a video stream including an overlay is provided, including: capturing a first image; adding an overlay to the first image at a first position, and encoding the first image in a first frame of a video stream; capturing a second image of the scene; determining a desired position of the overlay in the second image; encoding the second image in a second frame marked as a no-display frame, and generating and encoding a third frame including temporally predicted macroblocks at the desired position of the overlay referencing the first frame with motion vectors based on a difference between the desired position and the first position, and skip-macroblocks outside of the desired position of the overlay referencing the first frame. A corresponding device, computer program and computer program product are also provided.