Image Encoding Using Virtual Reference Frames for Boundary Interpolation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing video encoding technologies, such as VVC, face challenges in improving prediction accuracy for inter prediction due to limitations in motion compensation pixel interpolation when scaling windows are applied, especially when reference pictures are magnified or reduced, or when blocks within the screen boundary lack motion information.
Innovation Solution
An image encoding apparatus that generates a prediction image by referencing a previous frame, interpolates pixels outside the frame boundary using a subsequent frame, and adjusts the resolution of the frame through transformation units to enhance interpolation accuracy and efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If motion compensation pixel interpolation is used to generate pixels outside the screen boundary, then prediction accuracy in inter prediction can be improved, but it cannot be applied when a scaling window is set to a reference picture or when blocks inside the screen boundary do not include motion information
Solution Approach 1:
The patent introduces an intermediary process: first generating a virtual reference picture through resolution transformation (scaling window application) to serve as a bridge between the current picture and the original reference picture. This virtual reference picture contains motion information that can be used for motion compensation pixel interpolation even when the original reference picture lacks applicable motion information due to scaling window constraints.
Solution Approach 2:
The patent performs preliminary resolution transformation on the reference picture to generate a virtual reference picture before performing inter prediction. This preliminary action ensures that motion information is available in the virtual reference picture, enabling subsequent motion compensation pixel interpolation to be applied successfully.
2Adaptability or versatility
If a scaling window is set to magnify or reduce a reference picture, then inter prediction can account for magnification or reduction of objects across frames, but motion compensation pixel interpolation cannot be applied
Solution Approach 1:
The patent segments the reference picture processing into two distinct stages: first applying resolution transformation to generate a virtual reference picture, then using this virtual reference picture for motion compensation pixel interpolation. This segmentation allows each stage to optimize for its specific purpose without conflict.
Solution Approach 2:
The virtual reference picture serves as an intermediary between the scaled reference picture and the inter prediction process. It preserves the magnification/reduction effects while providing the necessary motion information for accurate pixel interpolation.
3Ease of manufacture
If simple replication interpolation is used for pixels outside the screen boundary, then the process is simple and efficient, but prediction accuracy is limited
Solution Approach 1:
The patent dynamically selects the interpolation method based on the availability of motion information. When motion information is available in the virtual reference picture, motion compensation pixel interpolation is used for higher accuracy. When motion information is unavailable, simple replication interpolation serves as a fallback, maintaining implementation efficiency.
Data Source
AI summary
An image encoding apparatus comprises predicting unit configured to generate a prediction image for a target block in a first frame to be encoded by referencing a second frame encoded before the first frame; encoding unit configured to encode a prediction error of the target block for the prediction image; interpolating unit, in a case where pixels outside of a boundary of the second frame are referenced, configured to interpolate the pixels outside of the boundary of the second frame using pixels of a third frame encoded before the second frame; and transforming unit configured to change a resolution of a frame before the first frame.