Warped Temporal Context for Latent Entropy Coding in INR Video Compression
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing image and video coding schemes face challenges in achieving high compression efficiency due to limitations in leveraging spatial and temporal redundancy, particularly in the context of neural networks, which often result in high computational complexity and inefficient entropy coding of latent features.
Innovation Solution
The use of Feature-based Implicit Neural Representation (INR) with warped temporal context for entropy encoding, where latent values from reference frames are warped using optical flow to provide a more informative context for entropy encoding, reducing the bitstream size by optimizing the distribution of latent features based on spatial and temporal contexts.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If traditional prediction and transform methods are used to leverage spatial and temporal redundancy, then compression efficiency is improved, but computational complexity increases when using neural networks
Solution Approach 1:
The patent replaces traditional mechanical prediction and transform operations with a neural network-based Implicit Neural Representation (INR) system. The INR model learns continuous signal representations that inherently capture spatial and temporal redundancies without requiring explicit prediction blocks or transform operations, thereby reducing computational complexity while maintaining compression efficiency.
Solution Approach 2:
The patent changes the parameterization approach from discrete block-based representations to continuous coordinate-based implicit representations. By representing signals as continuous functions of spatial and temporal coordinates through neural networks, the system achieves better compression efficiency without the computational overhead of traditional transform methods.
2Loss of information
If neural networks are used for signal compression, then compression efficiency is improved, but entropy coding of latent features becomes inefficient
Solution Approach 1:
The patent applies preliminary warping operations to reference frame latent features using optical flow before entropy coding. This preliminary alignment of temporal contexts creates more predictable and structured latent representations, which significantly improves the efficiency of subsequent entropy coding operations by reducing the entropy of the coded data.
Solution Approach 2:
The patent introduces optical flow-based warping as an intermediary step between reference frame extraction and entropy coding. This intermediary operation transforms reference latent features to match current frame contexts, creating a bridge that enables more efficient entropy coding by reducing temporal discrepancies and improving prediction accuracy.
3Loss of information
If temporal context from reference frames is used for entropy coding, then bitstream size is reduced, but accuracy of temporal prediction decreases without proper warping
Solution Approach 1:
The patent applies dynamic warping operations to temporal contexts using optical flow fields that adapt to motion patterns in the video content. This dynamic alignment ensures that temporal predictions remain accurate even during scene changes or camera movements, while still enabling effective entropy coding by reducing temporal redundancies in stationary or smoothly moving regions.
Solution Approach 2:
The patent applies local warping operations that adapt to spatially varying motion patterns. Different regions of the frame are warped according to their local optical flow characteristics, preserving temporal prediction accuracy in high-motion areas while maintaining compression efficiency in low-motion areas. This localized approach ensures both accuracy and bitstream reduction.
Data Source
Figure 1
Figure 2~3
Figure 4
AI summary
A method for encoding an input signal is provided, wherein a current latent representative of features of a current frame of the input signal is obtained, at least one reconstructed part of a reference latent representative of features of a reference frame of the input signal is warped to the current frame, a temporal context is determined for at least of one value of the current latent from the warped at least one reconstructed part of the reference latent, and the at least one value of the current latent is entropy encoded based at least on the temporal context. The input signal can be time-varying signal, such as a video, dynamic 3D object or 3D scene.