Inter-Coding Transform for Autoencoder Latent Vectors
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Modern video coding systems, such as autoencoders, face significant processing and memory overhead when encoding residual frames, particularly due to the need to perform both encoding and decoding operations.
Innovation Solution
The proposed solution involves performing inter-frame coding in the latent domain instead of the pixel domain, where the video encoder converts frames into tensors of latent attributes using a machine learning model, combines these with previous frame attributes, and transmits a latent vector for decoding.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If machine learning models are used to encode video frames, then data compression is improved, but processing overhead and memory consumption increase significantly
Solution Approach 1:
The video encoding process is segmented into two distinct domains: latent domain encoding using machine learning models for compression, and pixel domain operations only when necessary for inter-frame differential coding. This segmentation allows the system to benefit from ML-based compression while minimizing the computational overhead of full pixel-domain processing.
Solution Approach 2:
A latent domain intermediary is introduced between the pixel domain and the machine learning model. Video frames are first transformed into latent representations (vectors of latent attributes) that capture essential visual information in a compressed form. This intermediary enables differential coding operations to be performed efficiently without requiring full pixel-domain processing, thus reducing processing overhead while maintaining compression benefits.
2Quantity of substance
If inter-frame coding is performed in the pixel domain, then compression efficiency is improved, but processing complexity increases due to transforming latent attributes back to pixel domain
Solution Approach 1:
Instead of the conventional approach of transforming latent attributes back to the pixel domain for differential coding, this invention inverts the process by performing differential coding directly in the latent domain. The system computes differences between latent representations of current and reference frames, then transmits only these differential latent attributes. This inversion eliminates the need for computationally intensive pixel-domain transformations while maintaining compression efficiency.
Data Source
AI summary
This disclosure provides methods, devices, and systems for video coding. The present implementations more specifically relate to autoencoders that support infer-frame coding in the latent domain. A video encoder may convert a frame of video from the pixel domain to the latent domain based on a machine learning model. For example, the machine learning model may be trained to transform the video frame into a tensor of latent attributes. In some aspects, the video encoder may combine the resulting tensor with a tensor of latent attributes associated with a previously-encoded video frame and transform the resulting tensor into a vector that includes latent attributes from both the current video frame and the previous video frame based on an inter-coding transform. More specifically, the inter-coding transform may reduce a dimensionality of the combined tensor so that the resulting vector is smaller or more compressible than the original tensor of latent attributes.