Spatio-Temporal Neural Video Compression With Cross-Resolution Learning

The patent replaces traditional block-based linear transforms (mechanical/mathematical systems) with fully connected neural networks (machine learning models). This substitution enables nonlinear, dynamic coding that optimizes compression across groups of pictures, achieving higher compression efficiency while balancing encoding complexity through learned transformations rather than fixed mathematical operations.

The patent changes from fixed transform parameters in block-based methods to dynamic model parameters (neural radiance network weights) that are trained to optimize compression. These learned parameters adapt to different video content characteristics, enabling holistic optimization across GoP structures while maintaining manageable encoding complexity through parameter sharing and pre-training.

The patent merges cross-resolution representation learning with sequence-level spatio-temporal neural representation. The hybrid framework integrates a transformer attention network that processes multiple resolutions with a spatio-temporal neural model that captures temporal dynamics across video frames. This combination allows the model to learn from diverse resolution training data while simultaneously preserving and utilizing temporal information for coherent video sequence reconstruction.

The patent adds the temporal dimension to the cross-resolution learning framework. While the transformer attention network handles spatial and resolution variations, the spatio-temporal neural model introduces temporal processing through recurrent connections and temporal attention mechanisms, enabling the system to reconstruct video sequences by leveraging information across time while maintaining high-resolution spatial details.

The patent performs preliminary training of the hybrid neural model offline using large datasets and significant computational resources. Once trained, the model parameters (neural radiance network weights) are fixed and can be efficiently deployed for real-time or near-real-time video compression. This preliminary action separates the heavy training phase from the deployment phase, allowing high compression efficiency during actual use while managing system resource demands through pre-computed optimized parameters.

The patent uses the trained hybrid model to generate compressed representations that can be efficiently transmitted and reconstructed. The model creates compact latent representations that capture essential visual and temporal information, reducing the amount of data that needs to be processed and transmitted, thereby lowering system resource demands during deployment while maintaining high compression efficiency.