Deep Image Decoder Subnetwork Selection for Adaptive Compression
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing end-to-end trainable deep models for image and video compression, while competitive with traditional methods, struggle to adapt to the diverse complexity of specific content due to fixed or non-adaptive subnetworks, leading to suboptimal rate-distortion trade-offs.
Innovation Solution
Adaptively select and fine-tune a subset of subnetworks within a deep decoder based on input data, encoding the selection and weights, and decode using the updated model to improve rate-distortion performance for individual images or videos.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If fixed or non-adaptive subnetworks are used in deep decoder models, then device complexity is reduced and ease of operation is improved, but rate-distortion performance deteriorates due to inability to adapt to diverse content complexity
Solution Approach 1:
The patent implements dynamic subnetwork selection where the decoder adaptively chooses which subnetworks to activate based on the complexity characteristics of the input content. This allows the system to transition from a static, fixed-architecture decoder to a dynamic one that adjusts its computational resources according to the specific encoding task, thereby improving rate-distortion performance without requiring a completely complex fixed architecture for all scenarios.
Solution Approach 2:
The deep decoder is segmented into multiple independent subnetworks, each potentially optimized for different types or complexities of content. This segmentation allows the system to select only the necessary subnetworks for each specific encoding task, improving adaptability while managing overall complexity by not requiring all subnetworks to be active simultaneously.
2Productivity
If all subnetworks are always active in the deep decoder, then processing capability is maximized, but encoding efficiency deteriorates due to unnecessary computation for simple content
Solution Approach 1:
The patent applies partial action by activating only the necessary subset of subnetworks required for the current encoding task rather than running all subnetworks. By evaluating content complexity and selectively engaging only the needed computational resources, the system avoids excessive computation on simple content while maintaining full processing capability when needed, thereby improving encoding efficiency and reducing unnecessary energy consumption.
3Manufacturing precision
If the deep decoder model is updated with fine-tuned weights for each input, then reconstruction quality is improved, but loss of time increases due to fine-tuning computation
Solution Approach 1:
The patent performs preliminary action by pre-training multiple subnetworks offline before actual encoding operations. These pre-trained subnetworks are then selectively activated and optionally fine-tuned for specific content. This preliminary preparation reduces the computational burden during real-time encoding, as the system only needs to perform lightweight selection and minimal fine-tuning rather than extensive training from scratch, thereby improving reconstruction quality while minimizing time loss.
4Adaptability or versatility
If traditional compression methods are used, then reliability and stability are maintained, but adaptability to modern deep learning approaches deteriorates
Solution Approach 1:
The patent creates a universal decoder architecture that can handle both traditional compression requirements and modern deep learning capabilities. By designing a flexible framework that incorporates multiple subnetworks with different functions and characteristics, the system can adapt to various content types and complexity levels while maintaining stable and reliable compression performance. This multi-functional design allows the same system to serve both conventional and advanced compression needs.
Data Source
Figure 1
Figure 2
Figure 3~4
AI summary
A deep neural network-based coding system for images selects a set of subnetworks of a deep neural network model for decoding an input image and fine-tunes the selected subnetworks for a specific image or video. Weights of the selected set of subnetworks are optimized. An embedding corresponding to the input image is determined by a deep neural network model for encoding. The embedding, an information representing the set of selected subnetworks and the optimized weights are encoded and provided to a decoder. The decoder selects a set of subnetworks of a deep neural network model based on the information representing the set of selected subnetworks and replaces weights of the selected set of subnetworks by the optimized weights. An output image is reconstructed using the updated deep neural network model. A variant implementation applies the principle to loop filters of a video coding system. In this variant, the deep neural network decoder model is a loop filter, the input of the updated deep neural network decoder is an image patch and the output computed is a modified image patch.