Multi-level latent fusion for screen content image coding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current deep-learning-based image and video compression methods, particularly using Convolutional Neural Networks (CNNs), are not optimal for coding diverse content such as screen content images, as they struggle with maintaining detail and readability due to limited spatial resolution and large receptive fields, leading to subpar performance compared to conventional methods.
Innovation Solution
The implementation of multi-level latent fusion in neural networks, which processes input images using consecutive convolution networks to generate fused latent outputs, up-scales and combines latent features from different layers, and applies arithmetic encoding for efficient coding, allowing for adaptive fusion and coding of latents with varying spatial scales and characteristics.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If conventional CNN-based compression methods are used, then the coding process is simple, but the preservation of details and readability deteriorates due to limited spatial resolution and large receptive fields
Solution Approach 1:
The patent divides the feature extraction process into multiple levels (first level, second level, third level) with different spatial resolutions and receptive fields. Each level processes features at its own scale, allowing the system to capture both fine details (from lower levels with smaller receptive fields) and broader contextual information (from higher levels with larger receptive fields), thereby resolving the contradiction between detail preservation and processing complexity.
Solution Approach 2:
The patent introduces a multi-level hierarchical dimension to the feature processing architecture. By organizing convolutional networks at different levels with progressively varying spatial resolutions and receptive field sizes, the system adds a dimensional structure that enables simultaneous access to multiple feature scales, improving detail preservation without overwhelming complexity.
2Adaptability or versatility
If single-level latent features are used, then the coding process is efficient, but the adaptability to diverse content types deteriorates
Solution Approach 1:
The patent implements dynamic adaptability by allowing the system to selectively process and fuse latent features from multiple levels based on content requirements. The multi-level architecture enables dynamic adjustment of which feature levels are emphasized for different content types (e.g., screen content vs. natural images), providing versatility without requiring a completely different processing pipeline for each content type.
Solution Approach 2:
The patent creates a universal multi-level feature processing framework that can handle diverse content types through a single integrated architecture. The same multi-level convolutional network structure serves multiple functions: extracting fine details, capturing contextual information, and adapting to different content characteristics, thereby achieving versatility without proportionally increasing complexity.
3Manufacturing precision
If multi-level latent fusion is implemented, then the rate-distortion performance is improved, but the processing complexity increases
Solution Approach 1:
The patent merges latent features from multiple levels through a fusion process that combines the strengths of different feature representations. By systematically fusing features from the first level (higher spatial resolution, smaller receptive field) with features from the second and third levels (lower spatial resolution, larger receptive field), the system achieves improved rate-distortion performance while managing processing complexity through structured combination operations.
Data Source
AI summary
Methods, systems, and bitstream syntax are described for the fusion of latent features in multi-level, end-to-end, neural networks used in image and video compression. The fused architecture may be static or dynamic based on image characteristics (e.g., natural images versus screen content images) or other coding parameters, such as bitrate constrains or rate-distortion optimization. A variety of multi-level fusion architectures are discussed.