Line-wise Feature Map Compression for Real-time Super Resolution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing CNN-based super-resolution methods are difficult to implement in real-time on low-complexity hardware due to high computational complexity and memory requirements, making it challenging to efficiently convert low-resolution images to high-resolution ones, such as 2K FHD to 4K UHD.
Innovation Solution
An image processing device and method utilizing line-wise operations, including a receiver, first and second convolution operators, compressors, and quantizers, which perform 1D and 2D convolution operations, and store parameters efficiently, reducing memory usage and computational load by using depth-wise separable convolutions and residual connections.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If CNN-based super-resolution method is used, then image quality is improved, but computational complexity and memory requirements increase
Solution Approach 1:
The patent divides the image processing into line-wise operations, processing one scan line at a time instead of the entire image. This segmentation approach reduces the memory buffer requirements from needing to store the whole image to only storing current and previous scan lines, while maintaining the CNN-based super-resolution quality through line-by-line convolution operations.
Solution Approach 2:
The patent extracts and processes only the essential features needed for super-resolution in a simplified manner. By using line-wise processing and selective feature extraction through convolution operators, it removes unnecessary computational overhead while preserving the core super-resolution functionality, reducing both computational complexity and memory requirements.
2Manufacturing precision
If CNN-based super-resolution method is used, then image quality is improved, but hardware implementation difficulty increases
Solution Approach 1:
The patent segments the image processing task into line-wise operations that can be implemented with simple hardware buffers and convolution operators. This segmentation enables hardware implementation on low-complexity devices by processing data in small, manageable units (scan lines) rather than requiring the entire image to be loaded and processed at once.
Solution Approach 2:
The patent changes the processing parameters from full-image batch processing to line-wise sequential processing. This parameter change allows the system to operate with reduced memory buffers and simpler hardware architecture, making it feasible to implement CNN-based super-resolution on low-specification hardware while maintaining image quality through the same convolutional operations.
3Manufacturing precision
If frame buffer of great capacity is used, then super-resolution performance is improved, but hardware complexity and cost increase
Solution Approach 1:
The patent segments the feature map into line-wise units and processes them sequentially, requiring only small buffers to store current and previous scan lines instead of a large frame buffer. This segmentation maintains super-resolution performance by preserving necessary spatial context while dramatically reducing the memory capacity requirements.
Solution Approach 2:
The patent transitions from two-dimensional full-image processing to one-dimensional line-wise processing. By changing the dimensionality of processing from handling the entire 2D image at once to processing 1D scan lines sequentially, it reduces the memory requirements from needing to store the full image to only storing a few scan lines, thereby reducing frame buffer capacity needs while maintaining performance.
Data Source
AI summary
Disclosed are an image processing method and device using a line-wise operation. The image processing device, according to one embodiment, comprises: a receiver for receiving an image; a first convolution operator for generating a feature map by performing a convolution operation on the basis of the image; and a compressor for compressing the feature map into units of at least one line; and a decompressor for reconstructing the feature map compressed into units of lines.