CRNN Image Restoration with Overlapping Frame Inference Synthesis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Convolutional recurrent neural networks (CRNNs) used for image restoration in moving images suffer from artifacts like ringing and spurious resolution when processing frames exceeding the learning sequence length, leading to varying restoration accuracy due to the use of initialized recurrent information.
Innovation Solution
An image processing apparatus that utilizes two separate CRNNs for frame groups with overlapping frames, synthesizes their inference results, and initializes recurrent information differently between these CRNNs to suppress artifacts and stabilize restoration accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Object-affected harmful factors
If recurrent information is initialized to suppress artifacts, then image quality artifacts are reduced, but restoration accuracy varies greatly before and after initialization
Solution Approach 1:
The patent divides the processing of identical frames into two separate inference units (first inference unit and second inference unit), each handling different frame groups with overlapping frames. By segmenting the inference process and synthesizing results from both units, the system reduces artifacts while maintaining stable restoration accuracy, avoiding the sharp variations that occur with single-unit initialization.
Solution Approach 2:
The patent merges the inference results from two different inference units through a synthesizer. The first inference unit processes one frame group while the second inference unit processes another frame group with overlaps, and their results are synthesized to produce the final output. This combining approach suppresses artifacts that would appear in either single-unit processing while maintaining consistent restoration accuracy.
2Device complexity
If a single CRNN processes all frames, then processing is simple, but artifacts like ringing and spurious resolution occur when frames exceed learning sequence length
Solution Approach 1:
The patent segments the frame processing into two separate CRNN inference units, each handling different frame groups. The first inference unit processes a first frame group and the second inference unit processes a second frame group that overlaps with the first. This segmentation allows each unit to work within its optimal range while the synthesis of their results prevents artifacts that would occur in single-unit processing of extended frame sequences.
Solution Approach 2:
The patent introduces dynamic switching between two inference units by synthesizing their results. The system dynamically determines which inference result to use for each frame based on the frame group assignments, allowing flexible processing that adapts to different frame positions while maintaining consistent quality and avoiding artifacts associated with static single-unit processing beyond learning sequence length.
Data Source
AI summary
In the present disclosure, an appropriate inference is made using a recurrent-type neural network in image restoration processing. An input-image-acquisition unit acquires input image data. A learning-parameter-acquisition unit acquires a learning parameter to learn an image restoration network. An initialization determination unit acquires learning-sequence-length-information from the learning parameter and generates initialization information specifying a frame where two degradation restoration units perform initialization. Each degradation restoration unit receives one frame (current frame) of the input image data, sees the initialization information, determines whether to initialize recurrent information in a current frame inference, makes an inference about the current frame using initial image data or an inference result about a preceding frame immediately before the current frame as recurrent information, and outputs the inference result to a synthesis unit and the degradation restoration unit itself. The synthesis unit synthesizes two inference results about the same frame from the two degradation restoration units.