Rolling Shutter Distortion Correction via Line-Specific Posture Calculation

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Conventional electronic image stabilization methods struggle to accurately correct rolling shutter distortion, especially in images captured by CMOS sensors, due to variations in exposure time across pixel lines, leading to reduced image stabilization accuracy.

Innovation Solution

A signal processing device that calculates posture information for each pixel of an image using a posture calculation unit, associates this information to generate corresponding virtual lines, and performs interpolation to correct distortion, allowing for precise posture information association and correction even with large camera posture changes.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If conventional electronic image stabilization using motion sensor information is used, then image stabilization can be performed, but the position of the entire image is only moved according to the movement amount and it is difficult to deal with a difference in the movement amount depending on the position in an image plane

Engineering Contradiction:

Improveimage stabilization accuracyVSAvoidability to handle position-dependent movement differences

Core Design Contradiction:

ReliabilityVSAdaptability or versatility

Solution Approach 1:

The patent divides the image into multiple regions (center region and peripheral regions) and applies different stabilization processing to each region. The center region uses one type of correction while peripheral regions use another type of correction, allowing the system to handle position-dependent movement differences effectively.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

Different stabilization methods are applied to different parts of the image based on their specific characteristics. The center region, which has different motion characteristics from peripheral regions, receives specialized processing. This local differentiation improves overall stabilization accuracy by addressing the unique movement patterns of each image region.

Inventive Principle:

Principle #3Local quality

2Productivity

If rolling shutter method is used for imaging, then imaging can be performed, but deformation due to the focal plane phenomenon occurs and cannot be properly corrected by conventional image stabilization

Engineering Contradiction:

Improveimaging capabilityVSAvoidimage deformation accuracy

Core Design Contradiction:

ProductivityVSManufacturing precision

Solution Approach 1:

The patent calculates and stores posture information for each line of the input image before performing the actual image stabilization processing. This preliminary calculation of posture information for each line allows the system to properly correct rolling shutter distortion during the subsequent stabilization process, addressing the deformation issue caused by the focal plane phenomenon.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system dynamically adjusts the stabilization processing based on the rolling shutter characteristics of the imaging device. By calculating posture information specific to each line and applying position-dependent correction methods, the system adapts to the time-varying nature of rolling shutter distortion, enabling accurate correction despite the sequential exposure of different image lines.

Inventive Principle:

Principle #15Dynamics

3Device complexity

If posture information of the line as the first virtual line is used for the output image cut out by coordinate conversion, then processing is simplified, but it is not always possible to obtain appropriate posture information for each pixel when camera posture changes are large

Engineering Contradiction:

Improveprocessing complexityVSAvoidposture information accuracy

Core Design Contradiction:

Device complexityVSMeasurement precision

Solution Approach 1:

The patent introduces a second virtual line that corresponds to each pixel in the output image, in addition to the first virtual line. This additional dimensional approach allows the system to obtain appropriate posture information for each pixel even when camera posture changes are large, while maintaining reasonable processing complexity through efficient calculation methods.

Inventive Principle:

Principle #17Another dimension (Dimensionality change)

Data Source

PatentUS11196929B2Signal processing device, imaging device, and signal processing method

Publication Date: 2021.12.07 SONY GROUP CORP

AI summary

Posture information corresponding to a line of an input image is calculated. Then, a first virtual line, which is a line in a line direction of the input image for which the posture information calculated is common, is rotated according to the posture information to obtain a second virtual line on the output image. For each pixel of the output image, a corresponding second virtual line is calculated, and posture information is obtained on the basis of the corresponding second virtual line. Then, for coordinates of each pixel of the output image, the reference coordinates on the input image corresponding to the pixel coordinates is calculated using the associated posture information. By cutting out the output image from the input image on the basis of the reference pixel, highly accurate distortion correction can be realized even in a case where the camera posture differs depending on the line.