Rolling Shutter Image Processing for Real-Time Sensor Simulation

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing image processing systems face challenges in reproducing rolling shutter distortion and motion blur in virtual captured images, leading to increased processing loads that hinder real-time operation, especially at high frame rates, which is crucial for simulations like hardware in the loop simulations for automated driving systems.

Innovation Solution

An image processing apparatus that performs addition processing of rolling shutter distortion and motion blur on object unit images, reducing the number of rendering times by incorporating information on rolling shutter time differences and motion information to generate a composite image that accurately reproduces the actual operation of an image sensor.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Measurement precision

If rolling shutter distortion is added to virtual captured images, then the accuracy of simulation is improved, but the processing load increases

Engineering Contradiction:

Improvesimulation accuracyVSAvoidprocessing load

Core Design Contradiction:

Measurement precisionVSDevice complexity

Solution Approach 1:

The image processing is segmented into distinct functional units: a distortion addition processing unit that applies rolling shutter distortion to object unit images, and a composite image generation unit that combines these processed images. This segmentation allows each unit to be optimized independently, reducing overall processing complexity while maintaining simulation accuracy.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

Rolling shutter distortion is added to object unit images before they are combined into the final composite image. This preliminary application of distortion avoids the need for complex post-processing corrections, reducing the overall processing load while ensuring accurate simulation of rolling shutter effects.

Inventive Principle:

Principle #10Preliminary action

2Reliability

If rolling shutter distortion is added to virtual captured images, then the realism of captured image is improved, but the real-time operation cannot be satisfied

Engineering Contradiction:

Improverealism of captured imageVSAvoidframe rate

Core Design Contradiction:

ReliabilityVSProductivity

Solution Approach 1:

The processing is divided into parallel segments where distortion is applied to individual object unit images simultaneously, rather than processing the entire frame sequentially. This parallelization enables real-time operation at high frame rates (30 fps, 60 fps, or higher) while maintaining realistic rolling shutter distortion effects.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent replaces complex mechanical image processing systems with computational methods that can be efficiently executed on modern processors. The distortion addition and composite image generation use mathematical transformations rather than physical image manipulation, enabling real-time processing at high frame rates.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

3Measurement precision

If rolling shutter distortion is added in units of lines, then the accuracy of distortion reproduction is improved, but the number of rendering times increases significantly

Engineering Contradiction:

Improvedistortion reproduction accuracyVSAvoidnumber of rendering times

Core Design Contradiction:

Measurement precisionVSDevice complexity

Solution Approach 1:

Instead of processing each line separately, the patent segments the image into object unit images and applies distortion to each object unit. This approach maintains distortion reproduction accuracy for moving objects while significantly reducing the number of rendering operations compared to line-by-line processing.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

Multiple object unit images with distortion applied are merged into a single composite image in one rendering pass. This combining approach reduces the total number of rendering times while preserving the accuracy of rolling shutter distortion effects across the entire frame.

Inventive Principle:

Principle #5Merging (Combining)

Data Source

PatentUS12538027B2Image processing apparatus, image processing method, and simulation system

Publication Date: 2026.01.27 SONY SEMICON SOLUTIONS CORP

AI summary

Provided is an image processing apparatus including a distortion addition processing unit that inputs object unit images that are images for a plurality of objects constituting one frame image, and performs addition processing of rolling shutter distortion for each of the object unit images on the basis of information on a rolling shutter time difference. The image processing apparatus further includes a composite image generation unit that generates a composite image obtained by combining the object unit images subjected to the addition processing by the distortion addition processing unit into one frame image.