In-Stream Piecewise Perspective Transform for Rolling Shutter Stabilization

Solution Overview

Problem

Existing image stabilization techniques for cameras with rolling shutters often introduce visual distortions and require significant bandwidth and power due to memory transfers between system memory and processing units, especially when applying transformations to correct for camera motion and rolling shutter effects.

Innovation Solution

An image signal processor configured to apply in-stream piecewise perspective transformations to image data, using existing filters and scalers, which reduces bandwidth and power consumption by performing transformations before writing data to system memory, and determines motion using gyroscopic data or optical flow calculations to adjust for camera motion and rolling shutter distortions.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If traditional image stabilization methods are used with CPU or GPU processing, then image transformations can be applied to correct camera motion, but significant bandwidth and power are consumed due to memory transfers between system memory and processing units

Engineering Contradiction:

Improveimage stabilization effectivenessVSAvoidpower consumption

Core Design Contradiction:

ReliabilityVSUse of energy by moving object

Solution Approach 1:

The patent applies image stabilization transformations in-stream during the image capture process itself, before the data needs to be written to system memory. By performing the perspective transformations and corrections during the rolling shutter capture process, the system eliminates the need for subsequent memory transfers to CPU or GPU for processing, thereby reducing power consumption while maintaining stabilization effectiveness

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent introduces an in-stream processing unit that acts as an intermediary between the camera sensor and system memory. This intermediary performs the image transformations and stabilization corrections directly during data flow, preventing the need for costly memory transfers to external processing units like CPU or GPU, thus reducing overall power consumption

Inventive Principle:

Principle #24Intermediary (Mediator)

2Reliability

If traditional image stabilization methods are used with CPU or GPU processing, then image transformations can be applied to correct camera motion, but significant bandwidth is consumed due to memory transfers between system memory and processing units

Engineering Contradiction:

Improveimage stabilization effectivenessVSAvoidbandwidth consumption

Core Design Contradiction:

ReliabilityVSLoss of energy

Solution Approach 1:

The patent applies image stabilization transformations in-stream during the image capture process itself, before the data needs to be written to system memory. By performing the perspective transformations and corrections during the rolling shutter capture process, the system eliminates the need for subsequent memory transfers to CPU or GPU for processing, thereby reducing power consumption while maintaining stabilization effectiveness

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent introduces an in-stream processing unit that acts as an intermediary between the camera sensor and system memory. This intermediary performs the image transformations and stabilization corrections directly during data flow, preventing the need for costly memory transfers to external processing units like CPU or GPU, thus reducing overall power consumption

Inventive Principle:

Principle #24Intermediary (Mediator)

3Measurement precision

If rolling shutter image acquisition is used, then high-resolution images can be captured, but visual distortions and motion artifacts are introduced due to sequential scanning

Engineering Contradiction:

Improveimage resolutionVSAvoidvisual distortions

Core Design Contradiction:

Measurement precisionVSObject-affected harmful factors

Solution Approach 1:

The patent applies image stabilization transformations in-stream during the image capture process itself, before the data needs to be written to system memory. By performing the perspective transformations and corrections during the rolling shutter capture process, the system eliminates the need for subsequent memory transfers to CPU or GPU for processing, thereby reducing power consumption while maintaining stabilization effectiveness

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent introduces an in-stream processing unit that acts as an intermediary between the camera sensor and system memory. This intermediary performs the image transformations and stabilization corrections directly during data flow, preventing the need for costly memory transfers to external processing units like CPU or GPU, thus reducing overall power consumption

Inventive Principle:

Principle #24Intermediary (Mediator)

Data Source

PatentUS9462189B2Piecewise perspective transform engine

Publication Date: 2016.10.04 APPLE INC

AI summary

An image signal processor of a device, apparatus, or computing system that includes a camera capable of capturing image data may apply piecewise perspective transformations to image data received from the camera's image sensor. A scaling unit of an Image Signal Processor (ISP) may perform piecewise perspective transformations on a captured image to correct for rolling shutter artifacts and to provide video image stabilization. Image data may be divided into a series of horizontal slices and perspective transformations may be applied to each slice. The transformations may be based on motion data determined in any of various manners, such as by using gyroscopic data and/or optical-flow calculations. The piecewise perspective transforms may be encoded as Digital Difference Analyzer (DDA) steppers and may be implemented using separable scalar operations. The image signal processor may not write the received image data to system memory until after the transformations have been performed.