Reconfigurable Image Processing Pipeline Reducing Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Image processing hardware pipelines face delays in updating parameters due to slow communications buses and complex auto-exposure and auto-white balance corrections, leading to unacceptable frame latency and throughput in applications like driver assistance systems.
Innovation Solution
A system comprising an image signal processor (ISP) and a control processor, with a local memory, that calculates auto-exposure and auto-white balance statistics in parallel with pixel data processing, allowing quick storage and transmission of these statistics to determine image processing parameters, and selects appropriate tone mapping look-up tables based on lighting conditions, thereby reducing latency and improving throughput.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If the image processing hardware pipeline uses traditional read-write operations via memory map registers for parameter updates, then the system can maintain simplicity in architecture, but the parameter update latency becomes too slow (200 ns per 32 bit word) to respond to fast-changing environments
Solution Approach 1:
The patent segments the parameter update process into two independent parallel paths: (1) a fast path for critical parameters using dedicated hardware interfaces and local memory, and (2) a traditional path for non-critical parameters using memory map registers. This segmentation allows critical parameters to be updated rapidly without affecting the overall system architecture, resolving the contradiction between speed and complexity.
Solution Approach 2:
The patent introduces an intermediary local memory structure positioned between the image signal processor and the control processor. This local memory acts as a buffer that receives critical parameters from the control processor and makes them immediately available to the image signal processor, eliminating the need for slow read-write operations via memory map registers and achieving rapid parameter updates.
2Productivity
If the image processing pipeline updates all parameters in memory-to-memory mode towards the end of the frame, then comprehensive parameter updates are achieved, but the effective processing throughput is slowed down and frame latency increases
Solution Approach 1:
The patent implements preliminary action by updating critical parameters during the vertical blanking period at the end of the previous frame, rather than waiting until the end of the current frame. This allows the image signal processor to have critical parameters ready at the start of the next frame, eliminating delays and maintaining high processing throughput while still achieving comprehensive parameter updates.
Solution Approach 2:
The patent introduces dynamic parameter update modes that adapt to different operating conditions. The system can switch between updating only critical parameters during vertical blanking periods and updating all parameters when time permits. This dynamic approach allows the system to maintain high throughput by updating only essential parameters when needed, while still providing the option for comprehensive updates when frame latency is not critical.
3Loss of time
If the image processing pipeline selects only critical parameters for updates during vertical blanking periods, then parameter update time is reduced, but non-critical parameters are not updated and may become outdated
Solution Approach 1:
The patent implements a feedback mechanism where the control processor continuously monitors environmental conditions and determines which parameters are critical for current operating conditions. Based on this feedback, the control processor selectively updates only the necessary parameters during vertical blanking periods, ensuring that the most relevant parameters are always current while minimizing update time. This feedback-driven approach maintains parameter accuracy without requiring updates to all parameters.
Data Source
AI summary
A reconfigurable image processing pipeline includes an image signal processor (ISP), a control processor, and a local memory. ISP processes raw pixel data for a frame based on an image processing parameter and provides lines of processed pixel data to control processor via a first interface. For each region of interest (ROI) in the frame, ISP generates auto-exposure and auto-white balance (2A) statistics based on the lines for the ROI and writes them to the local memory via a second interface. Control processor reads 2A statistics from the local memory, determines the image processing parameter based on them, and provides the image processing parameter to ISP. ISP also generates an integer N bin histogram for control processor, which sums a portion of the N total bins and compares the summed bin count to a lighting transition threshold. The image processing parameter is further based on the comparison.