Image Sensor Architecture for Efficient AI Tensor Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing image sensors face challenges in achieving optimal processing efficiency when performing Computer Vision (CV) processes for input and output tensors of an artificial intelligence model deployed in the image sensor.
Innovation Solution
The image sensor is configured with a pixel array unit, a frame memory, an image processing unit, and an inference processing unit. The image processing unit performs image processing on data stored in the frame memory, and the inference processing unit uses an artificial intelligence model to perform inference processes on the processed image data as input tensors.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If CV processes are performed by using an ISP (Image Signal Processor), then image processing can be achieved, but optimal processing efficiency is difficult to achieve
Solution Approach 1:
The patent divides the image processing system into distinct functional units: an image processing unit dedicated to CV processes and an inference processing unit dedicated to AI model execution. This segmentation allows each unit to be optimized for its specific function, with the image processing unit handling tensor preparation and the inference processing unit handling neural network computations, thereby achieving optimal processing efficiency without requiring the ISP to handle all processing tasks.
Solution Approach 2:
The patent introduces a dedicated image processing unit as an intermediary component between the pixel array and the inference processing unit. This intermediary unit specifically handles CV processes such as color space conversion, scaling, and tensor formation, preparing data in the appropriate format for the AI model while offloading these preprocessing tasks from the ISP, thus optimizing overall processing efficiency.
2Productivity
If an artificial intelligence model is deployed in the image sensor, then inference processing can be performed, but processing efficiency of input and output tensors needs improvement
Solution Approach 1:
The patent implements preliminary action by having the image processing unit perform all necessary CV processes and tensor preparations before data is passed to the inference processing unit. This includes color space conversion, resolution scaling, and format transformation, ensuring that the AI model receives pre-processed input tensors in the optimal format, thereby reducing inference processing time and improving overall efficiency.
Solution Approach 2:
The patent merges the image processing unit and inference processing unit within the same image sensor device, allowing seamless data transfer between units without external intervention. This integration reduces data transmission overhead and processing delays, enabling efficient handling of input tensors and output tensors for the deployed AI model.
Data Source
AI summary
An image sensor according to the present technology includes a pixel array unit where multiple pixels are two-dimensionally arrayed, a frame memory that stores image data output from the pixel array unit, an image processing unit that performs image processing for the image data stored in the frame memory, and an inference processing unit that performs an inference process using an artificial intelligence model on the basis of, as an input tensor, the image data for which image processing has been performed by the image processing unit.