Image Sensor with Embedded NPU for Real-Time Object Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Real-time image processing in image sensors is challenging due to large data sets and complex operations, leading to high power consumption and inefficiencies, especially in detecting specific objects like pedestrians and vehicles, where traditional methods lack the necessary abstraction and modern deep models like CNNs can improve processing speed and efficiency.
Innovation Solution
Incorporating a Neural Processing Unit (NPU) and SRAM memory within the image sensor chip to facilitate AI-related imaging tasks, enabling rapid object detection with reduced power consumption by performing high-level image processing and outputting semantic information.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional image processing methods are used, then the image sensor can capture images, but real-time processing is difficult to achieve due to large data sets and complex operations
Solution Approach 1:
The patent segments the image processing task by introducing a Neural Processing Unit (NPU) that is specifically dedicated to AI-related imaging tasks such as object detection and recognition. This separates the processing workload from the traditional image signal processor, allowing real-time processing of specific objects by the NPU while the main processor handles other tasks.
Solution Approach 2:
The patent introduces an intermediary NPU between the image sensor and the main processor. This NPU acts as a specialized mediator that performs high-level image processing and outputs semantic information, reducing the computational burden on the main processor and enabling real-time processing.
2Productivity
If modern deep models like CNNs are used for object detection, then processing efficiency improves, but power consumption increases
Solution Approach 1:
The patent applies local quality by creating a specialized NPU with architecture optimized specifically for neural network operations. This local processing unit has dedicated hardware components (MAC units, activation function units, data buffers) that are tailored for CNN operations, improving efficiency while controlling power consumption through specialized design rather than using general-purpose processors.
Solution Approach 2:
The patent changes the processing parameters by implementing an NPU with configurable parameters including number of MAC units, data buffer sizes, and support for different neural network configurations. This allows the system to optimize power consumption based on the specific detection task requirements while maintaining high processing efficiency.
3Adaptability or versatility
If an image signal processor with embedded CPU is used, then image processing functions are provided, but power consumption is excessive due to general purpose processing
Solution Approach 1:
The patent achieves universality by designing an NPU that can handle multiple AI-related imaging tasks including object detection, recognition, and classification. The NPU is configured to execute different neural network models and algorithms, providing versatile processing capability while maintaining low power consumption through specialized hardware design.
Solution Approach 2:
The patent substitutes the mechanical/general-purpose CPU processing system with a specialized NPU hardware system. This replacement uses dedicated neural network processing circuits instead of general-purpose instruction execution, dramatically reducing power consumption while maintaining or improving processing capability for AI tasks.
4Loss of information
If full image data is processed, then comprehensive analysis is achieved, but processing time and power consumption increase
Solution Approach 1:
The patent extracts only the necessary semantic information from the full image data through the NPU. Instead of processing all pixel data through the main processor, the NPU performs rapid object detection and recognition, extracting only the relevant semantic information (object identities, locations, classifications) which is then output to the main processor. This selective extraction maintains information completeness for AI tasks while dramatically reducing processing time.
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
The integration of NPU and SRAM within the image sensor chip allows for faster and more efficient image processing with reduced power consumption, enabling real-time detection of objects such as pedestrians, vehicles, and gestures, while reducing the overall size of the imaging system.
Implementation Method 1
a photodiode, a transfer transistor, a source follower amplifier transistor and a readout circuit are disposed within a semiconductor chip for accumulating an image charge in response to light incident upon the photodiode
Data Source
AI summary
An imaging system has a imaging array on a semiconductor chip which also includes circuit the elements NPU and SRAM to rapidly identify target objects in the imaging data and output their high level representations with low power consumption.