On-Sensor DNN for Single Object Detection in Image Sensors
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current object detection techniques, such as face detection, are too memory-intensive and computationally demanding for on-sensor implementation in image sensors, requiring separate processors and significant memory resources, leading to inefficiencies in memory usage and accuracy.
Innovation Solution
An on-sensor implementation of a single object class detection deep neural network (DNN) integrated within a stacked image sensor, which includes a pixel array layer and a logic and DNN layer that performs preprocessing, detection, and postprocessing directly on the sensor, reducing memory footprint and computational complexity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If existing deep neural networks are used for object detection, then detection accuracy is improved, but memory usage and computational complexity increase significantly
Solution Approach 1:
The patent segments the object detection task into two distinct parts: a lightweight on-sensor DNN for rapid single-class detection (e.g., face detection) and a separate off-sensor processor for more complex multi-class detection tasks. This segmentation allows the image sensor to maintain high detection accuracy for common objects while using minimal memory resources, resolving the contradiction between accuracy and memory usage.
Solution Approach 2:
The patent implements partial action by deploying only a single-object-class DNN directly on the image sensor rather than a complete multi-class detection system. This partial implementation provides sufficient detection accuracy for the most common detection needs while dramatically reducing memory footprint and computational complexity, allowing the sensor to handle basic detection independently and transfer only relevant data to external processors.
2Productivity
If existing deep neural networks are implemented on-sensor, then detection speed is improved, but device complexity increases
Solution Approach 1:
The patent divides the detection system into simple on-sensor components and complex off-sensor components. The on-sensor DNN is deliberately kept simple and specialized for single-class detection, maintaining low device complexity while achieving fast detection speeds for common objects. Complex processing tasks are segmented out to external processors, preventing the sensor itself from becoming overly complex.
Solution Approach 2:
The image sensor with the lightweight DNN performs self-service for common detection tasks, independently identifying and flagging objects of interest without requiring constant external processor intervention. This self-service capability increases detection speed for routine tasks while the modular architecture prevents device complexity from escalating, as the sensor handles only what it can efficiently manage autonomously.
3Power
If a separate processor is used for object detection, then computational power is sufficient, but data transfer overhead and latency increase
Solution Approach 1:
The patent applies partial action by implementing just enough computational power directly on the sensor through a lightweight DNN to handle the most common detection scenarios. This eliminates the need to transfer all image data to external processors for every frame, significantly reducing data transfer time and latency. The sensor performs partial detection autonomously, keeping only essential data transfer operations.
Solution Approach 2:
The on-sensor DNN performs preliminary detection and identification of objects of interest before data is transferred to external processors. This preliminary action filters out irrelevant data, so that only detected objects or regions of interest need to be transferred for further processing. This dramatically reduces data transfer time and latency while maintaining sufficient overall computational power through the combination of on-sensor and off-sensor processing.
Data Source
AI summary
An image sensor, electronic device and method thereof that performs on-sensor single object class detection using an on-sensor single object class detection deep neural network (DNN), such as a face detection DNN. The single object class detection DNN includes a pixel array layer configured to capture an image and transfer image data of the captured image, and a logic and single object class detection deep neural network (DNN) layer that receives the image data directly from the pixel array layer and outputs the image data with the single object class detection data to a communication bus of an electronic device.