Stacked CMOS Image Sensor With Adaptive On-Sensor DNN Switching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Implementing deep neural networks (DNNs) on image sensors for advanced processing and privacy protection in devices like digital cameras is challenging due to the need for efficient integration and diversification of image processing functions.
Innovation Solution
A solid-state image capturing device with a laminated structure integrating a CMOS image sensor and a DSP, featuring a DNN processing unit, storage, evaluation, and control units, allowing dynamic DNN model changes based on environmental conditions and performance requirements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If deep neural network functions are integrated on the image sensor, then advanced image processing capability and privacy protection are improved, but device complexity and power consumption increase
Solution Approach 1:
The image sensor is divided into multiple functional layers including a first substrate with pixel array and readout circuits, and a second substrate with DNN processing units. This segmentation allows advanced processing capabilities to be added without significantly complicating the overall device structure, as each layer performs specific functions independently.
Solution Approach 2:
The patent transitions from traditional planar integration to three-dimensional stacked architecture. By stacking the first substrate (pixel array) and second substrate (DNN processing) vertically, the system achieves advanced image processing capabilities while maintaining a compact form factor and managing device complexity through spatial separation of functions.
2Adaptability or versatility
If deep neural network functions are integrated on the image sensor, then advanced image processing capability is improved, but power consumption increases
Solution Approach 1:
The system dynamically switches between different operating modes: capturing only raw image data when advanced processing is not needed, and activating DNN processing only when required. This dynamic operation reduces overall power consumption while maintaining advanced image processing capability when needed.
Solution Approach 2:
The DNN processing units continuously receive and process image data from the pixel array without requiring data transfer to external devices. This continuous local processing eliminates transmission overhead and enables real-time advanced image processing while optimizing power usage through efficient on-sensor computation.
3Adaptability or versatility
If multiple DNN models are stored for different environments, then adaptability to different conditions is improved, but storage requirements and device complexity increase
Solution Approach 1:
The DNN processing units are designed with universal architecture capable of executing multiple different DNN models. Instead of dedicating separate hardware to each model, the same processing units can load and execute different models (e.g., first DNN model for indoor environments, second DNN model for outdoor environments) as needed, reducing storage requirements while maintaining adaptability.
4Speed
If DNN processing is performed on the image sensor, then processing speed is improved, but device complexity increases
Solution Approach 1:
The patent merges the image capture function and DNN processing function into a single integrated device. The DNN processing units are directly coupled to the pixel array through the substrate structure, enabling immediate processing of captured images without external device communication. This merging achieves fast processing speed while managing complexity through unified design.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A solid-state image capturing device (100) includes: a DNN processing unit (130) configured to execute a DNN for an input image based on a DNN model; and a DNN control unit (160) configured to receive control information generated based on evaluation information of a result of the execution of the DNN and change the DNN model based on the control information.