Scenario-Based AI Image Processing for ISP Computing Limits
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The limited computing capability of the image signal processor (ISP) in intelligent terminals hinders the effective deployment and implementation of image processing algorithms, particularly when combined with artificial intelligence (AI) algorithms, leading to poor image processing effects due to increased model complexity and memory constraints.
Innovation Solution
Implementing a system with an AI processor that runs multiple dedicated image processing models, each optimized for specific scenarios, using fewer convolutional layers and nodes, and an ISP that adjusts parameters based on scenario information to enhance processing efficiency and effectiveness.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a same network model is used for processing image signals in various scenarios, then the model structure complexity increases, but the image processing effect deteriorates
Solution Approach 1:
The patent divides the single network model into multiple scenario-specific network models (first network model for first scenario, second network model for second scenario, etc.). Each model is trained and optimized for its specific scenario, reducing the complexity required for each individual model while improving overall adaptability and processing effect across different scenarios.
2Adaptability or versatility
If a same network model is used for processing image signals in various scenarios, then the model training process complexity increases, but the deployment difficulty increases
Solution Approach 1:
The training process is segmented into multiple independent training tasks, each focused on a specific scenario and its corresponding network model. This reduces the overall training complexity for each model and simplifies deployment, as each model can be independently trained, validated, and deployed without requiring the entire system to be retrained together.
Solution Approach 2:
The patent uses scenario information (such as motion status, ambient light luminance, focus status) as parameters to dynamically select which network model to deploy. This parameter-based selection approach simplifies deployment by allowing the system to switch between pre-trained models based on current conditions, rather than requiring complex real-time model adaptation.
3Productivity
If a conventional ISP is used in the terminal device, then the computing capability is limited, but the image processing effect deteriorates
Solution Approach 1:
The patent implements a dynamic image processing system that adapts its behavior based on scenario information. The system dynamically selects different network models and adjusts processing parameters according to the current scenario (e.g., motion status, ambient light, focus status), enabling the conventional ISP to achieve improved processing effects without requiring increased computing capability. The dynamics principle allows the system to optimize performance for each specific scenario within the hardware constraints.
Data Source
Figure 1~2
Figure 3
Figure 4
AI summary
Embodiments of this application provide an electronic apparatus and an image processing method of the electronic apparatus. The electronic apparatus includes: an artificial intelligence AI processor, configured to select a first image processing model from a plurality of image processing models based on scenario information, and perform first image signal processing on a first image signal by using the first image processing model, to obtain a second image signal, where the first image signal is obtained based on first image data output by an image sensor, and the scenario information represents feature classification of the first image signal; and an image signal processor ISP, configured to perform second image signal processing on the second image signal to obtain a first image processing result. The electronic apparatus provided in embodiments of this application can improve an image processing effect.