ISP-NPU Image Fusion to Reduce Redundant ML Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing applications redundantly perform machine learning inferences on images, leading to inefficiencies and redundant processing, particularly in systems with multiple applications using different ML systems for the same tasks.
Innovation Solution
Integrating a neural processing unit (NPU) with an image signal processor (ISP) to generate a combined image and ML inference, which is communicated to other components using image steganography, allowing the ISP inference to be shared across applications without requiring each to perform the same inference independently.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If each application performs its own ML inference independently, then each application can process images with its own ML system, but redundant processing occurs and computational efficiency deteriorates
Solution Approach 1:
The patent merges the ML inference function into the ISP by integrating an NPU within the ISP architecture. The ISP performs the inference and embeds the result into the image data stream, allowing multiple applications to share this single inference operation rather than each performing independent inferences.
Solution Approach 2:
The ISP is enhanced with multi-functionality by incorporating NPU capabilities. This universal processing unit serves both traditional image signal processing functions and ML inference functions, enabling a single component to support multiple applications simultaneously without requiring separate dedicated hardware for each application's inference needs.
2Productivity
If a single NPU is integrated into the ISP, then redundant processing is eliminated and efficiency improves, but device complexity increases
Solution Approach 1:
The NPU is merged with the existing ISP architecture rather than being implemented as a separate standalone component. This integration allows the system to leverage the existing ISP infrastructure and data flows, reducing the need for additional complex interconnections and control mechanisms that would arise from a fully separate NPU implementation.
Solution Approach 2:
The patent uses image data as an intermediary carrier to transmit ML inference results from the NPU to multiple applications. By embedding the inference output within the image data stream using techniques like steganography, the system avoids creating complex dedicated communication channels between the NPU and each application, thereby simplifying the overall system architecture.
3Adaptability or versatility
If ML inference results are embedded in image data using steganography, then communication protocols remain unchanged and compatibility is maintained, but information density increases
Solution Approach 1:
The ML inference result is nested within the image data structure itself, similar to placing a smaller object inside a larger container. The inference output is embedded as hidden information within the pixel data using steganographic techniques, allowing the image to carry both its visual information and the ML inference result simultaneously without requiring separate transmission channels.
Solution Approach 2:
The patent creates a copy of the inference result and embeds it within the image data stream. Rather than transmitting the original inference data through a separate channel, the system embeds a copy within the image pixels themselves, allowing the same information to be delivered to multiple applications through the existing image communication infrastructure.
Data Source
AI summary
The technology described adds a ML inference to the output of an image signal processor (ISP) associated with a camera. The combined image and ML inference may be described herein as an augmented image. Once generated, the augmented image may be communicated to other components of a computing system associated with the camera and/or ISP. The initial inference may be generated by a neural processing unit (NPU) associated with the ISP. The ISP may communicate a generated image to the NPU prior to communicating the image to a computing system. In an aspect, the NPU inference is combined with the image using image steganography. Once communicated from the camera to the computing device, the augmented image may be separated into a base image and inference by a camera driver or other component associated with the image management.