Sparse Image Processing via Dynamic Sparsity Maps
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
High-resolution image processing on resource-constrained devices like mobile devices is inefficient due to high power consumption and resource usage, particularly since only a small subset of pixel cells and channels are relevant for object detection, leading to unnecessary computation and data transfer.
Innovation Solution
A dynamic sparse image processing system that generates data sparsity maps to selectively fetch and process only relevant data elements and weights, using neural network layers with channel and spatial gating operations to reduce memory and computation demands.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If high-resolution images are generated at high frame rate, then spatial and temporal resolution are improved, but power consumption and computation resources increase significantly
Solution Approach 1:
The patent extracts and processes only the relevant subset of pixel data corresponding to objects of interest, rather than processing all pixel data from high-resolution images. This is achieved through object detection algorithms that identify and isolate regions containing objects, thereby reducing the volume of data requiring processing while maintaining detection accuracy.
Solution Approach 2:
The patent segments the image processing task into separate stages: first detecting objects of interest, then processing only the relevant portions of the image. This segmentation allows the system to maintain high-resolution capabilities for object regions while reducing overall computational load by ignoring background areas.
2Measurement precision
If a large number of pixel cells are used to generate high-resolution images, then spatial resolution is improved, but memory resources and computation requirements increase
Solution Approach 1:
The patent extracts only the necessary pixel data corresponding to objects of interest from the full high-resolution image. By identifying object locations and processing only those regions, the system reduces memory requirements while preserving the spatial resolution needed for accurate object detection and tracking.
3Reliability
If all pixel data is processed for object detection, then detection accuracy is maintained, but computation efficiency deteriorates due to processing unnecessary data
Solution Approach 1:
The patent performs preliminary object detection to identify regions of interest before performing detailed processing. This preliminary action filters out irrelevant data early in the pipeline, ensuring that subsequent computations focus only on necessary pixel data, thereby maintaining detection accuracy while improving computation efficiency.
Solution Approach 2:
The patent extracts and processes only the subset of pixel data corresponding to detected objects, eliminating unnecessary computations on background and irrelevant areas. This extraction approach maintains detection accuracy by focusing computational resources on object regions while significantly improving overall computation efficiency.
Data Source
AI summary
In one example, an apparatus comprises: a memory to store input data and weights, the input data comprising groups of data elements, each group being associated with a channel of channels, the weights comprising weight tensors, each weight tensor being associated with a channel of the channels; a data sparsity map generation circuit configured to generate, based on the input data, a channel sparsity map and a spatial sparsity map, the channel sparsity map indicating channels associated with first weights tensors to be selected, the spatial sparsity map indicating spatial locations of first data elements; a gating circuit configured to: fetch, based on the channel sparsity map and the sparsity map, the first weights tensors and the first data elements from the memory; and a processing circuit configured to perform neural network computations on the first data elements and the first weights tensors to generate a processing result.