Sub-frame CNN Scheduler for Low Latency Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional computer vision systems incur high processing latency due to the need to wait for full frame data to process images, which is detrimental to time-sensitive applications, as they do not effectively utilize temporal scheduling for sub-frame processing.
Innovation Solution
Implementing a system that schedules and processes image data on a sub-frame portion basis, utilizing additional communications between a scheduler and processing units to reduce overall latency by processing available sub-frame portions incrementally across layers of a convolutional neural network.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If full frame data is waited for before processing, then processing completeness is ensured, but processing latency increases
Solution Approach 1:
The patent divides the image processing task into multiple independent sub-frame portions (e.g., tiles or regions of interest). Each sub-frame can be processed independently as soon as its data is available, without waiting for the complete frame. This segmentation enables parallel processing across different spatial regions, significantly reducing overall processing latency while maintaining processing quality through selective focus on critical regions.
2Loss of time
If sub-frame portions are processed incrementally, then processing latency is reduced, but system complexity increases
Solution Approach 1:
The patent performs preliminary identification of regions of interest (ROIs) and sub-frame portions before full processing begins. By pre-segmenting the image into manageable sub-frames and pre-determining processing priorities based on ROI detection, the system prepares the processing pipeline in advance. This preliminary action simplifies the real-time scheduling complexity by establishing a clear processing hierarchy before incremental processing commences.
Solution Approach 2:
The patent introduces an intermediary scheduling layer that manages the complex coordination between sub-frame processing, layer scheduling, and data flow management. This intermediary component abstracts the complexity by implementing a directed acyclic graph (DAG) based scheduler that automatically handles dependencies between processing stages, allowing sub-frame incremental processing without requiring complex manual coordination of all processing elements.
3Productivity
If conventional frame-based processing is used, then processing simplicity is maintained, but productivity decreases
Solution Approach 1:
The patent implements a dynamic processing architecture that adapts to varying data availability and processing requirements. The system dynamically adjusts which sub-frames are processed, the order of processing across CNN layers, and resource allocation based on real-time conditions. This dynamic approach enables the system to optimize productivity by processing available sub-frames immediately rather than waiting for complete frames, while the DAG-based scheduler dynamically manages the complexity of coordinated processing.
Data Source
AI summary
Methods and devices are provided for processing image data on a sub-frame portion basis using layers of a convolutional neural network. The processing device comprises memory and a processor. The processor is configured to receive frames of image data comprising sub-frame portions, schedule a first sub-frame portion of a first frame to be processed by a first layer of the convolutional neural network when the first sub-frame portion is available for processing, process the first sub-frame portion by the first layer and continue the processing of the first sub-frame portion by the first layer when it is determined that there is sufficient image data available for the first layer to continue processing of the first sub-frame portion. Processing on a sub-frame portion basis continues for subsequent layers such that processing by a layer can begin as soon as sufficient data is available for the layer.