Streaming CNN Pipeline for Real-Time Multidimensional Arrays
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional CNNs struggle to process streaming inputs of multidimensional arrays in real-time due to insufficient image sensor pixel throughput, latency, and battery power demands, limiting their application in vision-based tasks like autonomous robotics and augmented reality.
Innovation Solution
A streaming CNN system that computes OCHAN/OCMUX output channel features in parallel, divides rows into NSTRIP vertical strips for parallel evaluation, and tunes the M_CLK frequency to match incoming data rates, enabling efficient pipelined hardware for real-time processing of streaming inputs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional CNNs are used to process streaming inputs, then feature extraction capability is maintained, but real-time processing capability deteriorates due to insufficient throughput and high latency
Solution Approach 1:
The patent divides the input data stream into multiple independent channels, each processed by dedicated hardware pipelines. This segmentation enables parallel processing of different portions of the data stream simultaneously, increasing overall throughput while maintaining low latency for each individual processing path.
Solution Approach 2:
The patent transitions from sequential processing to multi-dimensional parallel processing by implementing multiple independent hardware pipelines that operate simultaneously. This dimensional expansion from single-threaded to multi-threaded architecture enables real-time processing of high-rate data streams while maintaining the feature extraction capabilities of conventional CNNs.
2Productivity
If conventional CNNs process high-rate data streams, then processing completeness is improved, but energy consumption increases due to battery power demands
Solution Approach 1:
The patent replaces general-purpose software-based CNN processing with specialized hardware circuits designed specifically for convolution operations. This hardware substitution eliminates the overhead of software interpretation and memory access, achieving high data processing rates with significantly reduced power consumption suitable for battery-operated devices.
Solution Approach 2:
The patent optimizes hardware parameters such as pipeline depth, parallelism degree, and data precision to achieve the minimum necessary computational resources for real-time processing. By carefully tuning these parameters, the system maintains high processing capability while minimizing energy consumption to meet battery power constraints.
3Measurement precision
If conventional CNNs are used for vision-based tasks, then feature extraction accuracy is maintained, but system complexity increases due to hardware requirements
Solution Approach 1:
The patent decomposes the complex CNN architecture into modular, reusable hardware blocks that can be systematically instantiated. Each convolutional layer is implemented as a standardized pipeline module, reducing design complexity while maintaining feature extraction accuracy through consistent, optimized processing paths.
Data Source
AI summary
A system and method of processing streaming data using convolutional neural networks (CNNs). The method includes receiving, by a CNN, a stream of multidimensional (MD) arrays at a constant data rate. The CNN includes a plurality of interconnected layers of a plurality of convolutional kernels, each interconnected layer of the plurality of interconnected layers is respectively associated with a respective kernel of the plurality of convolutional kernels. The method includes partitioning, by the CNN, a first MD array of the stream of MD arrays into a group of portions. The method includes processing, by the CNN at the data rate, the first MD array to generate a feature map by simultaneously applying a first convolutional kernel of a first layer of the plurality of interconnected layers to each portion of the group of portions to decrease a latency associated with processing the first MD array. The method includes pipelining layers.