Graph Streaming Thread Scheduling for Shared-Buffer Neural Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional graph streaming processing systems face challenges in executing workflows that combine neural network operations and general-purpose operations due to memory inefficiencies and the need for optimized memory management, particularly when using neural network accelerators that support only a fixed set of functions.
Innovation Solution
A graph streaming processing system with a first processor array, a second processor, and a thread scheduler that optimizes memory usage by dispatching threads between these processors based on data availability and predefined threshold buffer sizes, allowing parallel execution of operations and efficient data sharing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional graph streaming processing systems use neural network accelerators for all operations, then neural network operations are optimized and fastened, but general-purpose operations cannot be executed due to limited function support
Solution Approach 1:
The processing system is segmented into two distinct processor types: neural network accelerators for specialized neural network operations and general-purpose processors for other operations. This segmentation allows each processor type to be optimized for its specific workload while collectively handling diverse workflow operations.
Solution Approach 2:
The processing system achieves universality by combining specialized neural network accelerators with general-purpose processors, enabling the system to execute both neural network operations and general-purpose operations. The thread scheduler provides multi-functionality by dynamically routing different operation types to appropriate processor types.
2Reliability
If each processing operation uses traditional input data buffers and output data buffers, then data storage requirements are met, but memory space requirement increases significantly
Solution Approach 1:
The patent merges the input data buffer and output data buffer into a single shared buffer that is commonly accessed by multiple processing operations. This buffer sharing approach eliminates the need for separate buffers for each operation, significantly reducing total memory space requirements while maintaining data availability through proper synchronization mechanisms.
3Quantity of substance
If data buffers are optimized to reduce memory space, then memory requirement is reduced, but memory management complexity increases
Solution Approach 1:
A thread scheduler acts as an intermediary to manage the shared buffer and coordinate access between multiple processing operations. The scheduler handles buffer allocation, data transfer timing, and synchronization, abstracting the complexity of memory management from individual operations and providing a unified interface for buffer management.
Data Source
AI summary
Disclosed herein is a graph streaming neural network processing system comprising a first processor array, a second processor, and a thread scheduler. The thread scheduler dispatches a thread of a first node to the first processor array or the second processor, wherein the thread is executed to generate output data comprising a data unit stored in a private data buffer of the second processor. The thread scheduler determines that the data unit is sufficient for executing a thread of a second node. The second node is dependent on the output data generated by execution of a plurality of threads of the first node. Upon determining that the data unit is sufficient, the thread scheduler dispatches the thread of the second node. The thread scheduler determines to dispatch a subsequent thread of the first node for execution when a predefined threshold buffer size is available on the private data buffer.