Neural Network Command Buffer Management With Pointer-Based Scheduling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional graph streaming processing systems require significant memory for data buffers and command buffers, which are not essential for processing operations, leading to inefficient memory usage and a need for optimized memory management.
Innovation Solution
A graph streaming processing system with a thread scheduler that manages thread execution and data units using completion and free pointers to optimize memory usage, enabling parallel processing across processors.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If conventional graph streaming processing systems use data buffers and command buffers for each processing operation, then thread execution and data management are enabled, but memory requirements increase significantly
Solution Approach 1:
The patent extracts and eliminates the command buffer from the system by implementing a pointer-based command issuance mechanism. Instead of using buffers to store commands, the system uses completion pointers to track executed threads and issues commands directly through pointer manipulation, thereby removing the need for command buffers and reducing memory requirements.
Solution Approach 2:
The patent changes the parameter representation from buffer-based storage to pointer-based tracking. By using completion pointers to indicate thread execution status and data availability, the system transforms the memory management approach from allocating fixed buffer spaces to dynamically tracking data locations, significantly reducing memory footprint.
2Adaptability or versatility
If data buffers are allocated for each processing operation, then input and output data can be stored, but memory space is consumed that is not essential for processing
Solution Approach 1:
The patent makes the data buffer serve multiple functions: it acts as both the input buffer for reading data and the output buffer for writing data, eliminating the need for separate command buffers. The same buffer is reused across different processing stages, enhancing versatility while reducing total memory requirements.
Solution Approach 2:
The patent implements a mechanism where data buffers are discarded after being processed and recovered for reuse. By tracking buffer usage through completion pointers and reusing buffers across multiple operations, the system reduces the total memory space required while maintaining full data processing functionality.
Data Source
AI summary
Disclosed herein is a graph streaming processing system comprising a thread scheduler comprising a first component and a second component. The first component is configured to schedule a first set of threads of a first node to a first processor associated with the first node and initialize status of a completion pointer to an initial value. The completion pointer is associated with a command buffer of the first node. The first component is configured to detect the execution of the first set of threads and generation of a data unit and update the status of the completion pointer to an updated value indicating execution of the first set of threads in response to the generation of the data unit. The second component is configured to schedule a second set of threads of a plurality of second nodes to a second processor based on the status of the completion pointer. The second processor is associated with the plurality of second nodes and the second set of threads of the plurality of second nodes are dependent on execution of the first set of threads.