Asynchronous Stream Transfers for Ordered Sparse Memory Access
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep neural networks (DNNs) face inefficiencies in memory access due to sparse workloads, leading to random memory accesses, inefficient bandwidth utilization, long latencies, and data reuse issues, which hinder effective prefetching and processor performance.
Innovation Solution
A hardware/software interface for asynchronous data movement between off-core and core-local memory, utilizing stream transfers with a stream ordering model that processes stream instructions in-order and updates a sync flag for monotonic incremental progress, supporting indirect scatter/gather memory accesses and circular buffering.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If random memory accesses are used for sparse workloads, then flexibility in handling diverse access patterns is improved, but memory bandwidth utilization deteriorates and prefetching effectiveness is reduced
Solution Approach 1:
The patent segments memory access operations into stream-based ordered sequences, dividing the random access problem into manageable ordered segments that can be efficiently prefetched and processed
Solution Approach 2:
The patent implements preliminary action through ordered stream transfers and prefetching mechanisms, where data is fetched and prepared in advance according to predictable patterns, eliminating wait states and improving bandwidth utilization
2Quantity of substance
If data is fetched from off-chip memory with long latencies, then access to large parameter space is enabled, but processor performance deteriorates due to waiting for data
Solution Approach 1:
The patent applies preliminary action by prefetching data from off-chip memory before it is actually needed by the processor, using ordered stream transfers to fetch data in advance and hide latency
Solution Approach 2:
The patent introduces an intermediary stream transfer mechanism between the processor and off-chip memory, managing data flow asynchronously and allowing the processor to continue working while data is being fetched
3Loss of time
If asynchronous data movement is implemented, then memory access latency is hidden, but complexity of the hardware/software interface increases
Solution Approach 1:
The patent uses an intermediary stream transfer interface that manages asynchronous data movement, abstracting the complexity while enabling latency hiding through ordered processing and synchronization flags
Data Source
AI summary
Generally disclosed herein is a hardware/software interface for asynchronous data movement between an off-core memory and a core-local memory, referred to as “stream transfers”, and a stream ordering model. The stream transfers allow software to more efficiently express common data-movement patterns, specifically ones seen in sparse workloads. Direct stream instructions that belong to a stream are processed in-order. For indirect stream instructions, offset elements in an offset list are processed in order. A sync flag is updated to indicate monotonic incremental progress for the stream.