Compiler Scheduling for Predictable Off-Chip Memory Access
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Machine learning networks face performance bottlenecks due to unpredictable memory access times in parallel processing architectures, particularly in edge devices with limited resources, where contention for off-chip memory access introduces complexity and variability, making it difficult to statically schedule instructions.
Innovation Solution
A compiler generates statically scheduled instructions for a machine learning accelerator (MLA) that eliminates contention for DRAM access by scheduling memory access instructions to ensure data availability before it is required, using techniques like DMA to pre-load data into buffers, allowing predictable execution times.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a network is used between memory and processors to resolve conflicting accesses, then memory access conflicts are resolved, but memory access time becomes slow and unpredictable
Solution Approach 1:
The patent applies preliminary action by pre-loading data from off-chip memory into on-chip buffers before the processing elements need it. The compiler schedules memory access instructions to complete data transfers in advance, ensuring data is available when needed without requiring complex runtime conflict resolution. This eliminates the need for unpredictable network arbitration during critical computation phases.
Solution Approach 2:
The patent segments memory access operations into distinct phases: compilation-time scheduling of memory accesses, pre-loading data into on-chip buffers, and execution-phase computation. By separating memory access scheduling from execution and using dedicated buffer memory, the system avoids the need for complex networks to resolve conflicts during runtime, achieving predictable timing.
2Speed
If every processor is provided with a dedicated port into every memory bank, then memory access speed is improved, but device complexity and cost increase
Solution Approach 1:
The patent introduces on-chip buffer memory as an intermediary between the processing elements and off-chip DRAM. Instead of creating complex direct connections between every processor and every memory bank, the system uses the buffer as a mediator that receives data from DRAM via simple interfaces and provides fast access to processing elements. This dramatically reduces connection complexity while maintaining high access speeds.
Solution Approach 2:
The patent changes the memory access model from a direct processor-memory bank connection to a multi-layer hierarchy: off-chip DRAM → on-chip buffer → processing elements. This dimensional change in the memory architecture allows simple interfaces at each level while achieving high overall performance, avoiding the need for complex cross-connected networks.
3Adaptability or versatility
If memory access utilizes a network with conflict resolution, then multiple processors can access memory, but the access time cannot be predicted with certainty
Solution Approach 1:
The patent achieves predictable parallel memory access by performing all memory access scheduling at compilation time. The compiler analyzes the program and schedules memory access instructions to complete before dependent computation instructions begin, eliminating the need for runtime conflict resolution. This preliminary scheduling guarantees predictable access times while supporting multiple processors accessing memory in parallel.
Solution Approach 2:
The patent creates copies of data in on-chip buffer memory that can be simultaneously accessed by multiple processing elements. Instead of having processors compete for access to off-chip memory through a network, the system loads data into local buffers where multiple processors can access identical or different copies without conflict, achieving both parallelism and predictability.
Data Source
AI summary
A machine learning network is implemented by executing a computer program of instructions on a machine learning accelerator (MLA) comprising a plurality of interconnected storage elements (SEs) and processing elements (PEs). The instructions are partitioned into blocks, which are retrieved from off-chip memory. The block includes a set of deterministic instructions (MLA instructions) to be executed by on-chip storage elements and/or processing elements according to a static schedule from a compiler. The MLA instructions may require data retrieved from off-chip memory by memory access instructions contained in prior blocks. The compiler also schedules the memory access instructions in a manner that avoids contention for access to the off-chip memory. By avoiding contention, the execution time of off-chip memory accesses becomes predictable enough and short enough that the memory access instructions may be scheduled so that they are known to complete before the retrieved data is required.