Buffer Objects for Data Flow Across Memory Hierarchy
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data flow models for multi-compute unit architectures fail to efficiently account for the memory hierarchy of target hardware, leading to suboptimal runtime performance, especially in machine learning applications that require multi-producer and multi-consumer data flows.
Innovation Solution
A programming model and application programming interfaces (APIs) are introduced to specify and manage multi-dimensional data flows across various levels of the memory hierarchy and external memory, supporting multi-producer and multi-consumer semantics, which configures the data processing array to implement efficient data flows.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If current data flow models are used for multi-compute unit architectures, then computational power and parallelism are provided, but runtime performance is suboptimal due to inadequate accounting for memory hierarchy
Solution Approach 1:
The patent introduces buffer objects as intermediary entities that mediate between compute units and the memory hierarchy. These buffer objects abstract the complexity of multiple memory levels (L1, L2, L3 caches and external memory) by providing a unified interface for data flow specification. The compilation system automatically maps buffer objects to appropriate memory levels, eliminating the need for developers to manually manage memory hierarchy complexity while optimizing data access patterns for runtime performance.
Solution Approach 2:
The patent employs parameter changes by introducing buffer object parameters that define data flow properties such as access patterns, timing, and memory level mappings. The compilation system adjusts these parameters automatically based on the target hardware's memory hierarchy characteristics, transforming the generic data flow model into an optimized implementation that accounts for specific memory levels, cache sizes, and access latencies without requiring changes to the application logic.
2Productivity
If data flow models focus on data production and consumption properties, then computational operations are specified, but data movement efficiency across memory hierarchy is not optimized
Solution Approach 1:
The patent extends the traditional data flow model by adding a memory hierarchy dimension. While conventional data flow models specify data production and consumption between computational nodes, this patent introduces buffer objects that operate in an additional dimension representing memory levels. The compilation system maps buffer objects to specific memory levels (L1, L2, L3, external memory), creating a multi-dimensional data flow specification that simultaneously captures computational logic and memory access patterns, thereby optimizing data movement efficiency across the memory hierarchy.
3Adaptability or versatility
If available data flow models are used, then computational nodes are connected, but multi-producer and multi-consumer semantics are not supported
Solution Approach 1:
The patent introduces buffer objects as universal entities that can serve multiple functions simultaneously. A single buffer object can have multiple producers writing to it and multiple consumers reading from it, enabling multi-producer multi-consumer semantics. The buffer object universally handles data routing, synchronization, and memory level mapping for all connected compute units, eliminating the need for separate mechanisms to handle each producer-consumer relationship and simplifying the overall data flow model while increasing adaptability.
Data Source
AI summary
Implementing data flows of an application across a memory hierarchy of a data processing array includes receiving a data flow graph specifying an application for execution on the data processing array. A plurality of buffer objects corresponding to a plurality of different levels of the memory hierarchy of the data processing array and an external memory are identified. The plurality of buffer objects specify data flows. Buffer object parameters are determined. The buffer object parameters define properties of the data flows. Data that configures the data processing array to implement the data flows among the plurality of different levels of the memory hierarchy and the external memory is generated based on the plurality of buffer objects and the buffer object parameters.