Deterministic Tensor Streaming Memory Without Cache Bottlenecks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current CPU and GPU architectures are not deterministic and rely on out-of-order execution, leading to inefficiencies in memory subsystems, particularly in large machine learning tasks where bandwidth to main memory becomes a limiting factor, and existing DRAM technologies have limited random transaction rates and high power consumption.
Innovation Solution
A deterministic streaming processor (TSP) architecture with functional slices and a global memory access path that uses high-density memories like DRAM and MRAM directly, eliminating the need for cache hierarchies and enabling high bandwidth and compute density, allowing for efficient data streaming across processing elements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If traditional CPU/GPU architectures use out-of-order execution and cache hierarchies to improve performance, then instruction execution speed is improved, but memory subsystem complexity increases and bandwidth to main memory becomes a limiting factor
Solution Approach 1:
The patent removes the complex cache hierarchy (L1, L2 caches) from the memory subsystem, extracting only the essential memory access functionality. This simplification eliminates the bandwidth bottleneck that occurs when state space is swapped into caches, allowing direct access to high-capacity memory without the intermediary cache layers that limit throughput.
Solution Approach 2:
The memory subsystem is segmented into separate functional components: a simplified memory controller that manages deterministic memory access, and a high-capacity memory layer that provides both storage and bandwidth. This segmentation allows each component to be optimized independently, with the memory controller handling access patterns and the memory layer providing capacity and throughput.
2Quantity of substance
If DRAM memory density is increased to support large state space, then storage capacity is improved, but random transaction rate remains limited and power consumption increases
Solution Approach 1:
The patent changes the memory access parameters from traditional random access patterns to deterministic sequential access patterns. By organizing memory access in a predictable sequence based on the computational workload, the system achieves high bandwidth and capacity utilization while reducing the random transaction rate requirement, thereby lowering power consumption despite increased density.
Solution Approach 2:
The system performs preliminary actions by pre-loading data into the memory subsystem in an organized manner before it is needed for computation. This allows the memory to be accessed in a deterministic sequence, maximizing the efficiency of the increased density and reducing the need for frequent random access operations that consume power.
3Speed
If cache hierarchies are used to support high random access, then access speed is improved, but bandwidth to main memory becomes the limiting factor for large tasks
Solution Approach 1:
The patent extracts and removes the cache hierarchy from the memory subsystem, eliminating the bottleneck where bandwidth to main memory limits productivity. By using a simplified memory architecture with direct access to high-capacity memory, the system achieves both speed and bandwidth utilization without the intermediary cache layers that create bandwidth constraints.
Solution Approach 2:
The patent transitions from a two-level memory hierarchy (cache + main memory) to a different architectural dimension where memory capacity and bandwidth are achieved through a unified memory layer with deterministic access patterns. This dimensional change allows simultaneous optimization of both speed and bandwidth without the trade-off present in traditional hierarchies.
Data Source
AI summary
Embodiments are directed to a deterministic streaming system with one or more deterministic streaming processors each having an array of processing elements and a first deterministic memory coupled to the processing elements. The deterministic streaming system further includes a second deterministic memory with multiple data banks having a global memory address space, and a controller. The controller initiates retrieval of first data from the data banks of the second deterministic memory as a first plurality of streams, each stream of the first plurality of streams streaming toward a respective group of processing elements of the array of processing elements. The controller further initiates writing of second data to the data banks of the second deterministic memory as a second plurality of streams, each stream of the second plurality of streams streaming from the respective group of processing elements toward a respective data bank of the second deterministic memory.