Hierarchical Compute Memory Architecture for AI Workloads
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing compute-in-memory (CiM) architectures face inefficiencies due to the lack of integration with different levels of CiM and compute-outside-of-memory (CoM) technologies, leading to memory bottlenecks and increased power consumption, particularly in machine learning workloads.
Innovation Solution
A hierarchical compute and memory architecture that integrates CiM, compute-near-memory (CnM), and CoM elements, allowing for unified weight storage and computation at leaf node compute units, reducing memory bandwidth issues and enhancing processing efficiency by leveraging both analog and digital technologies.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If compute-in-memory (CiM) architecture is used, then computation speed is improved, but memory bandwidth requirements increase
Solution Approach 1:
The patent divides the memory hierarchy into multiple levels: compute-in-memory (CiM) elements at the lowest level, compute-near-memory (CnM) elements at an intermediate level, and compute-outside-of-memory (CoM) elements at the highest level. This segmentation allows data to be processed at the most appropriate level, reducing the need for high memory bandwidth while maintaining computation speed.
Solution Approach 2:
The patent introduces a hierarchical dimension to the memory architecture, organizing compute and memory resources across multiple levels (CiM, CnM, CoM) rather than a single flat level. This dimensional organization enables efficient data processing at each level, reducing overall memory bandwidth requirements while maintaining high computation speed through localized processing.
2Speed
If compute-in-memory (CiM) architecture is used, then computation speed is improved, but power consumption increases
Solution Approach 1:
The patent segments the compute resources across three hierarchical levels (CiM, CnM, CoM), allowing computations to be distributed according to their complexity and data availability. This segmentation enables energy-efficient processing by performing simple operations at the CiM level where data already resides, reducing the energy cost of data movement while maintaining high computation speed for critical operations.
Solution Approach 2:
The patent applies local quality by optimizing each hierarchical level for specific computation types and data characteristics. CiM elements handle operations requiring highest speed with minimal data movement, CnM elements handle intermediate operations, and CoM elements handle complex computations. This localized optimization reduces overall power consumption while maintaining high computation speed where needed.
3Quantity of substance
If integrated hierarchical architecture (CiM, CnM, CoM) is implemented, then memory bandwidth requirements are reduced, but device complexity increases
Solution Approach 1:
The patent segments the memory and compute resources into three distinct hierarchical levels (CiM, CnM, CoM), each with specific functions. This segmentation reduces memory bandwidth requirements by enabling local processing at each level, while the modular nature of the segmentation makes the complexity manageable through clear functional boundaries and standardized interfaces between levels.
Data Source
AI summary
Systems, apparatuses and methods include technology that executes, with a compute-in-memory (CiM) element, first computations based on first data associated with a workload, and a storage of the first data, executes, with a compute-near memory (CnM) element, second computations based on second data associated with the workload and executes, with a compute-outside-of-memory (CoM) element, third computations based on third data associated with the workload. The technology further receives, with a multiplexer, processed data from a first element of the CiM element, the CnM element and the CoM element, and provides, with the multiplexer, the processed data to a second element of the CiM element, the CnM element and the CoM element.