Processing-In-Memory On-the-Fly Type Conversion and Sparsity Support
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Computing devices face challenges in supporting larger and more complex machine-learned models due to memory, bandwidth, and power constraints, which limits the utilization of artificial intelligence.
Innovation Solution
The implementation of on-the-fly type conversion and sparsity support in a processing-in-memory architecture, allowing for efficient data processing by converting data types and bypassing operations involving zero values, thereby conserving memory and power.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If processing-in-memory is used to implement machine-learned models, then computational capability is improved, but memory capacity is consumed
Solution Approach 1:
The patent implements dynamic data type conversion within the memory device, allowing weights to be converted between different numerical data types (e.g., from lower precision to higher precision) on-the-fly during inference operations. This dynamic adaptation enables the system to optimize between memory capacity utilization and computational precision requirements, resolving the contradiction between improving computational capability and consuming memory capacity.
Solution Approach 2:
The system changes the numerical data type parameter of stored weights during operation. By converting weights between different data types (e.g., INT4 to INT8, or compressed formats to standard formats), the system can adapt to different computational requirements without permanently allocating additional memory capacity, thus improving computational capability while managing memory usage.
2Adaptability or versatility
If data type conversion is performed, then compatibility with logic circuits is improved, but processing time is increased
Solution Approach 1:
The patent performs data type conversion of weights in advance, during model loading or initialization phases, so that weights are converted to the appropriate data types before actual inference operations begin. This preliminary action ensures compatibility with logic circuits without adding conversion overhead during time-critical inference operations, thus improving compatibility while minimizing processing time loss.
Solution Approach 2:
The memory device performs data type conversion autonomously using integrated conversion circuits within the memory architecture itself, rather than requiring external processing. This self-service approach to type conversion eliminates the need for additional data movement and external processing steps, reducing the time penalty associated with type conversion while maintaining compatibility with various logic circuit requirements.
3Use of energy by moving object
If sparsity handling is implemented, then power consumption is reduced, but device complexity is increased
Solution Approach 1:
The patent extracts and handles sparse weights separately from dense weights in the processing-in-memory architecture. By identifying zero or near-zero weights and excluding them from certain operations (such as multiplication operations in neural network inference), the system reduces unnecessary computational activity and power consumption. The sparsity handling logic is integrated into the existing memory and processing architecture, minimizing additional device complexity while achieving power savings.
Data Source
AI summary
Techniques and apparatuses are described for overcoming memory, bandwidth, and/or power constraints in a processing-in-memory architecture. Example techniques include on-the-fly type conversion and/or sparsity support. On-the-fly type conversion converts data of a first numerical data type to a second numerical data type that matches an expected numerical data type of a logic circuit of a memory device. With on-the-fly type conversion, the memory device can conserve memory and realize a higher effective internal bandwidth while having a flexible design that can support a variety of different memory architectures and/or different machine-learned models. With sparsity support, the processing-in-memory can avoid performing operations that involve data having values equal to zero to conserve power. Also, sparsity support can increase an effective bandwidth for transferring data and conserve memory. With the described techniques, the memory device can utilize processing-in-memory to perform larger and more complex operations for implementing features associated with artificial intelligence.