Memory Device Parallel MAC Operation via Weight Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current AI systems face an IO bottleneck and inefficient MAC operation flow due to large amounts of input data and weights, leading to reduced efficiency and increased memory capacity requirements during multiply-and-accumulate operations.
Innovation Solution
A memory device and operation method that employs error-bit-tolerance data encoding, input data duplication, and weight data flattening techniques, along with parallel processing and high bandwidth accumulators to generate partial products and improve MAC operation efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional MAC operations are performed with large amounts of input data and weights, then computation accuracy is maintained, but IO bottleneck worsens and operation efficiency decreases
Solution Approach 1:
The patent segments the weight data into multiple parts (first part and second part) and stores them in different memory locations. This segmentation enables parallel reading of weight parts alongside parallel reading of input data, thereby reducing the IO bottleneck and improving MAC operation efficiency through concurrent data access.
Solution Approach 2:
The patent introduces a new dimension of parallelism by simultaneously reading input data from page buffers and weight data from memory cells in parallel. This dimensional expansion of data access operations eliminates the sequential IO bottleneck inherent in conventional MAC operations, achieving higher throughput without compromising computation accuracy.
2Measurement precision
If multi-bit inputs and multi-bit weights are used for MAC operations, then computation accuracy is improved, but IO bottleneck worsens and efficiency decreases
Solution Approach 1:
The patent segments multi-bit weight data into multiple parts that can be processed in parallel. By dividing the weight data and enabling simultaneous processing of multiple weight parts with input data, the system maintains high computation accuracy through multi-bit operations while improving efficiency through parallel execution, thus resolving the trade-off between precision and productivity.
Solution Approach 2:
The patent performs preliminary encoding of input data and segmentation of weight data before the MAC operation. This preliminary preparation enables the subsequent parallel processing to proceed efficiently, allowing multi-bit computations to be executed concurrently without the IO bottlenecks that would otherwise reduce productivity.
3Speed
If parallel processing is implemented for MAC operations, then operation speed is improved, but memory capacity requirements increase
Solution Approach 1:
The patent segments weight data into multiple parts that are stored in different memory locations, enabling parallel access without requiring all weight data to be simultaneously available in a single large memory buffer. This segmentation strategy allows parallel processing to proceed while utilizing existing memory capacity more efficiently, thus improving operation speed without proportionally increasing memory requirements.
Solution Approach 2:
The patent introduces parallelism in the time dimension by simultaneously reading input data and weight data from different memory hierarchies (page buffers and memory cells). This dimensional approach to parallel processing achieves high operation speed by utilizing available memory bandwidth from multiple sources concurrently, rather than requiring expanded memory capacity in a single dimension.
Data Source
AI summary
A memory device and an operation method thereof are provided. The operation method includes: encoding an input data, sending an encoded input data to at least one page buffer, and reading out the encoded input data in parallel; encoding a first part and a second part of a weight data into an encoded first part and an encoded second part of the weight data, respectively, writing the encoded first part and the encoded second part of the weight data into a plurality of memory cells of the memory device, and reading out the encoded first part and the encoded second part of the weight data in parallel; multiplying the encoded input data with the encoded first part and the encoded second part of the weight data respectively to parallel generate a plurality of partial products; and accumulating the partial products to generate an operation result.