MAC Operator Convolution Overlap for Neural Network Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current neural network systems face challenges in maintaining high bandwidth while minimizing delay due to the exponential increase in convolution filter size and input data, leading to prolonged computation times and reduced utilization.
Innovation Solution
A memory device and computing device that incorporate a MAC operator capable of performing convolution operations simultaneously with input and weight data, utilizing a buffer memory to read weight data before input data is provided, allowing overlapping data periods for efficient processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If the size of convolution filter and input data increases to meet neural network requirements, then the amount of weight data and input data increases exponentially, but the computation time increases and utilization decreases
Solution Approach 1:
The buffer memory reads weight data from memory cells in advance before input data is provided to the MAC operator. This preliminary action allows weight data to be ready for processing when input data arrives, eliminating waiting time and enabling continuous operation of the MAC operator during overlapping periods.
Solution Approach 2:
The MAC operator performs convolution operations continuously during overlapping periods when input data is being provided. By maintaining continuous useful action without idle time, the system maximizes processing efficiency and reduces overall computation time despite increased data volumes.
2Productivity
If the convolution filter size increases to maintain high bandwidth, then the amount of data increases, but the processing delay increases
Solution Approach 1:
Weight data is read from memory cells to buffer memory in advance before input data arrives. This preliminary preparation ensures that when input data is provided to the MAC operator, the weight data is already available, eliminating access delays and enabling immediate processing.
Solution Approach 2:
The patent introduces a time dimension by implementing overlapping periods where input data provision and convolution operation occur simultaneously. This temporal dimension allows the system to maintain high bandwidth utilization while reducing processing delay through parallel execution of data transfer and computation.
3Adaptability or versatility
If more multiplication and summation operations are performed to process increased input data, then the neural network capability improves, but the computation time increases
Solution Approach 1:
The MAC operator executes convolution operations continuously during overlapping periods without interruption. This continuous useful action ensures that increased computational requirements are met through sustained processing rather than intermittent operation, maintaining neural network capability while minimizing computation time.
Solution Approach 2:
By preparing weight data in the buffer memory in advance, the system eliminates data access delays during the actual computation phase. This preliminary action allows the MAC operator to focus entirely on multiplication and summation operations without interruption for data retrieval, improving computational efficiency.
Data Source
AI summary
A memory device is provided. The memory device includes: a cell region including a first metal pad, a memory cell in the cell region configured to store weight data, a peripheral region including a second metal pad and vertically connected to the memory cell by the first metal pad and the second metal pad, a buffer memory in the peripheral region configured to read the weight data from the memory cell, an input/output pad in the peripheral region configured to receive input data; and a multiply-accumulate (MAC) operator in the peripheral region configured to receive the weight data from the buffer memory and receive the input data from the input/output pad to perform a convolution operation of the weight data and the input data, wherein the input data is provided to the MAC operator during a first period, and wherein the MAC operator performs the convolution operation of the weight data and the input data during a second period overlapping with the first period.