MAC Circuit Array With Controller Gearbox For Neural Network Efficiency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural network implementations, such as convolutional neural networks (CNNs), face bottlenecks between processing units and memory, leading to reduced computational efficiency due to limited data supply, which increases latency as batch size increases and increases the cost of implementing high-speed control logic.
Innovation Solution
The proposed solution involves an array of multiply-and-accumulate (MAC) circuits with a controller that operates at a slower frequency than the MAC circuits, using time-division multiplexing and specialized address and control patterns to maximize MAC cycle utilization, and employing a control gearbox and data gearbox to adapt data and control rates to match the MAC rate, allowing multiple threads to be interleaved and kernels to be accessed efficiently.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If batch processing is used to reduce computational costs, then processing efficiency is improved, but latency increases as batch size increases
Solution Approach 1:
The patent segments the batch processing into multiple independent MAC circuits that can operate in parallel. Each MAC circuit processes a portion of the batch simultaneously, allowing the system to maintain high computational efficiency while reducing the effective batch size per processing unit, thereby lowering latency.
Solution Approach 2:
The patent transitions from sequential batch processing to parallel processing by adding spatial dimension through multiple MAC circuits. This dimensional change allows simultaneous processing of multiple data elements, improving throughput while maintaining low latency for individual operations.
2Speed
If controller frequency is increased to match MAC circuit speed, then data supply speed is improved, but implementation cost increases
Solution Approach 1:
The patent introduces specialized address and control patterns as intermediaries that enable the slower controller to efficiently supply data to faster MAC circuits. These patterns optimize data flow and addressing without requiring the controller to operate at full MAC circuit speed, thus maintaining high data supply efficiency while reducing implementation cost.
Solution Approach 2:
The patent changes the operational parameters by allowing the controller to operate at a lower frequency than the MAC circuits. Through optimized control patterns and address generation, the system achieves high-speed data supply without increasing controller frequency, thereby reducing implementation complexity and cost.
3Adaptability or versatility
If multiple kernels are accessed during consecutive MAC cycles, then computational diversity is improved, but control complexity increases
Solution Approach 1:
The patent employs periodic action by cycling through multiple kernels in a systematic pattern during consecutive MAC cycles. The controller uses periodic address and control patterns to access different kernels in sequence, enabling computational diversity and adaptability while maintaining manageable control logic through regular, predictable access patterns.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A circuit arrangement includes an array (106) of MAC circuits(200-0, 200- 1, 200-2, 200-15, 200-n, 201-0, 201-1, 201-2, 201-15, 201-n, 202-0, 202-1, 202- 2, 202-15, 202-n, 231-0, 231-1, 231-2, 231-15, 231-n), wherein each MAC circuit includes a cache (332) configured for storage of a plurality of kernels. The MAC circuits are configured to receive a first set of data elements of an IFM at a first rate (718). The MAC circuits are configured to perform first MAC operations on the first set of the data elements and a first one of the kernels associated with a first OFM depth index during a first MAC cycle (710), wherein a rate of MAC cycles is faster than the first rate. The MAC circuits are configured to perform second MAC operations on the first set of the data elements and a second one of the kernels associated with a second OFM depth index during a second MAC cycle (710) that consecutively follows the first MAC cycle.