Tensor Processing Elements with Local Mask Memory
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing circuits face bottlenecks in implementing convolution matrices due to data movement inefficiencies between memory and processing circuitry, particularly in convolution operations that require repeated processing of large datasets, leading to suboptimal utilization of processing capabilities.
Innovation Solution
A system with global memory circuitry and processing elements arranged in rows and columns, where processing elements perform tensor operations on input tensors using local memory-stored masks, allowing for high-throughput operations by hiding memory loading latencies and reusing data, and utilizing pipelined and parallel data paths to optimize data transmission and processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If data is frequently updated from global memory to processing elements, then processing accuracy is maintained, but memory throughput bottleneck increases
Solution Approach 1:
The patent applies preliminary action by pre-loading mask data into local memory circuitry before it is needed for processing. The system loads masks in advance during idle periods or before processing begins, so that when processing elements need to access mask data, it is already available in local memory without requiring frequent global memory accesses. This resolves the contradiction by maintaining processing accuracy (having required data available) while reducing memory throughput demands (avoiding frequent global memory reads).
Solution Approach 2:
The patent segments memory into global memory and local memory circuitry, with local memory attached to individual processing elements. This segmentation allows frequently accessed mask data to be stored locally near processing elements, reducing the need for frequent global memory accesses. The segmentation resolves the contradiction by maintaining data availability for accurate processing while reducing the burden on global memory throughput.
2Reliability
If processing elements wait for data from global memory, then data is provided accurately, but processing speed decreases
Solution Approach 1:
The system performs preliminary loading of mask data into local memory before processing operations require it. This ensures that when processing elements need data, it is already available in local memory without requiring waits for global memory transfers. This resolves the contradiction by ensuring data accuracy (data is loaded correctly in advance) while improving processing speed (no waiting during processing).
Solution Approach 2:
The patent enables continuous processing by keeping mask data in local memory, eliminating idle wait states where processing elements would otherwise be stalled awaiting global memory transfers. Processing elements can continuously perform operations using data already present in local memory, maintaining both reliability (correct data) and speed (continuous operation without stalls).
3Productivity
If local memory stores multiple masks, then processing efficiency increases, but local memory complexity increases
Solution Approach 1:
The patent applies universality by designing local memory circuitry that can store multiple different masks and serve multiple processing elements. The local memory is configured to hold various mask datasets that can be selectively accessed by different processing elements based on their current processing needs. This multi-functional local memory resolves the contradiction by improving processing efficiency (multiple masks available locally) while managing complexity through a standardized, reusable memory structure that serves multiple purposes.
4Quantity of substance
If data is reused across multiple processing elements, then memory bandwidth is reduced, but data transmission complexity increases
Solution Approach 1:
The patent merges the data storage function into local memory circuitry attached to each processing element. Instead of repeatedly transmitting data from global memory across the network, each processing element has access to its own local memory containing the necessary mask data. This merging of storage and processing at the local level reduces memory bandwidth requirements while the standardized local memory interface keeps transmission complexity manageable.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A system includes global memory circuitry (102, 104) configured to store input tensors and output tensors. Row data paths (106, 108) are each connected to an output port of the memory circuitry. Column data paths (124, 126) are connected to an input port of the memory circuitry. Processing elements (112, 114, 116, 118) are arranged in rows and columns along the row data paths (106, 108) and column data paths (124, 126), respectively. The processing elements (112, 114, 116, 118) include local memory circuitry (122) configured to store multiple masks and processing circuitry (120). The processing circuitry (120) is configured to receive portions of the input tensors from one of the row data paths (106, 108); receive masks from the local memory circuitry (122); perform multiple tensor operations on a same received portion of an input tensors by applying a different retrieved mask for each tensor operation; and generate, using results of the multiple tensor operations, an output for a corresponding column data path (124, 126).