Computational Array Formatting to Reduce Memory Read Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional array computational operations for machine learning and artificial intelligence face performance penalties due to latency incurred by reading data from memory, especially in wide convolution operations, limiting throughput in graphical processing unit (GPU) solutions.
Innovation Solution
A microprocessor system with a computational array and hardware data formatter that aligns data to minimize memory reads by loading consecutive elements in parallel, allowing non-consecutive elements across subsets, reducing the need for individual memory address calculations and cache checks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional memory reading methods are used for each data element, then data can be loaded from memory, but latency is incurred and throughput is limited
Solution Approach 1:
The patent divides the computational array into multiple segments or banks that can operate in parallel. Each segment processes a subset of data elements simultaneously, allowing the system to load and process multiple data elements in parallel rather than sequentially, thereby reducing memory access latency and increasing throughput
Solution Approach 2:
The patent implements preliminary action by pre-loading data into the computational array before computation begins. Data is fetched from memory and staged in the computational array in advance, allowing the computational units to immediately begin processing without waiting for individual memory reads during computation, thus reducing latency
2Productivity
If individual memory address calculations are performed for each element, then data can be accessed, but computational overhead and latency increase
Solution Approach 1:
The patent merges the address calculation function into the hardware data formatter, which automatically generates memory addresses for data elements based on their position in the input matrix. This eliminates the need for separate address calculation operations for each data element, reducing computational overhead while maintaining ease of data access
3Productivity
If multiple GPU cores are used for parallel processing, then computation time is reduced, but memory read latency still limits throughput
Solution Approach 1:
The patent transitions from traditional von Neumann architecture to a dimensionally different computational array architecture where computation occurs in parallel across multiple spatial dimensions. The computational array is organized as a two-dimensional grid of processing elements that can simultaneously access and process multiple data elements, effectively adding a spatial dimension to parallel processing that bypasses sequential memory read limitations
Data Source
AI summary
A microprocessor system comprises a computational array and a hardware data formatter. The computational array includes a plurality of computation units that each operates on a corresponding value addressed from memory. The values operated by the computation units are synchronously provided together to the computational array as a group of values to be processed in parallel. The hardware data formatter is configured to gather the group of values, wherein the group of values includes a first subset of values located consecutively in memory and a second subset of values located consecutively in memory. The first subset of values is not required to be located consecutively in the memory from the second subset of values.