Inference Engine Inline Post Processing for ML Data Bottlenecks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current computing architectures are not scalable and inefficient for the inference phase of machine learning, particularly due to data movement bottlenecks and high time spent on loading and storing data, which hampers performance and energy efficiency as input data and model sizes grow.
Innovation Solution
A hardware-based programmable architecture with a dedicated inference engine that includes a POD engine for dense matrix computations and a two-dimensional array of processing tiles, capable of performing dense and sparse operations, and utilizing lookup tables for efficient tanh and sigmoid functions, along with piecewise approximations for floating-point operations, to reduce data processing time and improve efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional computing architecture is used for ML inference, then data processing can be performed, but data movement becomes a bottleneck and time spent on loading/storing data increases significantly
Solution Approach 1:
The system segments data processing into compute-intensive operations (performed on processing elements) and memory operations (performed on memory elements). By separating these functions, the architecture reduces unnecessary data movement while maintaining processing capability, directly addressing the time loss problem.
Solution Approach 2:
The patent introduces intermediary memory structures (such as on-chip buffers and caches) between data sources and processing units. These intermediaries reduce the frequency and volume of data transfers to main memory, thereby reducing the bottleneck effect while improving overall processing throughput.
2Adaptability or versatility
If input data and model sizes grow, then ML inference capability increases, but data movement bottleneck worsens and current computing architecture becomes less scalable
Solution Approach 1:
The architecture introduces a hierarchical memory dimension with multiple storage levels (registers, on-chip memory, off-chip memory) organized by access frequency and capacity. This dimensional organization allows efficient handling of growing data sizes by keeping frequently accessed data in faster, smaller memory structures while maintaining access to larger datasets in slower memory.
Solution Approach 2:
The system dynamically adapts its memory hierarchy and data movement patterns based on the size and characteristics of input data and models. Processing elements can dynamically allocate memory resources and adjust data loading strategies, enabling the architecture to scale with growing ML workloads without fixed structural limitations.
3Productivity
If three operations (load, processing, store) are performed for each data element, then data can be processed, but the operations become burdensome as data amount grows
Solution Approach 1:
The patent merges multiple operations into more efficient sequences. By combining data loading with processing operations and using in-place computations where possible, the system reduces the total number of discrete operations needed per data element, making the processing burden manageable even for large datasets.
Data Source
AI summary
A processing unit to support inference acceleration for machine learning (ML) comprises an inline post processing unit configured to accept and maintain one or more lookup tables for performing a tanh and/or sigmoid operation/function. The inline post processing unit is further configured to accept data from a set of registers configured to maintain output from a processing block instead of streaming the data from an on-chip memory (OCM), perform the tanh and/or sigmoid operation on each element of the data from the processing block on a per-element basis via the one or more lookup tables, and stream post processing result of the per-element tanh and/or sigmoid operation back to the OCM after the tanh and/or sigmoid operation is complete.