Lookup Table-Based Inference Acceleration for ML Data Bottlenecks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The inference phase of Machine Learning (ML) is computationally and data-intensive, with data movement becoming a bottleneck as input data and model sizes grow, leading to inefficiencies in current computing architectures that are not scalable for ML applications.
Innovation Solution
A hardware-based programmable architecture is introduced, featuring a dedicated inference engine with a POD engine and processing tiles that perform dense and sparse operations, utilizing lookup tables for efficient computation of non-linear functions like logarithms, reducing the need for data transfer and enhancing processing efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional computing architecture is used for ML inference, then data can be processed using standard load-process-store operations, but data movement becomes a bottleneck and processing efficiency decreases as data size grows
Solution Approach 1:
The computing architecture is segmented into specialized functional units: a POD engine for dense mathematical operations and processing tiles for sparse operations. This segmentation allows each unit to be optimized for its specific function, reducing overall data movement requirements and improving processing efficiency for different types of ML computations.
Solution Approach 2:
Lookup tables are introduced as intermediary structures to accelerate non-linear function computations. Instead of performing complex mathematical operations directly, the system pre-computes and stores results in lookup tables, enabling faster inference by simply retrieving pre-computed values during data processing.
2Productivity
If three operations (load, processing, store) are performed for each data element, then data can be processed using standard memory operations, but computational burden increases significantly as data size grows
Solution Approach 1:
The POD engine merges multiple operations into a single unified processing unit that handles both dense matrix operations and non-linear function computations. This consolidation reduces the number of discrete operations needed per data element while maintaining processing capability, thereby reducing computational burden.
Solution Approach 2:
Lookup tables are pre-computed and stored with common non-linear function values before inference runs. This preliminary action eliminates the need to perform complex mathematical operations during actual inference, reducing the number of operations required per data element and improving processing throughput.
3Adaptability or versatility
If current computing architecture is used, then infrastructure can be deployed with existing hardware, but scalability for growing ML applications is poor due to high data loading and storing overhead
Solution Approach 1:
The system extracts and separates data movement operations from computation operations by introducing dedicated processing tiles and POD engines that can process data in-place or with minimal data movement. This extraction allows the system to scale by adding more processing units without proportionally increasing data loading and storing overhead.
Solution Approach 2:
The architecture introduces a new dimension of organization by arranging processing units in a hierarchical structure with POD engines handling dense operations and processing tiles handling sparse operations. This dimensional organization enables efficient data management and processing at different scales, improving scalability for growing ML applications.
Data Source
AI summary
A processing unit to support inference acceleration for machine learning (ML) comprises an inline post processing unit configured to accept and maintain one or more lookup tables for performing each of one or more non-linear mathematical operations. The inline post processing unit is further configured to accept data from a set of registers maintaining output from a processing block instead of streaming the data from an on-chip memory (OCM), perform the one or more non-linear mathematical operations on elements of the data from the processing block via their corresponding lookup tables, and stream post processing result of the one or more non-linear mathematical operations back to the OCM after the one or more non-linear mathematical operations are complete.