Color Bus Synchronization for ML Inference Tiles
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The inference phase of machine learning is computationally and data-intensive, leading to bottlenecks in data movement and processing due to current computing architectures that are not scalable, resulting in inefficiencies in energy and capital usage.
Innovation Solution
A hardware-based programmable architecture is introduced, featuring a data streaming engine, an instruction streaming engine, and an inference engine with a two-dimensional array of processing tiles, utilizing on-chip memory and programmable units for dense and sparse operations, along with a color bus for synchronization and data communication, to optimize data processing and reduce bottlenecks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If current computing architecture is used for machine learning inference, then data processing can be performed, but data movement becomes a bottleneck and processing time increases
Solution Approach 1:
The system divides data processing into distinct phases (loading phase and processing phase) and uses separate hardware paths for each. The loading phase handles data transfer from memory to on-chip buffers, while the processing phase executes machine learning operations. This segmentation allows parallel execution of data loading and processing operations, eliminating the bottleneck where data movement slowed down processing speed.
Solution Approach 2:
The system performs data loading into on-chip buffers before the actual processing begins. By pre-loading data into the processing engine's on-chip memory, the system eliminates the need to load data during processing operations, thereby reducing the time lost to data loading and storing operations.
2Reliability
If three operations (load, processing, store) are performed for each data, then complete data processing is achieved, but computational burden increases
Solution Approach 1:
The system extracts and separates the data loading function into a distinct phase that occurs before processing. By using on-chip buffers, the loading operation is removed from the processing loop, allowing the processing engine to focus solely on computation without the overhead of continuous load/store operations during processing.
Solution Approach 2:
The system merges the data loading and processing operations into a time-separated pipeline where loading occurs in one phase and processing occurs in another phase. This merging of operations into coordinated phases reduces the total operational complexity compared to sequential execution of load-process-store for each data element.
3Measurement precision
If data and model sizes grow, then more accurate machine learning can be achieved, but data movement bottleneck worsens
Solution Approach 1:
The system implements nested data handling where smaller on-chip buffers are contained within the larger memory hierarchy. Data is loaded from external memory into on-chip buffers in a nested fashion, allowing efficient handling of large models by processing them in manageable chunks that fit within the nested buffer structure, thereby reducing overall data movement time.
Solution Approach 2:
The system introduces a temporal dimension to data handling by separating loading and processing into distinct time phases. This dimensional change from spatial-only data handling to spatio-temporal handling allows parallel operations and reduces the bottleneck effect of data movement on processing speed.
4Ease of manufacture
If conventional architecture is used, then infrastructure is simple, but energy efficiency and scalability are poor
Solution Approach 1:
The system uses dynamic phase management where the processing engine transitions between loading phase and processing phase based on operational needs. This dynamic behavior allows the system to optimize energy consumption by keeping processing units active only when needed, rather than continuously managing data transfer operations, thereby improving energy efficiency.
Data Source
AI summary
A system to support a machine learning (ML) operation comprises an array-based inference engine comprising a plurality of processing tiles each comprising at least one or more of an on-chip memory (OCM) configured to maintain data for local access by components in the processing tile and one or more processing units configured to perform one or more computation tasks on the data in the OCM by executing a set of task instructions. The system also comprises a data streaming engine configured to stream data between a memory and the OCMs and an instruction streaming engine configured to distribute said set of task instructions to the corresponding processing tiles to control their operations and to synchronize said set of task instructions to be executed by each processing tile, respectively, to wait current certain task at each processing tile to finish before starting a new one.