Data Streaming Engine Single Gather Instruction for ML Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The inference phase of machine learning is computationally and data-intensive, with data movement becoming a bottleneck due to the need for multiple operations like loading, processing, and storing data, which is inefficient and not scalable with growing input data and model sizes.
Innovation Solution
A hardware-based programmable architecture with a data streaming engine and an instruction streaming engine that uses a single data gathering instruction to efficiently transfer data from DDR memory to on-chip memory, optimizing data processing by performing operations like data gathering, processing, and storing within the inference engine.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If multiple operations (load, processing, store) are performed for each data, then data processing can be completed, but data movement becomes a bottleneck and processing time increases
Solution Approach 1:
The patent combines multiple data operations (loading, processing, storing) into a single unified operation called 'gather and process'. The gather instruction simultaneously loads data from memory into on-chip buffers and prepares it for processing, eliminating the need for separate load and store operations. This merging of operations directly reduces data movement time and improves processing throughput.
Solution Approach 2:
The gather instruction is designed as a universal operation that performs multiple functions: it loads data from external memory, transfers data to on-chip buffers, and prepares data for processing algorithms. This multi-functional instruction replaces multiple specialized instructions, reducing the overall operation count and improving system efficiency.
2Adaptability or versatility
If traditional three-operation instructions are used for data processing, then data can be loaded and processed, but the architecture becomes complex and less scalable with growing data sizes
Solution Approach 1:
The patent merges multiple instructions into a single gather instruction that handles data loading, buffer transfer, and processing preparation in one operation. This reduction in instruction complexity makes the architecture more scalable, as the simplified instruction set can more efficiently handle growing data sizes without increasing operational burden.
Solution Approach 2:
The system segments data processing into distinct phases: data gathering (loading and buffering), processing execution, and result output. This segmentation allows each phase to be optimized independently and enables better scalability, as the gather phase can handle large data volumes efficiently before processing begins.
3Use of energy by moving object
If data is loaded and stored using traditional instructions, then data movement is performed, but energy consumption increases due to repeated operations
Solution Approach 1:
By combining multiple data movement operations into a single gather instruction, the system reduces the total number of memory transactions and operational cycles. This consolidation directly reduces energy consumption associated with data movement, as fewer operations mean less energy spent on memory access, data transfer, and buffer management.
Data Source
AI summary
A system to support data gathering for a machine learning (ML) operation comprises a memory unit configured to maintain data for the ML operation in a plurality of memory blocks each accessible via a memory address. The system further comprises an inference engine comprising a plurality of processing tiles each comprising one or more of an on-chip memory (OCM) configured to load and maintain data for local access by components in the processing tile. The system also comprises a core configured to program components of the processing tiles of the inference engine according to an instruction set architecture (ISA) and a data streaming engine configured to stream data between the memory unit and the OCMs of the processing tiles of the inference engine wherein data streaming engine is configured to perform a data gathering operation via a single data gathering instruction of the ISA at the same time.