Machine Learning Streaming Engine for Reduced Data Movement
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current computing architectures for machine learning are not scalable, inefficient, and burdened with high overhead due to excessive data movement and processing, particularly in training and inference phases, and lack flexibility to adapt to changing algorithms and environments.
Innovation Solution
A programmable hardware architecture for machine learning that includes a core, memory, data streaming engine, and inference engine, optimized for efficient processing of dense and irregular operations, with reduced overhead and adaptable to ML applications, using a streaming mechanism to minimize data and instruction operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If traditional computing architectures (CPU, GPU, FPGA, ASIC) are used for machine learning, then flexibility or performance is achieved, but hardware overhead and data movement burden increase significantly
Solution Approach 1:
The patent extracts and eliminates unnecessary hardware components and overhead from traditional computing architectures. By removing graphics effects, HPC overhead, and complex data movement mechanisms, the system achieves reduced hardware overhead while maintaining flexibility through a simplified architecture that processes data and instructions in parallel streams.
Solution Approach 2:
The patent creates a universal hardware architecture that can handle multiple types of machine learning algorithms and neural networks through a single streamlined design. The architecture uses unified data and instruction streaming mechanisms that work across different ML workloads, eliminating the need for separate specialized hardware for each algorithm type.
2Productivity
If data movement operations (load, process, store) are performed for each data element, then data processing is completed, but processing time and hardware resources increase linearly with data size
Solution Approach 1:
The patent merges data loading, processing, and storage operations into a single parallel stream processing mechanism. By combining these operations and executing them simultaneously in the data stream, the system eliminates the sequential time penalty of traditional load-process-store cycles, achieving constant processing time regardless of data size.
Solution Approach 2:
The patent implements continuous data stream processing where data elements are processed continuously without interruption. The streaming mechanism maintains continuous useful action by overlapping data loading with processing operations, ensuring that no time is wasted on sequential operations and that hardware resources are utilized continuously.
3Adaptability or versatility
If CPU performs flexible data loading and storing for all software forms, then software compatibility is maintained, but performance efficiency decreases due to extra hardware work
Solution Approach 1:
The patent replaces the mechanical sequential processing mechanism of traditional CPUs with a parallel stream processing system. This substitution eliminates the need for flexible but inefficient load-store operations by using a hardware architecture that natively supports parallel data and instruction streaming, achieving both software compatibility and high performance efficiency.
4Speed
If GPU performs high performance computing with graphics pipeline, then processing speed increases, but additional overhead for graphics effects and HPC reduces efficiency for ML applications
Solution Approach 1:
The patent extracts and removes the graphics effects and HPC overhead components from the GPU architecture. By taking out these unnecessary elements that are not required for machine learning applications, the system retains the high processing speed capability while eliminating the overhead that reduces efficiency for ML-specific workloads.
5Adaptability or versatility
If FPGA supports burning any form of logic into programmable gates, then algorithm implementation flexibility is achieved, but hardware efficiency decreases due to significant overhead
Solution Approach 1:
The patent extracts and eliminates the complex logic burning and reconfiguration mechanisms from FPGA architecture. By removing these overhead components, the system maintains algorithm implementation flexibility through a simplified programming model that uses data and instruction streaming, achieving high hardware efficiency without sacrificing adaptability.
Data Source
AI summary
A programmable hardware system for machine learning (ML) includes a core and a streaming engine. The core receives a plurality of commands and a plurality of data from a host to be analyzed and inferred via machine learning. The core transmits a first subset of commands of the plurality of commands that is performance-critical operations and associated data thereof of the plurality of data for efficient processing thereof. The first subset of commands and the associated data are passed through via a function call. The streaming engine is coupled to the core and receives the first subset of commands and the associated data from the core. The streaming engine streams a second subset of commands of the first subset of commands and its associated data to an inference engine by executing a single instruction.