OCM Data Streaming for Unaligned Matrix Operations in ML Hardware
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data format conversion for machine learning operations, typically performed by software, is inefficient and resource-intensive, requiring additional memory writes and latency.
Innovation Solution
Data format conversion is performed in hardware components such as DMA and data streaming engines as part of data preparation and transmission to ML hardware, eliminating the need for additional memory writes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If data format conversion is performed by software (driver) before sending data to ML hardware, then data can be converted to the required format, but the process requires additional memory reads, writes, and introduces latency making it inefficient and resource intensive
Solution Approach 1:
The patent replaces the software-based data format conversion mechanism with a hardware-based conversion mechanism. Specifically, a data conversion engine integrated into the ML hardware performs the format conversion directly within the hardware pipeline, eliminating the need for software drivers to mediate between memory and hardware. This substitution of mechanical/software system with hardware system resolves the contradiction by maintaining conversion accuracy while dramatically improving processing efficiency and eliminating additional memory operations.
Solution Approach 2:
The patent implements preliminary data format conversion within the hardware itself before data is processed by the ML compute units. The data conversion engine prepares data in the required format as it is streamed into the hardware, rather than requiring conversion to occur externally before data loading. This preliminary action within the hardware pipeline eliminates the need for subsequent conversion steps and additional memory writes, resolving the efficiency contradiction.
2Adaptability or versatility
If data is read from memory, converted to a different data format, and stored in memory before being sent to ML hardware, then data format conversion is achieved, but the additional memory write operation increases resource consumption and time
Solution Approach 1:
The patent merges the data format conversion operation with the data loading operation into a single integrated hardware process. The data conversion engine is positioned within the hardware pipeline to perform conversion as data is being loaded from memory, combining what were previously separate sequential operations (read from memory, convert format, write to memory) into a unified parallel process. This merging eliminates the need for additional memory writes and reduces overall latency while maintaining data format flexibility.
Solution Approach 2:
The patent introduces a data conversion engine as an intermediary component within the ML hardware that bridges the gap between external data sources and the ML compute units. This intermediary performs format conversion directly within the hardware pipeline, acting as a mediator that eliminates the need for data to be written back to memory. The intermediary approach resolves the contradiction by enabling format flexibility through hardware-based conversion while eliminating the time loss associated with additional memory operations.
Data Source
AI summary
A system comprises an on-chip memory (OCM) configured to maintain blocks of data used for a matrix operation and result of the matrix operation, wherein each of the blocks of data is of a certain size. The system further comprises a first OCM streamer configured to stream a first matrix data from the OCM to a first storage unit, and a second OCM streamer configured to stream a second matrix data from the OCM to a second storage unit, wherein the second matrix data is from an unaligned address of the OCM that is a not a multiple of the certain size. The system further comprises a matrix operation block configured to retrieve the first matrix data and the second matrix data from the first storage unit and the second storage unit, respectively, and perform the matrix operation based on the first matrix data and the second matrix data.