Deep Kernel Fusion for ML Processor Efficiency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional kernel fusion techniques in machine learning are limited by their inability to account for complex dependency patterns and all types of operations, leading to suboptimal hardware resource utilization and performance in neural network workloads.
Innovation Solution
Deep fusion of kernel operations that allows simultaneous execution of both arithmetic and non-arithmetic operations across multiple hardware modules, optimizing dependencies and resource utilization through a computation scheduling technique that identifies fused execution schedules agnostically across different hardware architectures.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional single operation approach is used, then hardware module simplicity is maintained, but hardware resource utilization is low and performance is poor
Solution Approach 1:
The patent merges multiple operations (matrix multiplication, element-wise operations, data transformations, memory loads/stores, networking operations) into a single fused kernel that executes simultaneously on heterogeneous hardware modules. This combining of operations resolves the contradiction by improving productivity through better hardware resource utilization while managing complexity through a unified execution model.
Solution Approach 2:
The fused kernel approach creates a universal execution framework that can handle multiple types of operations across different hardware modules (matrix multiplication units, vector units, transpose units, HBM interfaces, ICI links). This multi-functionality improves performance by keeping all hardware modules busy while the modular design manages the complexity through standardized interfaces.
2Productivity
If conventional kernel fusion is applied, then some memory roundtrips are reduced, but complex dependency patterns cannot be handled and not all operation types are accounted for
Solution Approach 1:
The patent implements dynamic scheduling within the fused kernel that can adapt to complex dependency patterns such as normalization across channels and softmax across tokens. The execution engine dynamically adjusts the timing and ordering of operations based on data dependencies, allowing the system to handle versatile dependency patterns while maintaining execution efficiency through parallel processing of independent operations.
Solution Approach 2:
The fused kernel maintains continuous useful action by keeping all hardware modules busy throughout execution, eliminating idle periods. By fusing memory loads, arithmetic operations, data transformations, and networking operations into a single continuous execution stream, the system achieves both high productivity and adaptability to complex operations without stopping for memory roundtrips or handling diverse dependency patterns.
3Productivity
If operations are executed sequentially, then hardware module design is simplified, but most hardware modules remain idle and performance is low
Solution Approach 1:
The patent segments the fused kernel execution into multiple stages that can be executed in parallel across different hardware modules. The execution schedule is divided into segments where memory loads, arithmetic operations, data transformations, and networking operations occur simultaneously in different segments, maximizing hardware resource utilization while managing complexity through structured segmentation of the execution flow.
Data Source
AI summary
Embodiments of the present disclosure include techniques for machine language processing. In one embodiment, the present disclosure includes configuring functional modules on a machine learning processor to execute a plurality of machine learning (ML) operations during a plurality of time segments. During the time segments, a first portion of the ML operations execute serially and at least one other ML operation executes during at least a majority of the time of each of the time segments. Serial ML operations may be processed simultaneously with the at least one other ML operation.