ML Accelerator Core Architecture for Variable-Shaped Matrix Parallelism
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current hardware platforms for machine learning, such as GPUs, are inefficient in terms of power consumption, size, and cost, and lack scalability and flexibility in handling variable-shaped matrices for machine learning operations.
Innovation Solution
A scalable machine learning accelerator architecture with a programmable processor, including cores, slices, and sub-compute engines, utilizing a system direct memory access engine for efficient data transfer and parallel computation, supporting variable-shaped matrices and scalable performance from low to high compute needs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If GPUs are used for machine learning operations, then parallel processing capability is improved, but power consumption and device size increase
Solution Approach 1:
The system segments machine learning workloads into distinct operator types (convolution, fully-connected, pooling, normalization) and assigns them to specialized hardware circuits. Each circuit is optimized for specific operator types, allowing parallel processing while reducing overall power consumption compared to general-purpose GPUs.
Solution Approach 2:
The patent implements local quality by creating heterogeneous processing circuits with different architectures optimized for different operator types. For example, convolution circuits use different data flow patterns than fully-connected circuits, matching the specific computational requirements of each operator type to minimize energy consumption.
2Productivity
If GPUs are used for machine learning operations, then parallel processing capability is improved, but device size and cost increase
Solution Approach 1:
The system divides the processing architecture into multiple specialized circuits, each handling specific operator types. This segmentation allows for more efficient space utilization compared to general-purpose GPU architectures, as each circuit is minimized for its specific function rather than being part of a monolithic parallel processing unit.
Solution Approach 2:
The patent implements dynamic workloads balancing that allocates processing tasks to available circuits based on current workload characteristics. This dynamic allocation allows the system to scale processing capacity flexibly without requiring dedicated hardware for every possible operation, reducing overall device size.
3Adaptability or versatility
If conventional hardware platforms are used, then general-purpose computing is achieved, but efficiency for variable-shaped matrices is reduced
Solution Approach 1:
The patent changes key parameters of the hardware architecture to accommodate variable-shaped matrices. The system dynamically adjusts circuit configurations and data flow patterns based on the specific dimensions and types of matrices being processed, enabling efficient handling of different matrix shapes without sacrificing computational efficiency.
Solution Approach 2:
The system employs dynamic workload balancing that adapts to varying matrix shapes and sizes. The controller dynamically routes different operator types to appropriate circuits and adjusts processing parameters in real-time, maintaining high computational efficiency across diverse matrix configurations.
Data Source
AI summary
A machine learning accelerator includes a scalable processor with a plurality of cores that receive data from system memory via a system direct memory access (DMA) engine. Each core may include local memory, a compute sub-system, and one or more slices, each of which includes a descriptor execution engine and one or more compute engines. Each compute engine includes input data memory, one or more sub-compute engines, and partial data memory. The sub-compute engines are separately connected to the input data memory and are configured to independently perform compute operations, such as multiply-accumulate (MAC) operations, on the input data and to provide partial output data to the partial data memory. The cores, slices and sub-compute engines may be configured to operate independently to perform separate tasks in parallel that once completed are combined as part of a large artificial intelligence model.