CNN Convolution Partitioning for Memory-Efficient Activations and Kernels
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current graphics processing units (GPUs) face inefficiencies in memory usage and processing capabilities, particularly in handling diverse operations such as linear interpolation, tessellation, and texture mapping, due to fixed function computational units and limited parallel processing techniques.
Innovation Solution
Implementing a graphics processing unit (GPU) with a parallel processing architecture that includes a scheduler to dynamically allocate workloads across processing clusters, utilizing SIMT techniques for efficient thread execution and memory management, and incorporating dedicated circuitry for command processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If fixed function computational units are used in traditional GPUs, then the architecture is simpler and more reliable, but the adaptability to handle complex operations like machine learning and deep neural networks is limited
Solution Approach 1:
The patent implements a unified memory architecture where a single memory space is shared across all processing clusters, allowing the same memory resources to serve multiple functions including texture data, vertex data, and machine learning model parameters. This multi-functional memory system enables the GPU to handle diverse workloads from traditional graphics rendering to complex deep neural network operations without requiring separate dedicated memory for each function.
2Productivity
If traditional memory management is used in GPUs, then the memory structure is simpler, but the memory efficiency and bandwidth utilization are insufficient for complex computations
Solution Approach 1:
The patent merges previously separate memory structures (texture memory, vertex memory, and frame buffer memory) into a single unified memory architecture. This consolidation allows all processing clusters to access the same memory space, eliminating memory bottlenecks and improving bandwidth utilization for complex computations such as deep neural network training and inference, where multiple data types need to be accessed simultaneously.
Solution Approach 2:
The unified memory architecture implements dynamic memory allocation and address translation mechanisms that allow memory resources to be flexibly assigned to different processing clusters based on real-time workload requirements. The memory management system can dynamically adjust memory bandwidth allocation and address mapping to optimize performance for varying computational demands, from graphics rendering to machine learning operations.
3Productivity
If maximum parallel processing is implemented in SIMT architecture, then processing throughput is improved, but the memory bandwidth consumption and memory management complexity increase
Solution Approach 1:
The patent implements a memory management system where each processing cluster autonomously manages its own memory access patterns and cache utilization. The unified memory architecture allows processing clusters to self-organize memory access requests, reducing redundant memory traffic and optimizing bandwidth utilization. This self-managing approach enables high parallel throughput while minimizing unnecessary memory bandwidth consumption through localized memory access optimization.
Data Source
AI summary
Embodiments are generally directed to dynamically dividing activations and kernels for improving memory efficiency. An embodiment of a method in a compute engine performing machine learning comprises: receiving, by a convolutional layer of a convolutional neural network (CNN) implemented on the compute engine, a plurality of activation groups contained in an input data, wherein the convolutional layer includes one or more kernel groups and the one or more kernel groups each include a plurality of kernels; determining a plurality of memory efficiency metrics based on the number of activation groups of the plurality of activation groups and the number of kernels of the plurality of kernels; selecting a first optimal number of activation groups and a second optimal number of kernels that are associated with an optimal memory efficiency metric in the plurality of memory efficiency metrics; and performing a convolutional operation on the input data based on the first optimal number and the second optimal number.