General Convolution Padding on Systolic Arrays via Scratchpad Views
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural network frameworks have varying default padding configurations, leading to inconsistent system performance across different hardware architectures and frameworks, necessitating a general padding support for convolution computation on systolic arrays to achieve peak performance portability.
Innovation Solution
The system coordinates memory views between scratchpad and main memory, dynamically strides input data, and applies padding values as needed, utilizing faster data transfer to improve performance and reduce memory consumption, thereby supporting convolution computations efficiently across multiple frameworks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If padded input data is transferred from main memory to matrix multiplication unit, then convolution computation can be performed, but data transfer time increases and memory consumption increases
Solution Approach 1:
The patent segments the data transfer process by separating valid data transfer from padding data transfer. The scratchpad memory is divided into regions for valid input data and padding data, allowing the system to transfer only necessary valid data from main memory to scratchpad memory, then efficiently transfer to the matrix multiplication unit without redundant padding data transfer time.
Solution Approach 2:
The system performs preliminary padding operations in the scratchpad memory before data transfer to the matrix multiplication unit. By pre-padding the scratchpad memory with valid data and appropriate padding values, the system eliminates the need to transfer padding data from main memory, reducing data transfer time while maintaining convolution computation capability.
2Adaptability or versatility
If padded input data is stored in main memory, then convolution computation with various frameworks is supported, but memory consumption increases
Solution Approach 1:
The patent applies local quality by implementing padding only where necessary in the scratchpad memory rather than padding the entire input data in main memory. The scratchpad memory is organized with specific regions for valid data and padding, allowing the system to support different framework padding requirements locally without increasing overall main memory consumption.
Solution Approach 2:
The scratchpad memory serves as an intermediary between main memory and the matrix multiplication unit. It coordinates memory views between the two memory spaces, translating between unpadded main memory views and padded scratchpad memory views, thereby supporting multiple framework padding configurations without requiring padded data to be stored in main memory.
3Productivity
If matrix multiplication unit processes padded data, then convolution computation is completed, but unnecessary computations occur on padding data
Solution Approach 1:
The system extracts and separates valid computation data from padding data in the scratchpad memory. By organizing the scratchpad memory to clearly distinguish valid input data regions from padding regions, the matrix multiplication unit can process only the necessary valid data for convolution computation, eliminating unnecessary computations on padding data while maintaining computational throughput.
4Adaptability or versatility
If different padding configurations are used across frameworks, then framework-specific requirements are met, but system performance consistency deteriorates
Solution Approach 1:
The scratchpad memory is designed with universal multi-functionality to support padding configurations from multiple neural network frameworks simultaneously. It implements a coordinated memory view system that can translate between different framework padding conventions, allowing the same hardware architecture to achieve peak performance across TensorFlow, PyTorch, and other frameworks despite their different default padding configurations.
Data Source
AI summary
Methods and systems, including computer programs encoded on a computer storage medium. In one aspect, a method includes the actions of receiving a request to perform convolutional computations for a neural network on a hardware circuit having a matrix computation unit, the request specifying the convolutional computation to be performed on a feature tensor and a filter and padding applied to the feature tensor prior to performing the convolutional computation; and generating instructions that when executed by the hardware circuit cause the hardware circuit to perform operations comprising: transferring feature tensor data from a main memory of the hardware circuit to a scratchpad memory of the hardware circuit; and repeatedly performing the following operations: identifying a current subset of the feature tensor; and determining whether a memory view into the scratchpad memory for the current subset is consistent with a memory view of the current subset in the main memory.