Fixed-Point Scale Layer Mapping to GEMM Engine
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional fixed-point scale layer implementations in neural networks face challenges in performance and accuracy, particularly when converting floating-point data, and often require floating-point processing due to difficulties in achieving accurate fixed-point results.
Innovation Solution
The method involves converting a standalone floating-point scale layer into a fixed-point scale layer by obtaining channel-specific quantization parameters based on input data and floating-point parameters γi, βi, and then mapping this fixed-point scale layer to a fixed-point convolution layer for matrix multiplication on a GEMM engine.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If conventional fixed-point scale layer implementation is used, then computation speed may be improved, but accuracy deteriorates due to difficulties in achieving accurate fixed-point results
Solution Approach 1:
The patent changes the parameter representation from conventional fixed-point to a hybrid approach where scale parameters γ and β are maintained as floating-point values while input data is processed in fixed-point format. This parameter change allows the system to achieve both fast fixed-point computation and accurate scaling operations, resolving the contradiction between speed and accuracy.
2Measurement precision
If floating-point processing is used to maintain accuracy, then computation accuracy is improved, but performance deteriorates due to higher computational overhead
Solution Approach 1:
The patent segments the computation into two distinct parts: (1) fixed-point matrix multiplication for the convolution operation which provides high performance, and (2) floating-point operations only for the scaling parameters γ and β which ensure accuracy. This segmentation allows each part to operate in its optimal numeric format, achieving both high performance and accuracy simultaneously.
Solution Approach 2:
The patent merges fixed-point and floating-point operations into a unified computation graph where fixed-point input data flows through convolution layers and is then scaled by floating-point parameters. This hybrid merging allows the system to leverage the speed of fixed-point arithmetic while maintaining accuracy through floating-point scaling, achieving better overall performance than pure floating-point implementation.
3Ease of operation
If standalone floating-point scale layer is used, then implementation simplicity is maintained, but hardware efficiency deteriorates due to inability to utilize GEMM engine
Solution Approach 1:
The patent merges the previously standalone scale layer with the convolution layer by integrating the scaling operation into the convolution computation graph. This merging allows the combined operation to be executed as a single GEMM (General Matrix Multiplication) operation on the hardware accelerator, achieving high hardware efficiency while maintaining implementation simplicity through unified optimization.
Solution Approach 2:
The patent makes the scale layer universal by enabling it to work with both fixed-point and floating-point data types and to be executed on general-purpose GEMM engines rather than requiring specialized scale computation units. This universality allows the same hardware engine to handle both convolution and scaling operations efficiently, improving hardware utilization without complicating the implementation.
Data Source
AI summary
A method to implement a fixed-point scale layer in a neural network for data processing is provided in the present disclosure. The method includes: receiving fixed-point input data over a channel of a standalone floating-point scale layer, and converting the floating-point input data into fixed-point input data of the standalone floating-point scale layer; obtaining fixed-point quantization parameters in each channel based on the input data and floating-point parameters γi, βi in each channel; converting the standalone floating-point scale layer based on the fixed-point quantization parameters into a fixed-point scale layer for processing the fixed-point input data to generate fixed-point output data; and mapping the fixed-point scale layer to a fixed-point convolution layer and the computation of convolution is done by matrix multiplication that can be executed on a GEMM engine.