Machine Learning Parameter Server Using Low-Precision Data Conversion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Machine learning processing faces challenges in reducing bit length for enhanced parallelism and calculation speed under limited hardware resources and power constraints, as decreasing bit length leads to quantization errors and loss of data accuracy.
Innovation Solution
A computing system that includes a parameter server, communication path, and processing element, where the parameter server stores neural network model parameters as 32-bit floating point numbers and converts them into 10-bit or less floating point numbers for transmission, allowing for accurate calculation speed exceeding 16-bit floating point processing by using 10-bit or less data for computation and updating parameters as 32-bit floating point numbers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If the bit length of data is decreased to enhance parallelism and calculation speed, then calculation speed and parallelism are improved, but quantization error increases and data accuracy is lost
Solution Approach 1:
The patent segments the data processing into two distinct parts: computation operations performed on low-precision data (10-bit or less floating point) for high-speed parallel processing, and parameter updates performed on high-precision data (32-bit floating point) for accuracy. This segmentation allows each part to operate at its optimal precision level, resolving the contradiction between speed and accuracy.
Solution Approach 2:
The patent applies different precision requirements to different parts of the machine learning processing pipeline. Computation operations (multiplication and addition) use low-precision data for speed, while parameter updates use high-precision data for accuracy. This local differentiation of quality requirements optimizes both speed and accuracy in their respective domains.
2Productivity
If 10-bit or less floating point data is used for computation, then calculation speed exceeds 16-bit floating point processing, but quantization error affects neural network training
Solution Approach 1:
The patent divides the training process into computation phase (using 10-bit or less floating point for speed) and parameter update phase (using 32-bit floating point for reliability). This segmentation ensures that speed-critical operations use low-precision data while reliability-critical operations use high-precision data.
Solution Approach 2:
The patent changes the precision parameter of data used in different stages of processing. Computation operations use 10-bit or less floating point format for high speed, while parameter updates use 32-bit floating point format for high reliability. This dynamic parameter change optimizes both speed and reliability.
3Use of energy by stationary object
If the mantissa bit length is reduced to 3 bits or less, then hardware resources and power consumption are reduced, but trailing digit loss occurs in addition operations
Solution Approach 1:
The patent segments the processing into computation operations (where trailing digit loss is acceptable and corrected through accumulation) and parameter updates (where high precision is maintained using 32-bit floating point). This segmentation allows ultra-low precision arithmetic in the computation phase while preserving accuracy in the update phase.
Solution Approach 2:
The patent converts the potential harm of trailing digit loss in low-precision addition into a benefit by accumulating results over multiple operations. The quantization errors tend to average out over many operations, and the final high-precision parameter update corrects any systematic biases, ultimately achieving both low power consumption and high accuracy.
4Productivity
If 10-bit or less floating point data is used for parameter update, then calculation speed increases, but parameter accuracy deteriorates
Solution Approach 1:
The patent clearly segments the processing pipeline: computation operations use 10-bit or less floating point for speed, while parameter updates use 32-bit floating point for accuracy. This segmentation ensures that parameter updates maintain high precision even though computation operations use low precision.
Solution Approach 2:
The patent applies high precision (32-bit floating point) specifically to the parameter update operation where accuracy is critical, while using low precision (10-bit or less) only for computation operations where speed is critical. This local quality differentiation resolves the contradiction between update speed and parameter accuracy.
Data Source
AI summary
A computing system capable of obtaining a calculation speed exceeding that of 16-bit floating point processing while maintaining accuracy of calculation results. A computing system includes a parameter server, a communication path and a worker. The parameter server has a storage unit that stores a parameter value of a training target model, and a first conversion unit that converts the parameter value into data represented by a floating point number with 10 bits or less. The communication path transmits the data transmitted and received between the parameter server and the worker. The worker has a processing unit that computes a product and a sum of the data. The parameter server further has a second conversion unit that converts the data with 10 bits or less received from the worker into an updating difference, and an updating unit that updates the parameter value on the basis of the updating difference.