Vector Decomposition and Quantization for Neural Network Data Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural network models with deep hidden layers face challenges in training due to vanishing gradients, especially when dealing with data distributions that have outliers, leading to skewed distributions after normalization and quantization processing, which hinders proper prediction processing.
Innovation Solution
A data processing apparatus that performs vector decomposition and quantization processing, followed by convolution processing, using a weight coefficient matrix decomposed into a basis matrix and a real number coefficient vector, with adjustable data range processing to handle various data distributions, ensuring accurate prediction processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If batch normalization processing is performed on data with outliers, then the distribution of normalized values becomes concentrated near the average, but the data distribution becomes skewed and cannot achieve a moderately distributed distribution
Solution Approach 1:
The patent changes the parameters of normalization processing by introducing weighted coefficients that are learned during training. Instead of using fixed normalization parameters (mean and standard deviation), the system uses trainable parameters (gamma and beta) to scale and shift normalized values, allowing the model to adapt the distribution to achieve moderately distributed output even when input data contains outliers.
Solution Approach 2:
The patent introduces an intermediary mechanism (the batch normalization layer with learnable parameters) between the input data and subsequent processing layers. This intermediary transforms the skewed distributed data into moderately distributed data through learned scaling and shifting operations, preventing the propagation of skewed distributions to deeper layers while still handling outlier-containing input data.
2Productivity
If quantization processing is performed on skewed distributed data, then values after quantization are concentrated in a narrow range, but training processing does not progress properly
Solution Approach 1:
The patent performs preliminary normalization processing with learnable parameters before quantization processing. By adjusting the distribution characteristics through learned scaling and shifting operations prior to quantization, the system ensures that quantized values are properly distributed across the quantization range, preventing concentration in a narrow range and enabling proper training progress.
Solution Approach 2:
The patent changes the distribution parameters through learned normalization parameters (gamma and beta) before quantization. This parameter adjustment ensures that the data entering the quantization stage has an appropriate distribution, allowing quantization to produce well-distributed discrete values rather than concentrated values, thereby maintaining training effectiveness.
3Measurement precision
If deep hidden layers are added to increase model accuracy, then prediction accuracy improves, but gradients become so small that training processing does not proceed properly
Solution Approach 1:
The patent introduces batch normalization layers as intermediary components between deep hidden layers. These intermediaries normalize the activations and introduce learnable parameters to maintain gradient flow, preventing the vanishing gradient problem that occurs in very deep networks while allowing the network to achieve high prediction accuracy through increased depth.
Solution Approach 2:
The patent uses learnable normalization parameters (gamma for scaling and beta for shifting) that are updated during training to maintain appropriate gradient magnitudes throughout deep networks. These parameter changes allow the network to preserve gradient flow stability across many layers, enabling both deep architecture and reliable training.
4Reliability
If normalization processing is performed to prevent vanishing gradients, then training can proceed, but data distribution becomes skewed when outliers are present
Solution Approach 1:
The patent transforms fixed normalization parameters into learnable parameters that adapt during training. By learning optimal scaling (gamma) and shifting (beta) parameters, the system can handle outlier-containing data while maintaining moderately distributed outputs, resolving the contradiction between enabling training and preserving data distribution quality.
Solution Approach 2:
The patent makes the normalization parameters dynamic and adaptive rather than static. The learnable parameters are updated throughout training based on the actual data distribution, allowing the normalization process to adapt to outliers and maintain good data distribution characteristics throughout training, rather than being fixed beforehand.
Data Source
AI summary
Provide is data processing apparatus that performs highly accurate data processing accompanying vector decomposition processing, quantization processing, convolution processing and the like for any distribution of data. The data processing apparatus obtains a plurality of local solutions in the vector decomposition processing, selects a plurality of data adjustment processes performed before the quantization processing for each of the obtained local solutions of the vector decomposition processing, obtains the accuracy of the convolution processing, and then determines a local solution of the vector decomposition processing with highest accuracy and the data adjustment processing, with highest accuracy, performed before the quantization.