Perceptron Learning Rate Based on Input Standard Deviation

Solution Overview

Problem

Existing methods for training artificial neural networks face challenges in determining optimal hyperparameters, particularly the learning rate, which affects both training time and accuracy.

Innovation Solution

The proposed solution involves calculating individual learning rates for each perceptron based on the standard deviation of input data, eliminating the need for sweeping a range of learning rates and allowing for more accurate and efficient training.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Loss of time

If a high learning rate is used, then training time is reduced, but training accuracy deteriorates due to overshooting

Engineering Contradiction:

Improvetraining timeVSAvoidtraining accuracy

Core Design Contradiction:

Loss of timeVSMeasurement precision

Solution Approach 1:

The patent applies local quality by assigning different learning rates to different perceptrons based on their individual input data characteristics (standard deviation). Each perceptron receives a customized learning rate tailored to its specific data properties, allowing some perceptrons to learn faster while others maintain higher precision, thus resolving the contradiction between training speed and accuracy at the local level.

Inventive Principle:

Principle #3Local quality

Solution Approach 2:

The patent changes the learning rate parameter dynamically based on the standard deviation of input data for each perceptron. Instead of using a fixed or uniformly high learning rate, the system adjusts the learning rate parameter according to data characteristics, enabling faster convergence where appropriate while maintaining accuracy where needed.

Inventive Principle:

Principle #35Parameter changes

2Measurement precision

If a low learning rate is used, then training accuracy is improved, but training time increases excessively

Engineering Contradiction:

Improvetraining accuracyVSAvoidtraining time

Core Design Contradiction:

Measurement precisionVSLoss of time

Solution Approach 1:

The patent applies local quality by assigning different learning rates to different perceptrons based on their individual input data characteristics (standard deviation). Each perceptron receives a customized learning rate tailored to its specific data properties, allowing some perceptrons to learn faster while others maintain higher precision, thus resolving the contradiction between training speed and accuracy at the local level.

Inventive Principle:

Principle #3Local quality

Solution Approach 2:

The patent changes the learning rate parameter dynamically based on the standard deviation of input data for each perceptron. Instead of using a fixed or uniformly low learning rate, the system adjusts the learning rate parameter according to data characteristics, enabling faster convergence where appropriate while maintaining accuracy where needed.

Inventive Principle:

Principle #35Parameter changes

3Measurement precision

If hyperparameter sweeping is performed to find optimal values, then accuracy can be improved, but the complexity and time required for training increases significantly

Engineering Contradiction:

Improvetraining accuracyVSAvoidhyperparameter tuning complexity

Core Design Contradiction:

Measurement precisionVSDevice complexity

Solution Approach 1:

The patent applies self-service by enabling each perceptron to determine its own optimal learning rate based on its input data characteristics. The system automatically calculates the standard deviation of input data for each perceptron and uses this to set the learning rate, eliminating the need for manual or exhaustive hyperparameter sweeping while achieving optimal training parameters.

Inventive Principle:

Principle #25Self-service

Solution Approach 2:

The patent changes the learning rate parameter dynamically based on the standard deviation of input data for each perceptron. Instead of using a fixed or uniformly low learning rate, the system adjusts the learning rate parameter according to data characteristics, enabling faster convergence where appropriate while maintaining accuracy where needed.

Inventive Principle:

Principle #35Parameter changes

Data Source

PatentUS12271806B2Artificial neural network training

Publication Date: 2025.04.08 RAYTHEON CO

AI summary

An artificial neural network receives data for the inputs of a perceptron in the artificial neural network. The network determines an average of the data for each of the inputs of the perceptron, determines a standard deviation of the average for each of the inputs of the perceptron, and determines an average of the standard deviations for the perceptron. The network then sets a learning rate for the perceptron equal to the average of the standard deviations, and trains the artificial neural network using the learning rate for the perceptron.