Neural Network Regression Using Differentiable Extraction Gradients

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Trained neural networks used for regression tasks face accuracy issues due to multiple candidates with high likelihoods affecting the computation of real numbers, leading to deviations from true values.

Innovation Solution

A model generation apparatus trains a neural network module through machine learning, performing an extraction operation in forward propagation and replacing it with a differentiable alternative operation in backpropagation to compute an approximate gradient, allowing the network to be trained despite non-differentiability.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Measurement precision

If an extraction operation is used to select the most promising candidate from multiple candidates, then the accuracy of regression is improved by focusing on the most likely candidate, but the operation becomes non-differentiable which prevents training through backpropagation

Engineering Contradiction:

Improveaccuracy of regressionVSAvoiddifferentiability of operation

Core Design Contradiction:

Measurement precisionVSDevice complexity

Solution Approach 1:

The patent introduces a differentiable alternative operation as an intermediary to replace the non-differentiable extraction operation during backpropagation. This intermediary operation approximates the gradient of the extraction operation, allowing the neural network to be trained while maintaining the accuracy benefits of extracting the most promising candidate. The intermediary acts as a bridge between the forward propagation (where extraction is used) and backpropagation (where differentiability is needed).

Inventive Principle:

Principle #24Intermediary (Mediator)

2Adaptability or versatility

If multiple candidates with high likelihoods are considered in regression, then the computation covers more possibilities, but the accuracy decreases due to unintended candidates affecting the result

Engineering Contradiction:

Improvecoverage of candidate valuesVSAvoidaccuracy of computed real number

Core Design Contradiction:

Adaptability or versatilityVSMeasurement precision

Solution Approach 1:

The patent extracts only the most promising candidate from the set of multiple candidates by using an extraction operation that selects the element satisfying a predetermined condition (typically the one with maximum likelihood). This extraction focuses the computation on the most relevant candidate, filtering out the distracting effects of other high-likelihood candidates that would otherwise reduce accuracy. The extraction operation is applied during forward propagation to achieve precise regression results.

Inventive Principle:

Principle #2Taking out (Extraction)

Data Source

PatentUS12374095B2Model generation apparatus, regression apparatus, model generation method, and computer-readable storage medium storing a model generation program

Publication Date: 2025.07.29 OMRON CORP

AI summary

A model generation apparatus trains, through machine learning, a neural network module that includes an extraction operation to extract an element satisfying a predetermined condition from a set of targets. In the machine learning, the model generation apparatus performs the extraction operation in a phase of forward propagation with the neural network module, and replaces, in a phase of backpropagation, the extraction operation with a differentiable alternative operation and differentiates the alternative operation to compute an approximate gradient corresponding to differentiation for the extraction operation.