Meta-ML Backpropagation Using Detachment for Nondifferentiable Loss
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning optimization methods are limited by non-differentiable and discontinuous loss functions, leading to suboptimal models and inefficient optimization processes, particularly in domains like finance with missing data values.
Innovation Solution
A meta-ML optimization engine using a semi-discrete calculus detachment operator to simulate a variant of the backpropagation algorithm, replacing the derivative operator to optimize non-differentiable and discontinuous loss functions, and handle missing data values.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If gradient descent optimization is used for training neural networks, then the loss function can be optimized efficiently, but the method is limited to differentiable and continuous loss functions only
Solution Approach 1:
The patent transforms the continuous derivative-based optimization parameters into discrete detachment-based parameters. By computing detachments (signs of differences) instead of continuous derivatives, the optimization method can handle non-differentiable and discontinuous loss functions while maintaining iterative update capability through modified weight adjustment rules based on detachment sequences.
Solution Approach 2:
The patent replaces the mechanical derivative computation system with a discrete detachment computation system. Instead of relying on continuous calculus operations that require differentiability, the method uses discrete sign-based detachments that can be computed for any function including non-differentiable and discontinuous ones, thereby substituting the mathematical foundation of the optimization process.
2Reliability
If numeric derivative approximation is used to handle non-differentiable points, then the loss function can be optimized at anomalous points, but the optimization may converge to local non-optimum points and increase computational overhead
Solution Approach 1:
The patent inverts the conventional approach by not approximating derivatives at anomalous points, but rather by directly computing detachments that inherently handle anomalous points without approximation. Instead of trying to make non-differentiable points differentiable through numeric approximation, the method reverses the logic to work naturally with discontinuities and non-differentiable points through discrete detachment computations.
Solution Approach 2:
The patent introduces detachments as an intermediary between the loss function and the optimization process. Rather than directly computing derivatives that fail at anomalous points, detachments serve as a mediator that provides meaningful gradient-like information even for non-differentiable and discontinuous functions, enabling optimization to proceed through these problematic regions without approximation errors.
3Loss of information
If manual input of missing data values is performed, then the data completeness is improved, but erroneous data may be introduced and optimization accuracy is reduced
Solution Approach 1:
The patent enables the optimization process to handle missing data values autonomously without manual intervention. The detachment-based optimization naturally accommodates missing values by computing detachments based on available data patterns, allowing the system to self-correct and continue optimization without human input that could introduce errors.
Solution Approach 2:
The patent uses detachments as an intermediary mechanism that bridges the gap between incomplete data and optimization requirements. Rather than requiring complete data through manual input, detachments provide a mathematical intermediary that enables meaningful computation with partial data, preserving optimization accuracy while maintaining data completeness.
Data Source
AI summary
A method and system for building and implementing a meta-machine learning (meta-ML) optimization engine for a neural network (NN) or a machine learning (ML) connective model. A computer processor may iteratively simulate a backpropagation algorithm by executing a sequence of optimization steps. At each optimization step a position of a loss function may be determined that may be closer than a previously determined position of the loss function to a local minimum. A computer processor may compute and store after each iteration a detachment of the loss function, learning rate, and optimal learning rate. A computer processor may train a machine learning connective model to model the optimal learning rates of the simulated backpropagation algorithm. The meta-ML optimization engine may be implemented for a NN or ML connective model by generating a modified backpropagation algorithm in which algorithmic features of gradient descent may be replaced by the meta-ML optimization engine.