Self-Supervised Back Propagation via Differentiable Function
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep learning systems face challenges in unsupervised training as they lack knowledge of the correct answer, limiting the use of derivatives computed by back propagation for operational data, which are not directly applicable to training data.
Innovation Solution
Computing partial derivatives of a selected differentiable function using back propagation without requiring knowledge of the correct answer, allowing these derivatives to be used in a second machine learning system for various objectives, such as improving interpretability and robustness, and enabling tasks like building ensembles and confidence scoring.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If back propagation is used with loss function for supervised training, then training accuracy is improved, but the method cannot be applied to unsupervised training where correct answer is unknown
Solution Approach 1:
The patent introduces an intermediary differentiable function that acts as a mediator between the neural network output and the training process. This function computes gradients without requiring the correct answer, enabling back propagation to work in unsupervised settings. The intermediary function translates the network output into a form that can generate meaningful gradients for weight updates even when supervision is absent.
Solution Approach 2:
The patent changes the parameter being optimized from a loss function requiring correct answers to a differentiable function of the network output that does not require correct answers. This parameter change allows the gradient computation to proceed without supervised labels, transforming the training paradigm from supervised to unsupervised while maintaining the back propagation mechanism.
2Productivity
If derivatives are computed using loss function for training data, then training performance is improved, but the derivatives cannot be used for operational data without correct answers
Solution Approach 1:
The patent creates a universal differentiable function that serves multiple purposes: it enables gradient computation during training without requiring correct answers, and the same function can be applied during operational phases. This multi-functional approach allows the system to maintain consistent derivative computation across both training and operational contexts, improving versatility.
Solution Approach 2:
The differentiable function enables the system to compute its own gradients autonomously without external supervision or correct answers. The neural network system becomes self-sufficient in generating the necessary gradient information for learning and operation, eliminating the need for external supervisory signals.
3Adaptability or versatility
If unsupervised training is performed without correct answers, then adaptability to real-world data is improved, but the ability to compute meaningful derivatives for training is worsened
Solution Approach 1:
The differentiable function serves as an intermediary that transforms the unsupervised learning problem into a form where meaningful derivatives can be computed. It mediates between the raw network output and the gradient computation process, ensuring that even without correct answers, the system can generate accurate and useful gradient signals for training.
Data Source
AI summary
A computer-implemented method for analyzing a first neural network via a second neural network according to a differentiable function. The method includes adding a derivative node to the first neural network that receives derivatives associated with a node of the first neural network. The derivative node is connected to the second neural network such that the second neural network can receive the derivatives from the derivative node. The method further includes feeding forward activations in the first neural network for a data item, back propagating a selected differentiable function, providing the derivatives from the derivative node to the second neural network as data, feeding forward the derivatives from the derivative node through the second neural network, and then back propagating a secondary objective through both neural networks. In various aspects, the learned parameters of one or both of the neural networks can be updated according to the back propagation calculations.