Neural Network Combining Node for Robust Decision Boundaries
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep neural networks face limitations such as slow convergence due to myopic gradient descent, inability to represent sequential logic or conscious reasoning, difficulty in interpretation, vulnerability to adversarial attacks, and lack of introspection and explicit knowledge representation.
Innovation Solution
The introduction of a 'combining' node and potentially a 'judgment' node in the machine learning network, which allows for selective blocking of back propagation to certain nodes, enabling more explicit knowledge representation, improved decision boundaries, and enhanced robustness against adversarial attacks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If gradient descent is used for training deep neural networks, then the network can learn from data, but the convergence is very slow and it requires millions of iterations
Solution Approach 1:
The patent applies preliminary action by performing multiple updates to the learned parameters between each presentation of a training datum. Instead of updating parameters once per datum as in conventional gradient descent, the system performs a predetermined number of updates (e.g., 1000 updates) for each training example, allowing the network to converge much faster while maintaining training effectiveness.
2Adaptability or versatility
If deep neural networks are made more complex to improve performance, then they can model more complex functions, but they become vulnerable to adversarial attacks and lack interpretable decision boundaries
Solution Approach 1:
The patent introduces an intermediary element in the form of a combining node that selectively integrates information from multiple source nodes. This combining node acts as a mediator that can choose which inputs to propagate forward, creating more robust and interpretable decision boundaries while maintaining the network's ability to model complex functions. The selective combining mechanism provides a level of control and interpretability that mitigates the vulnerability to adversarial attacks.
3Productivity
If back propagation is applied to all nodes for training, then all nodes contribute to learning, but it prevents explicit knowledge representation and slows down training
Solution Approach 1:
The patent applies the taking out principle by selectively blocking back propagation to certain nodes (such as combining nodes) while allowing it to proceed to other nodes. This extraction of back propagation from specific nodes allows the network to maintain explicit knowledge representation in those blocked nodes, as their weights are not updated through gradient descent. Meanwhile, other nodes continue to learn through normal back propagation, maintaining learning efficiency.
Data Source
AI summary
Computer systems and computer-implemented methods modify a machine learning network, such as a deep neural network, to introduce judgment to the network. A “combining” node is added to the network, to thereby generate a modified network, where activation of the combining node is based, at least in part, on output from a subject node of the network. The computer system then trains the modified network by, for each training data item in a set of training data, performing forward and back propagation computations through the modified network, where the backward propagation computation through the modified network comprises computing estimated partial derivatives of an error function of an objective for the network, except that the combining node selectively blocks back-propagation of estimated partial derivatives to the subject node, even though activation of the combining node is based on the activation of the subject node.