Monotonic Neural Network Layers Using Composite Activation Functions
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for ensuring monotonicity in machine learning models, such as building monotonic models by construction or employing heuristic regularization, face challenges like vanishing gradient problems, structural rigidity, and high resource requirements, especially when training on noisy real-world data.
Innovation Solution
Implementing monotonic neural network layers using unsaturated monotonic activation functions like ReLU and ELU, combined with their point reflections, and determining selector weights to create monotonically increasing composite activation functions, which allow learning any monotonic function while reducing computational resources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If monotonic models are built by construction using classical activation functions, then monotonicity is ensured, but vanishing gradient problems occur and models cannot be properly trained
Solution Approach 1:
The patent changes the parameter selection of activation functions from classical saturated functions (sigmoid, tanh) to unsaturated monotonic functions (ReLU, ELU). This parameter change resolves the vanishing gradient problem while preserving monotonicity, enabling proper model training.
2Reliability
If monotonicity is enforced by restricting weights to nonnegative or nonpositive values, then monotonicity is maintained, but the model becomes structurally rigid and cannot approximate non-convex monotonic functions
Solution Approach 1:
The patent introduces dynamic selector weights that can be learned during training to switch between different activation functions (convex and concave). This dynamic mechanism allows the model to adapt to both convex and non-convex monotonic functions, resolving the structural rigidity issue.
Solution Approach 2:
The patent creates a composite activation function structure that combines multiple activation functions (e.g., ReLU and its point reflection) with learnable selector weights. This composite approach enables the model to approximate any monotonic function by dynamically selecting appropriate components.
3Reliability
If heuristic regularization is employed to make models monotonic, then monotonicity can be achieved, but multiple training rounds are required and computational resources are significantly increased
Solution Approach 1:
The patent applies monotonicity constraints directly to the activation functions and weight parameters during the initial model construction, rather than requiring iterative regularization. This preliminary action ensures monotonicity from the start and eliminates the need for multiple training rounds.
Solution Approach 2:
The monotonic neural network layers are designed to inherently maintain monotonicity through their structure (monotonic activation functions with constrained weights), making the model self-regulating without requiring external regularization mechanisms or multiple training iterations.
4Adaptability or versatility
If deep lattice networks are used to approximate non-convex monotonic functions, then function approximation capability is improved, but the number of parameters increases significantly and memory requirements increase
Solution Approach 1:
The patent changes the activation function parameters from saturated to unsaturated monotonic functions, which enables the network to learn non-convex monotonic relationships more efficiently with fewer parameters compared to deep lattice networks.
Solution Approach 2:
The patent uses composite activation functions combining convex and concave monotonic functions with learnable selector weights, providing a more parameter-efficient approach to approximating non-convex monotonic functions compared to traditional deep lattice network architectures.
Data Source
AI summary
A facility for generating monotonic fully connected layer blocks for a machine learning model is described. The facility receives an indication of a convex constituent monotonically increasing activation function and a concave constituent monotonically increasing activation function for a monotonic layer. The facility generates a composite monotonic activation function made up of the convex and concave constituent activation functions. The facility receives an indication of a monotonicity indicator vector for the monotonic dense layer block. The facility determines one or more selector weights for the composite activation function. The facility initializes a sign for each weight of one or more kernel weights included in the monotonic layer and initializes a bias vector. The facility generates the monotonic dense layer block based on the composite activation function, the monotonicity indicator vector, the selector weights, the sign for each kernel weight, and the bias vector.