Mapping Activation Functions to Data Properties in Deep Learning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep learning models face errors due to data properties like skewness, kurtosis, and discontinuities, particularly when concatenating time series data, leading to issues such as underfitting or overfitting.
Innovation Solution
The system maps specific activation functions to neural network layers based on input data properties and tunes model parameters, such as sample weights, to minimize errors around concatenation points, thereby aligning the neural network with the data characteristics and reducing discontinuity-related errors.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Length of moving object
If data is concatenated to form longer stretches for machine learning, then data length is increased, but modeling errors increase due to discontinuities
Solution Approach 1:
The patent applies different activation functions to different regions of the neural network based on local data characteristics. Specifically, it identifies discontinuity points in the concatenated time series data and applies specialized activation functions (such as ReLU or leaky ReLU) to neurons handling these discontinuous regions, while using different activation functions for continuous regions. This local adaptation allows the model to handle discontinuities without compromising overall modeling accuracy.
Solution Approach 2:
The patent dynamically changes activation function parameters based on data properties. It analyzes statistical properties (mean, variance, skewness, kurtosis) of different segments of the concatenated data and selects or adjusts activation function parameters accordingly. For example, it may adjust the alpha parameter in leaky ReLU functions based on the local variance or skewness of the data segment being processed.
2Device complexity
If standard activation functions are used in neural networks, then model simplicity is maintained, but modeling errors increase due to data properties like skewness and kurtosis
Solution Approach 1:
The patent implements local quality by selecting different activation functions for different neurons or layers based on the statistical properties of the data they process. Instead of using a uniform activation function throughout the network, it analyzes local data characteristics (skewness, kurtosis, range boundedness) and assigns activation functions tailored to each region's needs, thereby improving accuracy without requiring complete model redesign.
Solution Approach 2:
The patent introduces dynamic selection of activation functions based on data characteristics. The system evaluates statistical properties of input data and dynamically chooses appropriate activation functions during the modeling process. This dynamic adaptation allows the model to respond to varying data properties without requiring manual reconfiguration of the entire network architecture.
3Adaptability or versatility
If data exhibits skewness, kurtosis, or range boundedness, then data reflects real-world properties, but standard activation functions fail to account for these properties causing modeling errors
Solution Approach 1:
The patent addresses real-world data properties by applying localized activation functions matched to specific data characteristics. It identifies regions with skewness, kurtosis, or range constraints and applies activation functions designed to handle these properties (e.g., bounded activation functions for range-constrained data, asymmetric functions for skewed data). This localized matching improves the model's ability to represent real-world data properties accurately.
Solution Approach 2:
The patent converts the potentially harmful effect of data property mismatches into a benefit by using the statistical properties (skewness, kurtosis, boundedness) as selection criteria for activation functions. Instead of treating these properties as obstacles, the system uses them to guide activation function selection, transforming what would be sources of error into features that improve model accuracy and adaptability.
Data Source
AI summary
The disclosure relates to systems and methods of mapping deep learning activation functions to input data. For example, a system may select one or more activation functions for one or more layers of a neural network based on properties that cause modeling errors or otherwise should be accounted for. The properties that may cause modeling error or otherwise should be accounted for in deep learning may include skewness, kurtosis, range boundedness, and/or other properties. The selected activation functions may be placed at one or more layers of a neural network. In this manner, the neural network may be tuned with specific activation functions that align with the properties of the input data.