Neural Network Links With Piecewise Linear Functions
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional artificial neural networks face challenges in achieving high accuracy and efficiency while minimizing computational complexity and power consumption, especially in real-time applications with limited resources.
Innovation Solution
Implementing piecewise linear functions on links between neural network layers, which increases representation power without changing the network layout, and using efficient table-based evaluations to reduce runtime multiplications and power consumption.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If conventional activation functions (such as ReLU) are used to implement non-linear behavior in neural networks, then the network can achieve non-linear transformations, but the computational complexity and number of operations increase significantly
Solution Approach 1:
The patent merges the activation function application with the weight multiplication operation by implementing piecewise linear functions directly on the links between nodes. Instead of separately computing weighted sums and then applying activation functions at nodes, the piecewise linear transformation is integrated into the link evaluation process itself, combining multiple operations into a unified computational step that reduces overall complexity
Solution Approach 2:
The patent extracts the non-linear transformation operation from the node processing stage and relocates it to the link evaluation stage. By taking out the activation function application from the node and embedding it within the link's piecewise linear function evaluation, the system achieves non-linear behavior while simplifying the node operations to basic aggregation functions
2Measurement precision
If the number of trainable parameters is increased to improve representation power, then accuracy improves, but runtime computational complexity increases
Solution Approach 1:
The patent changes the parameterization approach by moving from traditional node-based parameters to link-based piecewise linear function parameters. Each link is characterized by a set of parameters defining the piecewise linear transformation, allowing for efficient computation while maintaining high representation power through the flexible piecewise linear models that can approximate complex non-linear relationships
3Measurement precision
If conventional neural network architectures are used to achieve high accuracy, then representation power is improved, but power consumption and computational requirements increase substantially
Solution Approach 1:
The patent substitutes the traditional mechanical computation approach (separate weight multiplication and activation function application) with a more efficient mathematical formulation using piecewise linear functions. This substitution replaces multiple computational steps with a single integrated evaluation that achieves the same functional result with reduced computational overhead and lower power consumption
Data Source
AI summary
Disclosed are systems, methods, and other implementations that include a machine-implemented artificial neural network including a plurality of nodes, with the nodes forming at plurality of layers including an input layer, at least one hidden layer, and an output layer, and a plurality of links, with each link coupling a corresponding source node and a receiving node. At least one link is configured to evaluate a piecewise linear function of a value provided by a first source node, from the plurality of nodes, to yield a value for a first receiving node coupled to the at least one link. Each node of a hidden layer is configured to aggregate values provided by links for which that node is the receiving node of the link, with the first receiving node providing non-linear output resulting, in part, from the at least one link configured to evaluate the piecewise linear function.