Mixed-Precision Deep Learning with Multi-Memristive Synapses
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current memristive devices used in artificial neural networks face challenges due to their low precision, which creates a gap when combined with high-precision digital computing components, leading to inefficiencies in cognitive computing systems.
Innovation Solution
A method for mixed-precision deep learning using multi-memristive synapses, where each synapse is represented by a combination of memristive devices, with a weight update scheme that utilizes an arbitration scheme to set a threshold value based on device significance, allowing for high-precision weight updates while optimizing power consumption and reliability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Use of energy by moving object
If memristive devices are used as synapses in artificial neural networks, then area efficiency and power consumption are improved, but precision deteriorates due to low-resolution characteristics of memristors
Solution Approach 1:
Each synapse is divided into multiple memristive devices (e.g., 2-8 devices per synapse), where each device contributes a portion of the total synaptic weight. This segmentation allows the system to achieve higher precision through combination while maintaining the area and power benefits of memristive devices.
Solution Approach 2:
The patent combines multiple memristive devices with different conductance ranges to form composite synapses. By using devices with complementary characteristics (e.g., different resistance ranges or precision levels), the system achieves higher overall precision than individual devices could provide alone.
2Measurement precision
If multiple memristive devices are combined per synapse to improve precision, then device complexity increases, but this enables mixed-precision computing
Solution Approach 1:
Different memristive devices within the same synapse can have different precision levels or conductance ranges tailored to specific needs. This allows critical synapses to use higher-precision device combinations while less critical ones use simpler configurations, optimizing overall system precision where needed.
Solution Approach 2:
The system dynamically adjusts parameters such as the number of memristive devices per synapse, their conductance ranges, and update thresholds based on precision requirements. This enables mixed-precision computing where different parts of the network use different precision levels to optimize performance and resource usage.
3Measurement precision
If frequent weight updates are performed to maintain precision, then measurement precision is improved, but reliability deteriorates due to reduced device lifetime from programming operations
Solution Approach 1:
The patent applies weight updates selectively rather than continuously. Updates are performed only when the accumulated error exceeds a threshold or when precision requirements demand it, reducing the total number of programming operations and extending device lifetime while maintaining adequate precision.
Solution Approach 2:
Weight updates are performed periodically based on training epochs or error accumulation thresholds rather than after every operation. This periodic update strategy maintains precision over time while significantly reducing the frequency of programming operations that degrade device lifetime.
Data Source
AI summary
A computer-implemented method of mixed-precision deep learning with multi-memristive synapses may be provided. The method comprises representing, each synapse of an artificial neural network by a combination of a plurality of memristive devices, wherein each of the plurality of memristive devices of each of the synapses contributes to an overall synaptic weight with a related device significance, accumulating a weight gradient ΔW for each synapse in a high-precision variable, and performing a weight update to one of the synapses using an arbitration scheme for selecting a respective memristive device, according to which a threshold value related to the high-precision variable for performing the weight update is set according to the device significance of the respective memristive device selected by the arbitration schema.