Neural Network Tensor Dropout With Compressed Mask Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current reconfigurable architectures face challenges in efficiently generating and storing dropout mask elements for neural network regularization, leading to high memory consumption and prolonged loading/unloading times.
Innovation Solution
A system utilizing a reconfigurable data processor with a coarse-grain architecture that implements dropout by generating and compressing mask elements, allowing for efficient storage and application of dropout masks through logical right shifting and reordering, reducing memory usage and operation time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If dropout mask elements are stored in full precision format, then dropout accuracy is maintained, but memory consumption increases
Solution Approach 1:
The patent changes the precision parameter of dropout mask elements from full precision (e.g., 32-bit floating point) to reduced precision (e.g., 8-bit integer or binary). This parameter change maintains the essential functionality of dropout while significantly reducing memory consumption, directly resolving the contradiction between accuracy and memory usage.
Solution Approach 2:
The patent extracts only the essential information needed for dropout functionality, discarding redundant high-precision data. By taking out only the necessary mask bits (0 or 1) rather than storing complete floating-point values, the system achieves accurate dropout with minimal memory requirements.
2Ease of operation
If dropout masks are loaded into memory before processing, then dropout operation can be performed, but loading time increases
Solution Approach 1:
The patent generates dropout masks on-demand during the forward propagation process rather than pre-loading them. This preliminary action of generating masks only when needed eliminates the loading time while ensuring dropout operations can be performed, resolving the contradiction between operational ease and time loss.
Solution Approach 2:
The system generates its own dropout masks internally during processing rather than relying on external pre-loaded masks. This self-service approach allows the neural network to create and apply dropout masks autonomously during each training iteration, eliminating loading delays while maintaining dropout functionality.
3Measurement precision
If full precision dropout mask elements are used, then dropout precision is maintained, but processing speed decreases
Solution Approach 1:
The patent changes the data type parameter of dropout mask elements from high-precision formats (32-bit float) to low-precision formats (8-bit integer or binary). This parameter change enables faster processing speeds while maintaining sufficient precision for dropout functionality, directly resolving the contradiction between precision and processing speed.
Solution Approach 2:
The patent uses simplified, low-precision dropout mask representations that are computationally inexpensive and can be rapidly generated and discarded. These lightweight mask objects enable fast processing while providing sufficient precision for the dropout operation, resolving the trade-off between precision and speed.
4Reliability
If dropout masks are uncompressed, then mask quality is maintained, but storage space increases
Solution Approach 1:
The patent changes the compression parameter by storing dropout masks in reduced precision formats (e.g., binary or 8-bit integer instead of 32-bit float). This parameter change achieves significant storage space reduction while maintaining the essential quality needed for accurate dropout operation, resolving the contradiction between mask quality and storage space.
Data Source
AI summary
A method for selectively dropping out feature elements from a tensor in a neural network includes receiving a first tensor from a first layer of a neural network and obtaining a compressed mask for the first tensor. N mask bits of the compressed mask are received at each of N lanes of a reconfigurable computing unit and feature elements of the first tensor are respectively received at the N lanes. Feature elements are selectively dropped out from the first tensor to generate feature elements to use as at least part of a second tensor by selecting, based on a single mask bit of the compressed mask selected based on the lane, either a zero value or a feature element received at the lane for a feature element of the second tensor. The second tensor is propagated to a second layer of the neural network.