Compressed Dropout Mask Generation for Neural Network Memory Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current reconfigurable architectures face challenges in efficiently generating and storing dropout mask elements for neural network regularization, leading to suboptimal performance in preventing overfitting in machine learning applications.
Innovation Solution
A reconfigurable data processor architecture that implements dropout by generating and compressing mask elements, allowing for efficient storage and application of dropout masks through a compressed format, reducing memory consumption and loading/unloading time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If dropout mask elements are stored in uncompressed format, then dropout operation accuracy is maintained, but memory consumption and loading/unloading time increase
Solution Approach 1:
The patent extracts only the essential information from full-precision dropout mask elements by identifying and storing only the unique values and their positions. This extraction approach removes redundant data while preserving the functional accuracy needed for dropout operations, thereby reducing memory consumption without sacrificing operational precision.
Solution Approach 2:
The patent changes the representation parameters of dropout mask elements by transitioning from storing complete floating-point or integer values to storing compressed representations consisting of unique value identifiers and position indices. This parameter transformation enables significant memory reduction while maintaining the ability to reconstruct and apply dropout masks with sufficient accuracy.
2Measurement precision
If dropout mask elements are stored in uncompressed format, then dropout operation accuracy is maintained, but loading and unloading time increase
Solution Approach 1:
The patent extracts only the essential information from full-precision dropout mask elements by identifying and storing only the unique values and their positions. This extraction approach removes redundant data while preserving the functional accuracy needed for dropout operations, thereby reducing memory consumption without sacrificing operational precision.
Solution Approach 2:
The patent performs preliminary compression of dropout mask elements during the training phase, organizing data into compact representations with unique value identifiers and position indices. This preliminary action reduces the data volume that needs to be loaded and unloaded during subsequent operations, thereby reducing I/O time while maintaining accuracy.
3Device complexity
If reconfigurable architectures use traditional dropout implementation, then simplicity is maintained, but performance in preventing overfitting deteriorates
Solution Approach 1:
The patent segments the dropout mask generation process into distinct functional components: random number generation, unique value identification, position encoding, and mask reconstruction. This segmentation allows each component to be optimized independently while maintaining overall system simplicity, enabling better overfitting prevention through more sophisticated dropout patterns without excessive complexity.
Solution Approach 2:
The patent introduces dynamic elements to the dropout implementation by using reconfigurable architectures that can adaptively generate and apply dropout masks based on training progress and performance metrics. This dynamic approach improves overfitting prevention by allowing the dropout behavior to evolve during training while maintaining implementation simplicity through modular design.
Data Source
AI summary
A method for selectively dropping out feature elements from a tensor is disclosed. The method includes generating a mask that has a plurality of mask elements. Each mask element includes a corresponding plurality of bits representing either a first value or a second value, to indicate whether a corresponding feature element of the tensor output by a neural network layer is to be dropped out or retained. Each mask element of the plurality of mask elements of the mask is compressed to generate a corresponding compressed mask element of a plurality of compressed mask elements of a compressed mask, thereby generating the compressed mask from the mask. Each compressed mask element of the plurality of compressed mask elements includes a corresponding single bit. Feature elements are selectively dropped from the tensor, based on the compressed mask.