Compressed Dropout Mask Reordering for Memory-Efficient Neural Layers
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current reconfigurable architectures face challenges in efficiently generating and storing dropout mask elements for neural network regularization, leading to suboptimal performance in machine learning applications.
Innovation Solution
A reconfigurable data processor architecture that implements dropout by generating and compressing mask elements, allowing for efficient storage and application of dropout masks through a compressed format, reducing memory consumption and loading/unloading time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If dropout mask elements are stored in uncompressed format, then ease of operation is improved, but memory consumption increases
Solution Approach 1:
The patent extracts only the essential information from the dropout mask by identifying and storing only the unique mask elements rather than storing all mask elements in their original uncompressed form. This reduction in stored data quantity directly addresses the memory consumption issue while maintaining operational efficiency.
Solution Approach 2:
The patent changes the representation parameter of the dropout mask from uncompressed binary format to a compressed format that encodes mask elements more efficiently. By transforming the data structure and using bitwise operations, the system achieves lower memory consumption without sacrificing the ability to apply dropout operations.
2Quantity of substance
If dropout mask elements are compressed, then memory consumption is reduced, but device complexity increases
Solution Approach 1:
The patent replaces complex compression algorithms with simpler bitwise operations and lookup table mechanisms. Instead of using sophisticated compression software or hardware, the system employs fundamental bit manipulation techniques that are natively supported by digital logic, thereby reducing device complexity while achieving effective compression.
Solution Approach 2:
The patent uses lookup tables to store pre-computed compression patterns and decompression rules. By copying and reusing these standardized patterns across different dropout operations, the system avoids implementing complex real-time compression logic, thus reducing device complexity while maintaining compression effectiveness.
3Quantity of substance
If dropout mask elements are reordered, then memory consumption is reduced, but ease of operation worsens
Solution Approach 1:
The patent performs reordering of mask elements in advance during the mask generation phase, organizing them in a compressed sequence before storage. By preparing the reordered structure beforehand, the system eliminates the need for complex real-time reordering operations during dropout application, thus maintaining ease of operation while achieving memory savings.
Solution Approach 2:
The patent introduces an intermediary data structure that bridges the original mask format and the compressed storage format. This intermediary representation maintains a logical connection to the original element positions while enabling compact storage, thus preserving ease of operation during mask application despite physical reordering for compression.
4Loss of time
If dropout mask elements are stored in compressed format, then loading/unloading time is reduced, but device complexity increases
Solution Approach 1:
The patent replaces time-consuming decompression algorithms with fast bitwise operations that can be executed in parallel. By substituting complex sequential decompression logic with native CPU bit manipulation instructions, the system achieves rapid loading/unloading of compressed masks without requiring sophisticated decompression hardware or software.
Data Source
AI summary
A method for selectively dropping out feature elements from a tensor is disclosed. The method includes generating a mask that has a plurality of mask elements arranged in a first order. A compressed mask is generated, which includes a plurality of compressed mask elements arranged in a second order that is different from the first order. For example, each mask element of the plurality of mask elements of the mask is compressed to generate a corresponding compressed mask element of the plurality of compressed mask elements of the compressed mask. Individual compressed mask element of the plurality of compressed mask elements is indicative of whether a corresponding feature element of the tensor output by a neural network layer is to be dropped out or retained. Feature elements are selectively dropped from the tensor, based on the compressed mask.