Neural Network Hashing for Large-Scale Classification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep neural networks face significant computational challenges in large-scale classification tasks due to the need for matrix multiplication across numerous layers, leading to high computation costs and potential losses in accuracy.
Innovation Solution
The implementation of a hash table-based method that uses a winner-takes-all (WTA) hashing technique to reduce computation time by storing weight matrices and retrieving only the top K weight vectors corresponding to the activation vector, allowing for efficient matrix multiplication with minimal loss in accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If matrix multiplication is performed across all nodes in deep neural networks for large-scale classification, then classification accuracy is maintained, but computation cost increases significantly
Solution Approach 1:
The patent segments the computation process into two phases: training phase where hash tables are built, and inference phase where only top-K nodes are computed. This segmentation allows the system to maintain accuracy by selectively computing only the most relevant nodes during inference, rather than performing exhaustive matrix multiplication across all nodes.
Solution Approach 2:
The patent performs preliminary action by pre-computing and storing hash tables during the training phase. These hash tables contain preprocessed weight information that enables rapid identification of top-K nodes during inference, eliminating the need for exhaustive computation and significantly reducing inference-time computation cost.
2Adaptability or versatility
If the number of output classes is increased to millions of classes, then large-scale classification capability is achieved, but computation time increases
Solution Approach 1:
The patent introduces hash tables as an intermediary data structure that decouples the number of classes from the computation time. The hash tables serve as a mediator between the input activation vector and the output class predictions, enabling the system to handle millions of classes by storing preprocessed weight information in a lookup-friendly format.
Solution Approach 2:
The patent performs preliminary action by pre-computing hash tables during training that encode the relationship between input features and all possible classes. This precomputation allows the inference process to rapidly identify top-K candidate classes using simple hash lookups and dot products, rather than computing all class predictions from scratch.
3Loss of energy
If top-K node selection is implemented to reduce computation, then computation cost decreases, but accuracy may be lost
Solution Approach 1:
The patent changes the parameter selection criterion from uniform node computation to ranked node selection based on hash code similarity. By computing hash codes for the input activation vector and selecting nodes with the highest similarity scores (top-K selection), the system prioritizes computation on the most relevant nodes, maintaining accuracy for dominant classes while reducing computation for less relevant classes.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for classification using a neural network. One of the methods for processing an input through each of multiple layers of a neural network to generate an output, wherein each of the multiple layers of the neural network includes a respective multiple nodes includes for a particular layer of the multiple layers: receiving, by a classification system, an activation vector as input for the particular layer, selecting one or more nodes in the particular layer using the activation vector and a hash table that maps numeric values to nodes in the particular layer, and processing the activation vector using the selected nodes to generate an output for the particular layer.