Reversible Sparse Dimensionality Reduction Encoding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning models require significant computational resources and storage due to their large size, especially when dealing with sparse, high-dimensional input and output vectors, which can lead to inefficiencies in training and inference processes.
Innovation Solution
Training machine learning models using encoded data structures like Bloom filters or count-min sketches, which reduce the dimensionality of input and output vectors, thereby decreasing the number of weights and resources needed, while maintaining accuracy through adjustments based on differences determined by loss functions like cross-entropy loss.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If machine learning models use large weight matrices to process high-dimensional vectors, then model accuracy is maintained, but computational resources and storage requirements increase significantly
Solution Approach 1:
The patent extracts only the essential information from high-dimensional vectors by identifying and processing non-zero elements (sparse structure). By taking out only the relevant data points rather than processing the entire high-dimensional space, the model achieves the same predictive accuracy with significantly reduced computational resources and storage requirements.
Solution Approach 2:
The patent transforms the problem from high-dimensional space to low-dimensional space by using dimensionality reduction techniques. It projects data onto a lower-dimensional subspace while preserving the essential structure and relationships needed for accurate predictions, thereby reducing the size of weight matrices and computational complexity.
2Loss of information
If machine learning models process sparse high-dimensional vectors directly, then complete information is retained, but processing time and computational complexity increase
Solution Approach 1:
The patent extracts only the non-zero elements from sparse vectors, ignoring the numerous zero elements that carry no information. This extraction approach retains all meaningful information while dramatically reducing the number of operations required during training and inference, thus decreasing processing time without information loss.
Solution Approach 2:
The patent applies partial action by processing only the necessary subset of data (non-zero elements) rather than the complete high-dimensional vector. This selective processing maintains information integrity while avoiding unnecessary computations on zero elements, thereby reducing processing time and computational complexity.
3Productivity
If machine learning models use reduced dimensionality encoding, then computational efficiency improves, but model size and accuracy may be compromised
Solution Approach 1:
The patent changes the parameters of the encoding process by using learned projection matrices that optimize the transformation from high-dimensional to low-dimensional space. These parameter transformations are designed to preserve the variance and structure of the original data, ensuring that accuracy is maintained even as dimensionality is reduced and computational efficiency improves.
4Adaptability or versatility
If conventional models process large weight matrices, then comprehensive pattern recognition is achieved, but storage requirements and memory usage increase
Solution Approach 1:
The patent extracts the essential pattern recognition capabilities by operating in reduced-dimensional space. By identifying and processing only the critical dimensions and non-zero elements, the model maintains comprehensive pattern recognition ability while requiring significantly less storage space for weight matrices and model parameters.
Solution Approach 2:
The patent transitions from high-dimensional weight matrices to low-dimensional representations, preserving the essential pattern recognition functionality. By projecting data and weights into a lower-dimensional subspace, the model achieves comprehensive pattern recognition with reduced storage requirements and smaller memory footprint.
Data Source
AI summary
The present disclosure is directed to reducing model size of a machine learning model with encoding. The input to a machine learning model may be encoded using a probabilistic data structure with a plurality of mapping functions into a lower dimensional space. Encoding the input to the machine learning model results in a compact machine learning model with a reduced model size. The compact machine learning model can output an encoded representation of a higher-dimensional space. Use of such a machine learning model can include decoding the output of the machine learning model into the higher dimensional space of the non-encoded input.