Data Processing with Riemannian Gradients for Fixed-Rank Models
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional machine learning models with millions or billions of parameters face high storage, transmission, and computing overheads, necessitating a solution to reduce these costs.
Innovation Solution
A data processing method utilizing Riemannian gradients to ensure fixed-rank attributes of data, specifically through Riemannian manifold optimization, which includes computing a Riemannian gradient and updating data based on this gradient to maintain a fixed rank, thereby reducing storage, transmission, and computing overheads.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional machine learning models with millions or billions of parameters are used, then model accuracy and capability are improved, but storage overheads, transmission overheads, and computing overheads increase significantly
Solution Approach 1:
The patent extracts and removes redundant model parameters through pruning techniques, eliminating unnecessary connections in neural networks while preserving essential functionality. This reduces the total number of parameters from millions or billions to a more manageable size, directly addressing the contradiction between model accuracy and storage overhead.
Solution Approach 2:
The patent employs low-rank factorization to decompose large parameter matrices into nested smaller matrices. By representing a large weight matrix as a product of multiple smaller matrices with lower ranks, the storage requirements are significantly reduced while approximately maintaining the original model's computational capability and accuracy.
2Measurement precision
If conventional machine learning models with millions or billions of parameters are used, then model capability is improved, but transmission overheads increase
Solution Approach 1:
The patent extracts and removes redundant parameters that contribute minimally to model capability, reducing the total parameter count that needs to be transmitted. This pruning approach maintains essential model functionality while significantly reducing transmission data volume.
Solution Approach 2:
By factorizing large parameter matrices into nested lower-rank matrices, the patent reduces the number of parameters that must be transmitted. The decomposed matrices require fewer elements to represent the same computational function, directly reducing transmission overhead while preserving model capability.
3Measurement precision
If conventional machine learning models with millions or billions of parameters are used, then model performance is improved, but computing overheads increase
Solution Approach 1:
The patent removes redundant parameters and computations from the model, eliminating unnecessary mathematical operations during inference and training. This pruning reduces the computational complexity from O(n²) or higher to lower complexity operations, directly decreasing computing overhead while maintaining essential model performance.
Solution Approach 2:
By decomposing large parameter matrices into nested lower-rank matrices, the patent transforms expensive matrix multiplications into sequences of cheaper operations. The factorized representation enables computation with fewer floating-point operations, reducing power consumption and computing overhead while approximately preserving model performance.
4Quantity of substance
If data is updated to reduce storage overheads, then storage efficiency is improved, but model convergence may be affected
Solution Approach 1:
The patent applies low-rank factorization with carefully selected rank parameters that balance storage efficiency and convergence. By optimizing the factorization rank and using techniques like alternating least squares for updating factor matrices, the patent ensures that storage-reduced representations maintain sufficient information for reliable model convergence during training.
Data Source
AI summary
Provided are a data processing method and a related device. The method includes: determining, based on first data in a data processing device A and second data from a first data processing device, a Riemannian gradient of the first data, where the first data and the second data are data having a Riemann characteristic; and then updating the first data based on the Riemannian gradient. During implementation of technical solutions provided in this application, because the first data and the second data are data having a Riemann characteristic, to be specific, the first data and the second data are data in a Riemannian manifold with a fixed rank, a gradient of the first data obtained based on the first data and the second data is a gradient on the Riemannian manifold, namely, a Riemannian gradient.