Language Model Embedding Masking for Decentralized Learning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large global machine learning models, such as language models, are not conducive to decentralized learning due to their size, which exceeds the time constraints of decentralized learning operations, leading to inefficiencies in transmission and update processes.
Innovation Solution
A sparsified global language model is generated by applying a masking technique to the global embedding matrix, reducing its size for trainability and transferability, enabling decentralized learning across multiple computing devices.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a large global language model is used for decentralized learning, then model precision and recall are improved, but transmission time and network resource consumption increase excessively
Solution Approach 1:
The patent extracts and removes a portion of the embedding matrix from the global language model to create a sparsified version. This extraction principle directly addresses the contradiction by taking out unnecessary or less important parameters, reducing the model size for transmission while preserving the core functionality needed for maintaining precision and recall in decentralized learning operations
Solution Approach 2:
The patent segments the embedding matrix into different portions, with some parts retained and others removed or compressed. This segmentation allows the system to separate critical model components from non-critical ones, enabling efficient transmission of only the necessary parts to client devices while maintaining model effectiveness
2Measurement precision
If a large global language model is used for decentralized learning, then model precision and recall are improved, but network resource consumption increases excessively
Solution Approach 1:
The patent applies the extraction principle by removing redundant or less important embedding parameters from the global model. This reduction in model size directly decreases the network bandwidth and computational resources required for transmission and processing, while the carefully selected retained parameters maintain the model's recall capability in decentralized learning
Solution Approach 2:
By segmenting the embedding matrix and selectively transmitting only essential portions to client devices, the patent reduces network resource consumption. The segmentation strategy ensures that only the most impactful parameters are transmitted, maintaining model recall while minimizing energy and bandwidth usage
3Productivity
If the global language model size is reduced for efficient transmission, then transmission time and network resource consumption are reduced, but model trainability and transferability may be compromised
Solution Approach 1:
The patent changes the parameters of the embedding matrix by selectively retaining or removing specific embeddings based on their importance or usage frequency. This parameter change approach allows the model to be compressed for efficient transmission while preserving the critical parameters needed for maintain trainability and transferability in decentralized learning scenarios
Solution Approach 2:
The patent applies local quality by differentiating between different parts of the embedding matrix, treating important embeddings differently from less important ones. This selective approach ensures that critical parameters are preserved for model transferability while non-critical parameters are reduced or removed to improve transmission efficiency
Data Source
AI summary
Implementations disclosed herein are directed to techniques for enabling decentralized learning of global language models (LMs). Remote processor(s) of a remote system can obtain a global LM that includes a global embedding matrix, generate a global embedding mask for the global embedding matrix using a masking technique, apply the global embedding mask to global embedding matrix to generate a sparsified global LM that includes a masked global embedding matrix that is a masked version of the global embedding matrix, transmit the sparsified global LM to computing device(s) that are participating in a given round of decentralized learning for the global language model, receive corresponding updates from the computing device(s), and cause the global LM to be updated based on the corresponding updates. By generating the global embedding mask and applying it to the global embedding matrix, the transferable size of the global LM is reduced thereby enabling decentralized learning thereof.