Shared Encoder Training Across Private Multi-Domain Corpora
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
It is difficult to obtain training corpus from a plurality of fields with a large cost, making it challenging to implement pre-training models effectively.
Innovation Solution
A method for generating a shared encoder using a master node that organizes child nodes to use private training samples, determining a target parameter set based on each child node's training results, thereby sharing training corpus across multiple fields while reducing costs and improving performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If training corpus is obtained from multiple fields through traditional methods, then the performance of pre-training models is improved, but the cost and difficulty of data acquisition increase significantly
Solution Approach 1:
The patent divides the training corpus acquisition process into multiple independent fields or domains, where each field contributes its own data to a unified pre-training model. This segmentation allows organizations to leverage their existing domain-specific data without needing to acquire data from all fields centrally, reducing the difficulty and cost of data acquisition while maintaining model performance across multiple domains.
2Adaptability or versatility
If pre-training models are implemented with comprehensive multi-field data, then the model's versatility and accuracy are enhanced, but the time and resources required for data collection and processing increase
Solution Approach 1:
The patent enables organizations to perform pre-training using their own existing domain-specific data before fine-tuning for specific tasks. This preliminary action eliminates the need to wait for comprehensive multi-field data to be collected centrally, allowing models to be developed and deployed more quickly while still achieving versatility through subsequent fine-tuning on task-specific data.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A method and an apparatus for generating a shared encoder, and an electronic device are provided by the present application, which belongs to a field of computer technology. The method includes: sending by a master node a shared encoder training instruction to child nodes, so that each child node obtains training samples based on a type of a target shared encoder included in the training instruction; sending an initial parameter set of the target shared encoder to be trained to each child node after obtaining a confirmation message returned by each child node, so that the initial parameter set is trained by each child node with its own training samples; obtaining an updated parameter set of the target shared encoder returned by each child node; determining a target parameter set corresponding to the target shared encoder based on a first preset rule and the updated parameter set of the target shared encoder returned by each child node. As a result, the method for generating the shared encoder may reduce the difficulty and cost of obtaining training corpus from a plurality of fields and improve the performance of the shared encoder.