Dual-Level Contrastive Learning for Resource-Efficient NLP Models
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Natural language processing (NLP) models with increased complexity for improved accuracy often incur high inference-stage costs, making them unsuitable for resource-constrained platforms and real-time applications due to increased system resource consumption and latency.
Innovation Solution
A training system that employs a multi-path training architecture incorporating external knowledge and contrastive learning at encoder and decoder levels, using FIFO buffer memory and selective back-propagation to produce a resource-efficient machine-trained model with high accuracy and low latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the complexity of NLP models is increased to improve accuracy, then the accuracy and versatility of the model are improved, but the system resource consumption and inference time increase
Solution Approach 1:
The training process is segmented into multiple training paths (first training path, second training path, third training path) that operate in parallel. Each path processes training examples with different configurations (e.g., different auxiliary items, different target items), allowing the model to learn diverse patterns without increasing the complexity of a single model architecture. This segmentation enables high accuracy through diverse training while maintaining resource efficiency during inference by using a single model structure.
Solution Approach 2:
The system performs preliminary actions during the training phase by processing multiple training paths with various configurations (source items, target items, auxiliary items) to pre-compute and store optimized model parameters. The buffer memory stores training examples and intermediate results for efficient reuse. This preliminary processing during training enables the model to achieve high accuracy while maintaining low resource consumption during the actual inference stage.
2Measurement precision
If the complexity of NLP models is increased to improve accuracy, then the accuracy and versatility of the model are improved, but the inference time increases
Solution Approach 1:
The training process is divided into multiple parallel training paths that process different aspects of the data simultaneously. The first training path processes examples with auxiliary items, the second path processes examples with target items, and the third path processes examples with both. This segmentation allows comprehensive training for high accuracy while the final model maintains a single efficient architecture for fast inference.
Solution Approach 2:
The buffer memory continuously stores and retrieves training examples and intermediate results across training iterations. The system maintains continuous useful action by reusing previously processed data and intermediate computations, avoiding redundant processing. This continuity ensures high accuracy through充分的 training while maintaining efficient inference time by preventing unnecessary recomputation.
3Measurement precision
If buffer memory is used to store training information for contrastive learning, then the accuracy is improved through more data, but the system resource consumption increases
Solution Approach 1:
The buffer memory stores a limited but sufficient number of training examples and intermediate results needed for effective contrastive learning. Rather than storing all possible training data, the system uses a carefully selected subset that provides the necessary diversity for high accuracy while constraining memory usage to practical levels. This partial action approach achieves the desired accuracy improvement without excessive resource consumption.
Data Source
AI summary
A training system produces a resource-efficient machine-trained model via a training architecture that employs plural processing paths. Some of the processing paths incorporate the use of auxiliary information that imparts external knowledge about source items being processed. The training architecture also employs contrastive learning that operates at different respective levels within the training architecture. For instance, the training architecture uses encoder-level contrastive learning to compare output information generated by different encoders within the training architecture. The training architecture uses decoder-level contrastive learning to compare output information produced by different decoders within the training architecture. An inference-stage system performs an application task using the model produced by the training system.