Gradient-Disentangled Embedding Sharing for NLU Pretraining

The patent segments the training process into two distinct phases: pretraining on vast corpora to learn general language patterns, and fine-tuning on task-specific datasets. This segmentation allows the model to achieve strong NLU performance through pretraining while reducing computational resource consumption during the specialized fine-tuning phase by leveraging the pre-learned representations.

The patent applies preliminary action by performing pretraining on vast language corpora before the actual task-specific training. This preliminary pretraining phase equips the model with general language understanding capabilities, so that subsequent fine-tuning requires fewer computational resources to achieve the same level of performance on specific NLU tasks.

The patent merges the training of upstream and downstream data embeddings by implementing gradient sharing between them. During fine-tuning, gradients from the downstream task flow back through the shared embedding layer, allowing both upstream and downstream embeddings to be updated simultaneously. This combining approach improves training efficiency while preserving the flexibility to handle different tasks through the modular encoder-decoder structure.

The patent creates universal data embeddings that serve multiple functions: they act as input representations for the decoder during generation and as updated parameters during fine-tuning. This multi-functionality allows the same embedding layer to adapt to different tasks while improving training efficiency through shared gradient updates, resolving the contradiction between flexibility and efficiency.