Lightweight Prompt Encoder for Frozen Embedding Models

The patent segments the embedding system into two distinct components: a frozen pretrained embedding model (e.g., BERT, RoBERTa) that provides the core embedding capabilities, and a lightweight prompt encoder that generates prompts for specific tasks. This segmentation allows the heavy computational burden to be divided, where the pretrained model handles general embedding tasks with its fixed parameters, while only the lightweight prompt encoder requires updating during fine-tuning, significantly reducing the learning cost while maintaining high embedding performance.

The patent uses the concept of copying by creating a prompt encoder that generates prompts as representations of input data. These prompts are then input to the frozen embedding model to generate embedding representations. This copying approach allows the system to leverage the pretrained model's knowledge without requiring full fine-tuning, as the prompts serve as compressed representations that guide the embedding generation process.

The patent extracts the task-specific adaptation component from the full embedding model by separating it into a dedicated prompt encoder. The pretrained embedding model remains frozen, extracting only the necessary adaptability functionality into the prompt encoder. This allows task-specific performance to be improved through prompt generation without requiring computational resources to be expended on retraining the entire large-scale embedding model.

Instead of performing full fine-tuning of the embedding model (excessive action), the patent applies partial action by only updating the prompt encoder parameters. This partial adaptation approach is sufficient for achieving task-specific performance improvements while consuming significantly fewer computational resources than would be required for complete model fine-tuning.