Multi-Stage Distillation for Query Intent Classification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large language models face challenges in query intent classification due to short and context-lacking search queries, where retrieval augmentation improves accuracy but increases latency, leading to a trade-off between model performance and distillation efficacy.
Innovation Solution
A computing system and method that employs multi-stage distillation to train query processing models, using retrieval augmentation in the first stage with a small subset of data and then distilling the model into a smaller form using the full data set without augmentation, reducing latency and computing resources while retaining performance gains.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If retrieval augmentation is used to improve query understanding, then model accuracy is improved, but input latency increases
Solution Approach 1:
The patent segments the training process into multiple stages: first training with retrieval-augmented data on a subset, then distilling to a smaller model on the full dataset without retrieval augmentation. This segmentation allows the system to capture the benefits of retrieval augmentation during critical learning phases while avoiding its computational overhead during inference.
Solution Approach 2:
The patent performs preliminary retrieval augmentation during the training phase to teach the model contextual understanding, then removes the need for retrieval augmentation during inference. The model learns to internally represent contextual information that would otherwise require external retrieval, thereby eliminating the latency penalty at deployment.
2Reliability
If retrieval augmentation is applied to increase training data quality, then model performance is improved, but computing resources and complexity increase
Solution Approach 1:
The patent applies retrieval augmentation partially - only to a subset of the training data in the first stage - rather than to the entire dataset. This partial application provides sufficient contextual learning signal to improve model performance while avoiding the prohibitive computational cost of augmenting the full dataset.
Solution Approach 2:
The patent extracts and distills the essential knowledge gained from retrieval-augmented training into a compact smaller model. This extraction process removes the computational complexity of the retrieval augmentation system while retaining the performance benefits in the distilled model.
3Manufacturing precision
If a larger distillation set is used to train the model, then distillation efficacy is improved, but the benefits of retrieval augmentation are reduced due to increased latency
Solution Approach 1:
The patent segments the training data usage: a subset with retrieval augmentation for initial learning, then the full dataset without augmentation for distillation. This segmentation allows the model to learn contextual understanding from augmented examples while scaling to large datasets during the distillation phase without incurring retrieval latency.
Solution Approach 2:
The patent changes the parameter of retrieval augmentation application - using it for a subset of data during stage one, then not using it during stage two with the full dataset. This parameter change enables the system to achieve both high distillation efficacy and low inference latency by strategically controlling when augmentation is applied.
Data Source
AI summary
Provided are computing systems, methods, and platforms that train query processing models, such as large language models, to perform query intent classification tasks by using retrieval augmentation and multi-stage distillation. Unlabeled training examples of queries may be obtained, and a set of the training examples may be augmented with additional feature annotations to generate augmented training examples. A first query processing model may annotate the retrieval augmented queries to generate inferred labels for the augmented training examples. A second query processing model may be trained on the inferred labels, distilling the query processing model that was trained with retrieval augmentation into a non-retrieval augmented query processing model. The second query processing model may annotate the entire set of unlabeled training examples. Another stage of distillation may train a third query processing model using the entire set of unlabeled training examples without retrieval augmentation.