Elastic Transformer Serving via Token Adaptation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing cloud-based transformer model serving systems face challenges in efficiently accommodating dynamic query loads and heterogeneous user requirements due to the resource-intensive nature of transformer models, leading to high training costs and latency issues.
Innovation Solution
An elastic transformer serving system via token adaptation, which dynamically adjusts the execution tokens of the transformer model through token prompting and token merging processes based on query characteristics and fluctuating query loads, optimizing inference result accuracy and latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If model adaptation is used to improve service quality for different user requirements, then service quality is improved, but training costs and time overhead increase significantly
Solution Approach 1:
The patent uses prompt templates as lightweight copies that encode task-specific knowledge without requiring full model retraining. Instead of adapting the entire transformer model for each task, the system creates prompt template copies that can be dynamically applied, reducing training time from days to minutes while maintaining service quality.
Solution Approach 2:
The patent changes the approach from modifying model parameters (weights) through training to modifying prompt parameters (template configurations). By adjusting prompt templates rather than model weights, the system achieves task adaptation with minimal computational overhead and no extensive retraining required.
2Adaptability or versatility
If multiple model variants with different sizes are trained to accommodate varying service demands, then adaptability is improved, but device complexity and resource requirements increase
Solution Approach 1:
The patent makes a single transformer model universal by equipping it with multiple prompt templates that can handle different task types and service demands. Instead of maintaining multiple specialized model variants, the system uses one model that can adapt to various tasks through prompt template selection, significantly reducing device complexity while maintaining versatility.
Solution Approach 2:
The patent introduces dynamic prompt template selection that adapts to varying service demands in real-time. The system dynamically chooses appropriate prompt templates based on the incoming query characteristics, eliminating the need for static model variant selection and reducing the complexity of managing multiple model versions.
3Reliability
If large transformer models are loaded to processing units for execution, then model capability is improved, but latency increases due to loading time
Solution Approach 1:
The patent performs preliminary action by pre-processing tasks into prompt templates before model execution. By converting tasks into structured prompt templates in advance and caching them, the system eliminates the need for complex model reconfiguration during execution, reducing latency while maintaining model capability.
4Measurement precision
If token number is increased to improve inference accuracy, then accuracy is improved, but processing latency increases
Solution Approach 1:
The patent applies local quality by concentrating computational resources on critical tokens identified through attention mechanisms. Instead of uniformly processing all tokens with high detail, the system focuses computational effort on locally important tokens while using prompt templates to guide attention, achieving high accuracy without proportionally increasing latency.
Data Source
AI summary
An elastic transformer serving system, referred to herein as an online token adaptation system (OTAS), is described that accommodates diverse user requests with fluctuating query louds while optimizing output accuracy and runtime latency. The OTAS uses a token adaptation technique that involves adding prompting tokens to improve accuracy and removing redundant tokens to accelerate inference. To cope with fluctuating query loads and diverse user requests, the OTAS further uses application-aware selective batching in combination with online token adaptation. In an example embodiment, the OTAS first batches incoming queries with similar service-level objects to improve the ingress throughput. Then, to strike a trade-off between the overhead of token increment and the potential for accuracy improvement, the OTAS adaptively adjusts the token execution settings by solving an optimization problem.