Fractional Inference on GPU and CPU for Transformer Models
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The high demand for GPU resources in transformer-based language models leads to infrastructure and latency constraints, as GPUs are expensive and have limited availability, causing delays in processing requests and limiting the use of these models.
Innovation Solution
A hybrid approach using a transformer-based model on a GPU and a lightweight task-specific model on a CPU, where intermediate outputs from the GPU are stored temporarily and processed by the CPU, shifting workload and reducing latency and infrastructure constraints.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If transformer-based language models are deployed using GPU resources, then model performance and processing capability are improved, but infrastructure cost and resource availability are worsened due to high expenses and limited GPU availability
Solution Approach 1:
The patent divides the transformer model into two separate models: a first model that processes input data and generates intermediate outputs, and a second model that processes these intermediate outputs to generate final predictions. This segmentation allows the computationally intensive first model to run on GPU while the second model runs on CPU, reducing infrastructure constraints and costs while maintaining overall model performance.
2Speed
If all transformer model processing is performed on GPU, then processing speed is improved, but resource availability and cost efficiency are worsened due to limited GPU availability and high expenses
Solution Approach 1:
The patent introduces an intermediary component (the first model running on GPU) that processes input data and generates intermediate outputs. These intermediate outputs are then passed to a second model running on CPU for final processing. This intermediary approach allows the system to leverage GPU acceleration where most needed while utilizing available CPU resources, improving cost efficiency and resource utilization.
3Measurement precision
If task-specific fine-tuning is performed on transformer models, then model accuracy for specific tasks is improved, but catastrophic forgetting is worsened where the model loses general language understanding capabilities
Solution Approach 1:
The patent segments the model architecture into a first model that maintains general language understanding capabilities and a second model that specializes in task-specific processing. By training these models separately with different datasets and objectives, the system achieves task-specific accuracy while preserving general language understanding in the first model, preventing catastrophic forgetting.
Data Source
AI summary
Systems and methods for fractional inference on GPU and CPU for large scale deployment of customized transformers based language models are disclosed herein. The method can include, receiving data for use in generation of a machine learning model output, ingesting the data with a first machine learning model on a Graphic Processing Unit, receiving at least one intermediate output from the first machine learning model at a temporary store, receiving the at least one intermediate output from the temporary store at a Central Processing Unit, ingesting the at least one intermediate output with a second machine learning model on the Central Processing Unit, and outputting a prediction with the second machine learning model.