Request Segmentation for Reduced Memory Consumption in Sequential Models
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large transformer-based machine learning models consume significant memory, leading to inefficient GPU resource utilization and increased operational costs due to conservative memory reservations, which result in unnecessary resource consumption and reduced throughput in cloud-based services.
Innovation Solution
Implementing a request segmentation technique that breaks down processing requests into multiple partial requests with reduced output size parameters, allowing the model to incrementally reserve memory only as needed, thereby reducing peak memory consumption and improving resource utilization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conservative memory reservations are used to ensure sufficient memory for large transformer models, then model reliability is improved, but memory consumption increases and GPU resource utilization becomes inefficient
Solution Approach 1:
The patent segments a single large processing request into multiple smaller partial processing requests. Each partial request specifies a smaller output size parameter, allowing the model to reserve memory incrementally for each partial request rather than reserving all memory upfront for the complete output. This segmentation approach reduces peak memory consumption while ensuring sufficient memory is available for each processing step, thereby resolving the contradiction between reliability and memory consumption.
2Reliability
If conservative memory reservations are used for large transformer models, then sufficient memory is guaranteed, but operational costs increase due to inefficient resource utilization
Solution Approach 1:
By dividing the processing into multiple partial requests with smaller output size parameters, the system reduces the total memory that needs to be reserved and paid for. Each partial request consumes memory only for its specific portion, avoiding the waste of reserving and paying for memory that would remain unused if a single large request were processed. This directly reduces operational costs while maintaining sufficient memory guarantees for each processing step.
3Device complexity
If single large processing requests are submitted, then throughput is simplified to manage, but request timeouts and transport issues increase
Solution Approach 1:
The patent breaks down single large processing requests into multiple smaller partial processing requests. Each partial request has a smaller output size parameter, which reduces the likelihood of timeouts and transport issues during transmission and processing. The system manages these partial requests sequentially, maintaining relatively simple request management while significantly improving reliability by avoiding the pitfalls of handling single large requests.
Data Source
AI summary
A disclosed method reduces memory consumption of a trained sequential model. The method includes receiving, from a client application, an initial processing request identifying an input sequence to be processed by the trained sequential model and an initial value for an output size parameter specifying a requested size of output from the trained sequential model. The method further includes sequentially transmitting, to the trained sequential model, multiple partial processing requests based on the initial processing request that each specify a fraction of the initial value as the output size parameter and receiving a sequence of output responses from the trained sequential model generated in response to processing the multiple partial processing requests. The method further provides for returning, to the client application, a final merged response that includes the sequence of output responses.