DNN Request Batching by Padding-Cost Selection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing solutions for deploying Deep Neural Networks (DNNs) face challenges in efficiently processing requests due to varying request lengths and computational costs, leading to suboptimal throughput and processing times.
Innovation Solution
A batching algorithm that selectively adds requests to batches based on cost values indicative of padding tokens needed to ensure all requests in a batch have the same length, optimizing processing efficiency by reducing the average processing time of requests and batches.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If requests are processed individually without batching, then processing simplicity is maintained, but throughput and resource utilization deteriorate
Solution Approach 1:
The patent applies preliminary action by pre-calculating cost values for potential batch compositions before final processing. The system evaluates multiple candidate requests and their associated padding costs in advance, selecting the optimal batch configuration that minimizes processing time while maintaining simplicity in the actual execution phase.
2Productivity
If requests of varying lengths are processed, then adaptability to different data sizes is improved, but processing efficiency deteriorates due to padding requirements
Solution Approach 1:
The patent applies parameter changes by dynamically adjusting the batch composition based on request length parameters. The system calculates cost values that reflect the padding required for each potential batch configuration and selects the configuration that optimizes processing efficiency while accommodating varying request lengths. This involves changing the batch size and composition parameters adaptively rather than using fixed parameters.
3Productivity
If batching is implemented to improve throughput, then resource utilization is improved, but computational overhead for batch management increases
Solution Approach 1:
The patent applies partial action by evaluating only the necessary subset of candidate requests for batch formation rather than exhaustively analyzing all possible combinations. The system calculates cost values for promising candidates and selects sufficient (but not excessive) batch configurations to achieve optimal resource utilization, avoiding unnecessary computational overhead while maintaining high throughput.
Data Source
AI summary
Methods and electronic devices for batching requests for processing by a processing unit are disclosed. The method includes acquiring a queue of requests to be processed and executing a current iteration for a given batch of requests including the given request from the queue. During the current iteration, the method includes determining a first cost value of adding a first candidate request from the queue to the given batch, determining a second cost value of adding a second candidate request from the queue to the given batch, and selectively adding a target request amongst the first candidate request and the second candidate request to the given batch. The target request being associated with a lowest one of the first cost value and the second cost value.