Dynamic Batch Configuration for Serverless Inference Memory Bottlenecks

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Current serverless inference systems face a bottleneck due to high memory consumption, which increases data processing delays, especially when server configurations are low.

Innovation Solution

A method for data processing that dynamically adjusts configuration combinations to optimize memory usage without increasing processing delays. This involves determining data processing periods under various configuration combinations, selecting a target configuration that maximizes data throughput, and creating processing processes accordingly.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Quantity of substance

If batch processing of requests is used to reduce memory occupation, then server memory usage is reduced, but data processing delay increases

Engineering Contradiction:

Improveserver memory usageVSAvoiddata processing delay

Core Design Contradiction:

Quantity of substanceVSLoss of time

Solution Approach 1:

The patent dynamically adjusts the batch size based on real-time system state and data characteristics. The batch processing configuration is not fixed but adapts during operation, allowing the system to optimize between memory usage and processing speed by changing batch parameters dynamically rather than using a static batch size

Inventive Principle:

Principle #15Dynamics

Solution Approach 2:

The system changes key processing parameters such as batch size, memory allocation, and processing configuration based on the type and characteristics of incoming data. By adjusting these parameters, the system can process different data types with optimal memory efficiency while minimizing additional processing delays

Inventive Principle:

Principle #35Parameter changes

2Quantity of substance

If batch processing is used when server configuration is low, then memory occupation is reduced, but request queuing time increases significantly

Engineering Contradiction:

Improvememory occupationVSAvoidrequest queuing time

Core Design Contradiction:

Quantity of substanceVSLoss of time

Solution Approach 1:

For servers with low configuration, the system dynamically determines whether to use batch processing or individual processing based on real-time conditions. When the system detects that batch processing would cause excessive queuing time, it automatically switches to individual processing mode, and vice versa, ensuring optimal performance for the given hardware constraints

Inventive Principle:

Principle #15Dynamics

Solution Approach 2:

The patent creates processing processes on-demand rather than maintaining persistent expensive processing instances. Each batch or individual request can trigger a temporary processing process that is created, executed, and then released, reducing the need for continuously running high-memory processes on low-configuration servers

Inventive Principle:

Principle #27Cheap short-living objects (Disposable)

Data Source

PatentUS20250086022A1Methods and apparatuses for data processing, storage media, and electronic devices

Publication Date: 2025.03.13 ZHEJIANG LAB

AI summary

A method for data processing is provided, and includes: obtaining each piece of to-be-processed data, determining whether a set amount of the to-be-processed data is capable to be processed under a current processing process by a data processing model, if not, obtaining data processing periods of the data processing model under multiple configuration combinations; for a data processing period of each of the multiple configuration combinations, determining an amount of data that is capable to be processed by the data processing model within the data processing period, as a target data amount; by taking the data processing model to be capable to process the set amount of the to-be-processed data as a target, according to the target data amount for a data processing period of each of the multiple configuration combinations, selecting a target configuration combination from the multiple configuration combinations.