Serverless Call Routing With Reserved Capacity to Reduce Cold Starts
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing serverless computing systems experience significant latency due to 'cold starts' when provisioning environments for code execution, especially in non-steady traffic scenarios, which can inhibit adoption and increase computational load.
Innovation Solution
Implement a distributed on-demand code execution system with reserved capacity and a routing mechanism using consistent hash rings to route calls to pre-provisioned environments, employing per-account and per-task arcs to minimize cold starts and spillover handling.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a serverless computing system provisions environments dynamically based on traffic demand, then scalability is improved, but cold start latency increases significantly
Solution Approach 1:
The system pre-provisions execution environments and maintains them in a warm state before they are actually needed. Virtual machine instances are created and kept ready in advance, so when traffic arrives, the environments are already prepared and can execute code immediately without cold start delays. This preliminary preparation resolves the contradiction by having scalability ready in advance while eliminating the time penalty of dynamic provisioning.
Solution Approach 2:
The patent introduces a warming pool manager as an intermediary component that maintains a pool of pre-warmed virtual machine instances. This intermediary layer sits between the traffic demand and the code execution environment, providing a buffer of ready-to-use resources. The warming pool manager creates and maintains VM instances in advance, acting as a mediator that decouples the scalability requirement from the cold start problem by providing pre-prepared environments.
2Loss of time
If pre-provisioned environments are maintained to reduce cold starts, then cold start latency is reduced, but system complexity and resource overhead increase
Solution Approach 1:
The system dynamically adjusts the size and composition of the warming pool based on actual traffic patterns and demand. Rather than maintaining a fixed large pool of pre-provisioned environments, the warming pool manager creates and destroys VM instances dynamically, scaling the pre-warmed pool up when traffic increases and down when traffic decreases. This dynamic approach reduces system complexity by only maintaining necessary pre-provisioned resources rather than over-provisioning for all possible scenarios.
Solution Approach 2:
The patent changes the operational parameters of the execution environments by maintaining them in a warm state with pre-loaded code and dependencies, rather than starting from a cold state. The system modifies the lifecycle parameters of VM instances, keeping them active and ready with code already loaded, which transforms the traditional cold-start parameter into a warm-start parameter, reducing latency without requiring fundamentally new system architecture.
3Speed
If reserved capacity is allocated for specific tasks, then execution speed is improved, but scalability to handle varying traffic patterns is inhibited
Solution Approach 1:
The pre-provisioned virtual machine instances in the warming pool are designed to be multi-functional and can execute different types of code workloads. Rather than dedicating specific environments to specific tasks, the warming pool contains general-purpose VM instances that can be dynamically assigned to different code executions based on demand. This universal approach allows the system to maintain execution speed through pre-warmed environments while simultaneously scaling to handle varying traffic patterns and different task types.
Solution Approach 2:
The system dynamically assigns and reassigns pre-provisioned environments to different tasks based on real-time traffic patterns and workload requirements. The warming pool manager can flexibly allocate VM instances to different code executions, scaling the allocation of reserved capacity dynamically rather than maintaining fixed task-specific assignments. This dynamic resource allocation maintains execution speed by keeping environments warm while preserving scalability through flexible task assignment.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Systems and methods are described for reducing cold starts code within a serverless code execution system by providing a set of environments reserved for the code. A frontend utilizes a consistent hash ring to distribute calls for execution among a set of manager devices that manage environments in the system, distributed in a manner that groups calls together and attempts to distribute calls of the same group to a stable subset of the manager devices. Each group is assigned an arc, representing a subset of manager devices. When a call is received to execute a set of code that has environments reserved, the frontend distributes the call to a manager device of a reserved arc. When a call is received to execute a set of code that does not have environment reserved, the frontend distributes the call to an arc associated with an arc for unreserved executions.