Serverless Call Distribution via Consistent Hash Ring
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Serverless computing systems face significant latency issues due to 'cold starts' when pre-provisioned execution environments are not available, particularly in non-steady traffic scenarios, leading to delayed code execution and potential adoption barriers.
Innovation Solution
Implementing an on-demand code execution system with reserved, pre-warmed capacity and a consistent hash ring routing mechanism to route calls efficiently, ensuring high probability of executing tasks in pre-configured environments, and handling spillover calls by maintaining warm environments even during capacity reductions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If pre-provisioned execution environments are maintained to avoid cold starts, then execution speed is improved, but system cost and resource consumption increase
Solution Approach 1:
The system performs preliminary actions by pre-provisioning and warming up execution environments before they are needed. Execution environments are created and kept in a ready state in advance, so when a function invocation occurs, the code can execute immediately without cold start delays. This resolves the contradiction by having resources prepared beforehand rather than provisioning them on-demand.
Solution Approach 2:
The system dynamically adjusts the number and state of execution environments based on actual usage patterns. Rather than maintaining a static pool of pre-warmed environments, the system scales the execution environment pool dynamically, warming up environments as needed and scaling down when not in use. This allows the system to optimize between execution speed and resource consumption in real-time.
2Reliability
If reserved capacity is allocated to specific functions, then execution reliability is improved, but system scalability is reduced
Solution Approach 1:
The system creates execution environments that are universal and can serve multiple functions. Rather than dedicating specific execution environments to specific functions (which would limit scalability), the warmed-up execution environments are made available to any function that needs them. The system uses function identifiers and routing mechanisms to direct invocations to appropriate execution environments, allowing the same pool of environments to serve multiple purposes and scale flexibly.
Solution Approach 2:
The system segments the execution environment pool and invocation routing based on function identifiers. By hashing function identifiers and using consistent hashing techniques, the system divides the workload across multiple execution environments in a scalable manner. This segmentation allows reliable routing of specific functions to appropriate execution environments while maintaining overall system scalability through the modular, distributed architecture.
3Loss of time
If execution environments are warmed up in advance, then latency is reduced, but device complexity increases
Solution Approach 1:
The system implements self-service mechanisms where execution environments automatically warm up themselves based on usage patterns and system state. Rather than requiring complex external orchestration to manage which environments should be warmed, the system uses automated detection and triggering mechanisms that cause execution environments to prepare themselves in advance. This reduces the operational complexity while maintaining the latency benefits of pre-warming.
Solution Approach 2:
The system uses feedback loops to monitor execution environment states and usage patterns, automatically adjusting which environments should be warmed up. By continuously monitoring invocation patterns, execution times, and resource availability, the system provides feedback to the provisioning mechanism, which then adjusts the warming strategy accordingly. This feedback-driven approach reduces complexity by using simple monitoring and automatic adjustment rather than complex manual orchestration.
Data Source
AI summary
Systems and methods are described for reducing cold starts code within a serverless code execution system by providing a set of environments reserved for the code. A frontend utilizes a consistent hash ring to distribute calls for execution among a set of manager devices that manage environments in the system, distributed in a manner that groups calls together and attempts to distribute calls of the same group to a stable subset of the manager devices. Each group is assigned an arc, representing a subset of manager devices. When a call is received to execute a set of code that has environments reserved, the frontend distributes the call to a manager device of a reserved arc. When a call is received to execute a set of code that does not have environment reserved, the frontend distributes the call to an arc associated with an arc for unreserved executions.