Concurrent Code Execution Limits in On-Demand Environments
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Developers face challenges in efficiently managing virtual machine instances for cloud computing, including deciding on the number and type of instances to purchase, predicting traffic changes, and dealing with instance and language runtime startup delays, which can lead to over- or under-utilization of resources and increased costs.
Innovation Solution
An on-demand code execution environment that automatically adjusts the number of concurrent code executions and distributes them across virtual machines based on user needs, using a load balancer, frontend, warming pool manager, and worker manager ring to enable rapid execution of code without initial delays, by pre-warming virtual machine instances and dynamically scaling resources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If virtual machine instances are pre-configured and kept ready, then code execution speed is improved, but resource utilization deteriorates due to over-provisioning
Solution Approach 1:
The system performs preliminary actions by pre-warming virtual machine instances before they are needed for code execution. The worker managers maintain a pool of pre-initialized VM instances that can immediately execute code when requested, eliminating startup delays. This resolves the contradiction by having instances ready in advance (improving speed) while only activating them when actually needed (maintaining resource efficiency).
Solution Approach 2:
The system dynamically adjusts the number and state of virtual machine instances based on real-time demand. Worker managers can scale the VM pool up or down, transitioning instances between active and standby states as needed. This dynamic adaptation allows the system to maintain execution speed when demand is high while conserving resources when demand is low, resolving the utilization-speed tradeoff.
2Productivity
If the number of worker managers is increased to handle more concurrent executions, then productivity is improved, but system complexity worsens
Solution Approach 1:
The system segments the worker manager functionality into multiple independent instances that can operate in parallel. Each worker manager handles a portion of the concurrent execution load, and they can be distributed across different servers. This segmentation increases productivity by distributing the workload while managing complexity through modular, independent units rather than a monolithic system.
Solution Approach 2:
The system introduces intermediary components such as load balancers and coordination services that manage communication between multiple worker managers. These intermediaries abstract the complexity of coordinating multiple instances, allowing worker managers to scale independently while maintaining system coherence. The intermediary layer handles the complexity of inter-instance communication, enabling productivity improvement without proportional complexity increase.
3Loss of energy
If virtual machine instances are initialized on-demand, then resource utilization is improved, but execution delay worsens due to startup time
Solution Approach 1:
The system performs preliminary initialization of virtual machine instances in advance, creating a warm pool of ready-to-execute instances. When code execution is requested, pre-warmed instances are immediately allocated rather than initializing new instances from scratch. This eliminates the startup delay while maintaining resource efficiency by only pre-warming instances that are likely to be needed based on demand patterns.
Solution Approach 2:
The system dynamically manages the lifecycle of virtual machine instances, transitioning them between different states (created, warmed, active, standby, terminated) based on real-time demand. Instances are warmed up in advance when demand is predicted, and scaled back when demand decreases. This dynamic state management resolves the contradiction by having instances ready when needed (reducing delay) while allowing termination when not needed (improving resource utilization).
Data Source
AI summary
Systems and methods are described for adjusting a number of concurrent code executions allowed to be performed for a given user on an on-demand code execution environment or other distributed code execution environments. Such environments utilize pre-initialized virtual machine instances to enable execution of user-specified code in a rapid manner, without delays typically caused by initialization of the virtual machine instances. However, to improve utilization of computing resources, such environments may temporarily restrict the number of concurrent code executions performed on behalf of the given user to a number less than the maximum number of concurrent code executions allowed for the given user. Such environments may adjust the temporary restriction on the number of concurrent code executions based on the number of incoming code execution requests associated with the given user and based on communication among the frontends processing the incoming code execution requests.