Dynamic Concurrency Limits in On-Demand Code Execution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Developers face challenges in efficiently managing and scaling virtual machine instances for on-demand code execution, including deciding on the number and type of instances, handling costs, and addressing issues like over- or under-utilization, as well as instance startup delays, which can impact performance and efficiency.
Innovation Solution
An on-demand code execution environment that automatically adjusts the maximum number of concurrent code executions, distributes code executions across virtual machines based on user entities, and uses a pre-warmed pool of virtual machine instances to enable rapid execution with minimal delay, incorporating a load balancer, frontend, warming pool manager, and worker manager ring to optimize resource utilization and scalability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If a fixed limit on concurrent code executions is imposed, then resource utilization is controlled and costs are managed, but system flexibility and ability to handle variable workloads are reduced
Solution Approach 1:
The patent implements dynamic adjustment of the maximum number of concurrent code executions based on current system conditions. The limit is not fixed but varies over time according to resource availability, workload characteristics, and system state, allowing the system to adapt between conserving resources during low-demand periods and handling high workloads during peak periods
2Loss of energy
If virtual machine instances are started on-demand, then resource usage is optimized and costs are reduced, but startup delays impact execution speed and performance
Solution Approach 1:
The patent pre-starts and maintains a pool of virtual machine instances in a ready state before they are needed for code execution. This preliminary action ensures that when code execution requests arrive, pre-configured instances are immediately available to execute the code without waiting for startup, thereby eliminating startup delays while maintaining resource efficiency through selective pre-warming of instances
Solution Approach 2:
The patent maintains a continuous pool of pre-warmed virtual machine instances that remain in a ready state, ensuring uninterrupted code execution capability. By keeping instances continuously prepared rather than starting them from scratch each time, the system eliminates idle time and maintains continuous productive action
3Productivity
If the number of concurrent code executions is increased, then system productivity and throughput are improved, but resource management complexity and cost control become more difficult
Solution Approach 1:
The patent implements a feedback mechanism that continuously monitors system resource usage, workload characteristics, and execution performance. Based on this feedback, the system automatically adjusts the maximum number of concurrent code executions and the size of the pre-warmed instance pool, enabling intelligent resource management that balances productivity gains with complexity and cost control
Data Source
AI summary
Systems and methods are described for adjusting a number of concurrent code executions allowed to be performed for a given user on an on-demand code execution environment or other distributed code execution environments. Such environments utilize pre-initialized virtual machine instances to enable execution of user-specified code in a rapid manner, without delays typically caused by initialization of the virtual machine instances. However, to improve utilization of computing resources, such environments may temporarily restrict the number of concurrent code executions performed on behalf of the given user to a number less than the maximum number of concurrent code executions allowed for the given user. Such environments may adjust the temporary restriction on the number of concurrent code executions based on the number of incoming code execution requests associated with the given user.