Pre-initialized Virtual Machine Pool for Low Latency Code Execution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Developers face challenges in managing virtual machine instances for low-latency execution of user codes, including high startup delays, over- or under-utilization of resources, and unpredictable traffic scaling, which can lead to performance issues and increased costs.
Innovation Solution
A virtual compute system maintains a pool of pre-initialized virtual machine instances with loaded software components, allowing for immediate use upon user requests, and employs a message queue or bus to facilitate event message propagation and automatic scaling based on demand.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Use of energy by moving object
If virtual machine instances are initialized on-demand when user requests are received, then resource utilization is improved, but startup latency increases
Solution Approach 1:
The system pre-initializes virtual machine instances before they are needed by user requests. The frontend pre-creates virtual machine instances and loads software components (operating systems, language runtimes, libraries) onto them in advance, so that when a user request arrives, the instance is already ready to execute code immediately, eliminating startup latency while maintaining resource availability
2Speed
If more virtual machine instances are maintained in the pool, then execution speed is improved, but resource cost increases
Solution Approach 1:
The system dynamically adjusts the size of the virtual machine instance pool based on actual demand. The frontend monitors user requests and automatically scales the number of pre-initialized instances up or down, ensuring that enough instances are available for immediate execution while avoiding over-provisioning that would waste resources. This dynamic scaling resolves the contradiction between maintaining fast execution speed and controlling resource costs
3Productivity
If virtual machine instances are pre-configured with software components, then execution readiness is improved, but device complexity increases
Solution Approach 1:
The system segments the virtual machine instance management into separate functional components: instance creation, software component loading, user request handling, and code execution. The frontend divides the pre-initialization process into distinct stages (creating virtual machine instances, loading operating systems, loading language runtimes, loading libraries) that can be managed independently. This segmentation reduces the perceived complexity by organizing the configuration process into manageable, modular steps
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A service manages a plurality of virtual machine instances for low latency execution of user codes. The service can provide the capability to execute user code in response to events triggered on an auxiliary service to provide implicit and automatic rate matching and scaling between events being triggered on the auxiliary service and the corresponding execution of user code on various virtual machine instances. An auxiliary service may be configured as an event triggering service to detect events and generate event messages for execution of the user codes. The service can request, receive, or poll for event messages directly from the auxiliary service or via an intermediary message service. Event messages can be rapidly converted to requests to execute user code on the service. The time from processing the event message to initiating a request to begin code execution is less than a predetermined duration, for example, 100 ms.