Preloading On-Demand Code in Cloud Networks

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Current cloud computing platforms face inefficiencies due to cold-start delays and high context-switching overheads when handling on-demand code, leading to performance lags and increased costs, especially when scaling to meet demand spikes or serving global user bases.

Innovation Solution

Implementing a distributed cloud computing network that preloads on-demand code pieces in isolated execution environments, allowing multiple code instances to run within a single process, reducing cold-start delays and context-switching overheads by preloading code based on anticipated requests through TLS handshakes or DNS lookups.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If containerized processes are spun up for on-demand code execution, then code isolation and resource efficiency are improved, but cold-start delays occur causing request hanging for 500ms to 10 seconds

Engineering Contradiction:

Improvecode execution efficiencyVSAvoidcold-start delay

Core Design Contradiction:

ProductivityVSLoss of time

Solution Approach 1:

The system performs preliminary actions by preloading code into isolated execution environments before actual requests arrive. Code is loaded into memory in advance based on predicted demand patterns, so when requests come in, the execution environment is already ready, eliminating cold-start delays of 500ms to 10 seconds.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system dynamically adjusts execution environment management by monitoring request patterns and automatically loading code into isolated environments only when needed. This dynamic approach balances the trade-off between keeping code ready (reducing cold-start) and avoiding unnecessary resource consumption.

Inventive Principle:

Principle #15Dynamics

2Productivity

If multiple containerized processes are run concurrently to handle demand spikes, then request handling capacity is improved, but context-switching overhead increases consuming 100 microseconds per process

Engineering Contradiction:

Improverequest handling capacityVSAvoidcontext-switching overhead

Core Design Contradiction:

ProductivityVSLoss of energy

Solution Approach 1:

The system segments code execution into isolated execution environments that are separate from the main process. Each isolated environment runs code independently without requiring frequent context-switches between processes, reducing the 100 microsecond overhead per process while still allowing concurrent request handling.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The system merges multiple code execution contexts within a single process by using isolated execution environments. This allows multiple concurrent request handlers to share the same process space, eliminating the need for expensive process-level context-switches while maintaining code isolation and concurrent processing capability.

Inventive Principle:

Principle #5Merging (Combining)

3Speed

If code is permanently loaded in memory to avoid cold-start, then execution speed is improved, but memory constraints are violated as each application requires minimum memory for compiled code and program state

Engineering Contradiction:

Improvecode execution speedVSAvoidmemory consumption

Core Design Contradiction:

SpeedVSQuantity of substance

Solution Approach 1:

The system performs preliminary code loading into isolated execution environments only when predicted to be needed, rather than permanently loading all code. This selective preloading ensures fast execution when code is active while minimizing memory consumption during idle periods.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system changes the state of code from completely unloaded to partially loaded (preloaded) based on demand predictions. Code is loaded into isolated environments in advance when requests are predicted, creating a middle state that balances memory usage and execution speed without requiring permanent memory allocation.

Inventive Principle:

Principle #35Parameter changes

4Productivity

If developers specify exact node locations for code execution, then resource optimization is improved, but system flexibility and ease of deployment are reduced

Engineering Contradiction:

Improveresource optimizationVSAvoiddeployment flexibility

Core Design Contradiction:

ProductivityVSEase of operation

Solution Approach 1:

The system provides a universal deployment interface where developers can specify code without needing to select specific node locations. The system automatically handles code distribution to appropriate isolated execution environments across the cloud infrastructure, maintaining resource optimization while greatly simplifying the deployment process.

Inventive Principle:

Principle #6Universality (Multi-functionality)

Data Source

PatentUS11621995B2Preloading on-demand code pieces in a distributed cloud computing network

Publication Date: 2023.04.04 CLOUDFLARE INC

AI summary

A compute server of a distributed cloud computing network receives an indication of a hostname in a Transport Layer Security (TLS) handshake message from a client device. The compute server determines that an on-demand code piece is configured to execute on the compute server responsive to receiving an HTTP request for the hostname. Prior to receiving the request for the hostname, the compute server preloads the on-demand code piece for execution. The compute server receives an HTTP request from the client device that triggers execution of the preloaded on-demand code piece. The compute server executed the preloaded on-demand code piece and generates a response to the HTTP request based in part on the executed code piece. The compute server transmits the response to the client device.