Fractional Replication for Cloud Tail Latency

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Cloud computing systems face challenges in meeting tail latency targets due to high resource provisioning costs and inefficiencies, as increasing the number of virtual machines may not always suffice to meet performance targets.

Innovation Solution

Implementing a combination of resource redundancy and workload redundancy using a fractional replication factor, where requests are replicated and distributed across multiple virtual machines, with proactive and speculative control to adjust resource allocation based on latency performance.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If a large number of virtual machines are provisioned to meet tail latency targets, then performance reliability is improved, but resource cost increases and resource efficiency deteriorates

Engineering Contradiction:

Improvetail latency target meetingVSAvoidnumber of virtual machines

Core Design Contradiction:

ReliabilityVSQuantity of substance

Solution Approach 1:

The system performs preliminary actions by proactively replicating requests before their actual execution. The replication factor is determined in advance based on historical performance data and predicted system state, allowing the system to prepare multiple copies of requests that can be executed if the original request encounters delays, thereby ensuring tail latency targets are met without needing to provision excessive virtual machines

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system dynamically changes the replication factor parameter based on predicted system conditions and historical performance. By adjusting this parameter according to workload characteristics, resource availability, and performance targets, the system optimizes the balance between meeting reliability requirements and minimizing resource consumption

Inventive Principle:

Principle #35Parameter changes

2Productivity

If the number of virtual machines is increased to improve performance, then processing capacity is improved, but resource utilization decreases

Engineering Contradiction:

Improveprocessing capacityVSAvoidresource utilization efficiency

Core Design Contradiction:

ProductivityVSLoss of energy

Solution Approach 1:

Instead of provisioning additional virtual machines to handle potential latency issues, the system creates copies of requests and distributes them across existing virtual machines. This approach increases processing capacity through request replication rather than resource multiplication, maintaining high resource utilization while improving performance reliability

Inventive Principle:

Principle #26Copying

Solution Approach 2:

Existing virtual machines are made multi-functional by handling both original requests and replicated requests. This universal utilization of resources maximizes the productivity of each virtual machine, allowing the same infrastructure to serve multiple purposes and improve overall processing capacity without additional hardware costs

Inventive Principle:

Principle #6Universality (Multi-functionality)

3Reliability

If workload redundancy is increased by replicating requests, then tail latency is reduced, but system complexity increases

Engineering Contradiction:

Improvetail latencyVSAvoidsystem complexity

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The system uses feedback mechanisms to dynamically adjust the replication factor based on observed performance and predicted system state. Historical performance data is continuously analyzed, and the replication factor is modified in response to feedback from the system's actual behavior, allowing the system to reduce tail latency while adapting to changing conditions and avoiding unnecessary complexity

Inventive Principle:

Principle #23Feedback

Data Source

PatentUS11455197B2Optimizing tail latency via workload and resource redundancy in cloud

Publication Date: 2022.09.27 INTERNATIONAL BUSINESS MACHINE CORPORATION

AI summary

A plurality of requests are received for computing processing. At least some of the plurality of requests are replicated. The requests are replicated based on a fractional replication factor. Each received request and each replicated request are transmitted to a computer resource for processing. At least some embodiments provide the capability for meeting tail latency targets with improved performance and reduced cost.