Failure Isolation via Request Shedding in Distributed Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed systems, a small degradation in service dependency can cause a magnified impact at upstream services due to the one-to-many relationship of upstream operations to downstream tasks, leading to increased failure rates and resource inefficiencies.
Innovation Solution
Implementing intelligent request shedding by associating multiple requests with a common key, partitioning the keyspace into 'drop' and 'keep' partitions based on the amount of traffic to be shed, and mapping keys to these partitions to deliberately discard requests, thereby limiting higher-level failures to match the failure rate at the downstream service.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If upstream services continue all requests to downstream services during degradation, then service completeness is maintained, but failure impact is magnified and reliability decreases
Solution Approach 1:
The patent extracts problematic requests from the request stream by identifying and shedding requests associated with degraded downstream services. The failure isolation component monitors service health and selectively removes requests that would propagate failures, keeping only healthy requests in the request stream.
Solution Approach 2:
The system changes the state of requests by marking them as healthy or unhealthy based on downstream service status. When service degradation is detected, the system alters request parameters including shedding unhealthy requests and modifying retry behavior, transforming the request handling strategy dynamically.
2Reliability
If requests are shed to limit failure propagation, then reliability is improved, but service completeness deteriorates
Solution Approach 1:
The patent applies different quality levels to different requests based on their association with degraded services. Healthy requests maintain full service quality and are processed normally, while unhealthy requests associated with degraded services are shed or modified, creating local quality differentiation in request handling.
Solution Approach 2:
Instead of shedding all requests to a degraded service, the system performs partial action by selectively shedding only unhealthy requests while maintaining healthy requests. This partial shedding approach limits failure propagation without completely blocking service access, balancing reliability and productivity.
3Reliability
If all failed requests are retried, then service completeness is maintained, but resource efficiency deteriorates due to cascading failures
Solution Approach 1:
The system implements feedback mechanisms where the failure isolation component continuously monitors downstream service health and adjusts retry behavior accordingly. When degradation is detected, the system receives feedback about unhealthy requests and modifies retry strategies, preventing resource waste on doomed requests while maintaining retries for healthy ones.
Solution Approach 2:
The patent performs preliminary action by identifying and shedding unhealthy requests before they are retried. The failure isolation component proactively removes requests associated with degraded services from the retry queue, preventing wasted computational resources on requests that would inevitably fail again.
4Measurement precision
If the system monitors all service dependencies closely, then failure detection precision is improved, but system complexity increases
Solution Approach 1:
The patent introduces a failure isolation component as an intermediary between upstream and downstream services. This mediator monitors service health and manages request shedding, providing precise failure detection without requiring complex monitoring infrastructure across the entire distributed system.
Data Source
AI summary
Methods, systems, and computer-readable media for failure isolation in a distributed system are disclosed. An operation is initiated at a first service in a distributed system. The operation is dependent on a plurality of requests to one or more services including a second service, and the requests are associated with a key. A keyspace comprising an ordered set of potential key values is partitioned into a first partition and a second partition, where the first partition represents potential key values to be subjected to deliberate request shedding. The key associated with the requests is mapped to the first partition. The requests to the second service are discarded based at least in part on the mapping, and the operation associated with the key fails at the first service.