Distributed Fabric Layer for Consistent State Replication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing distributed computation models lack solutions for providing strong consistency guarantees and high availability in the presence of failures or load balancing, often relying on developers to manage restarts, termination, and failover, and typically require intermediate results to use files or specialized stores.
Innovation Solution
The solution involves linking individual compute instances to represent distributed computations, using stateful or stateless computations with input and output queues, and employing message acknowledgments and unique identifiers to mitigate input loss and duplication, while replicating state to ensure consistency and availability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If distributed computation models use files or specialized stores for intermediate results to achieve high availability guarantees, then reliability is improved, but device complexity increases
Solution Approach 1:
The patent introduces a fabric layer as an intermediary between compute nodes and storage resources. This fabric layer abstracts the complexity of data persistence and recovery, providing high availability guarantees without requiring complex distributed computation models. The fabric layer handles message buffering, state replication, and fault tolerance transparently, allowing compute nodes to focus on computation while the fabric ensures reliability.
2Reliability
If distributed computation models require developers to manage restarts, termination, and failover, then ease of operation deteriorates, but reliability improves
Solution Approach 1:
The patent implements self-service mechanisms where the distributed computation system automatically manages restarts, termination, and failover without developer intervention. The fabric layer monitors compute node health, detects failures, and triggers automated recovery procedures including restarting failed computations and redistributing workloads. This eliminates the burden of manual fault management while maintaining strong consistency guarantees through automated state recovery and replication.
3Reliability
If state is replicated at the rendezvous point to ensure consistency when nodes fault or leave, then reliability is improved, but device complexity increases
Solution Approach 1:
The patent introduces a fabric layer as an intermediary between compute nodes and the rendezvous point, abstracting away the complexity of state replication. The fabric layer handles state synchronization, replication, and consistency maintenance transparently, allowing the rendezvous point to focus on service coordination without managing complex replication protocols. This reduces rendezvous point complexity while maintaining strong consistency guarantees through the fabric's automated state management.
Data Source
AI summary
The present invention extends to methods, systems, and computer program products for performing computations in a distributed infrastructure. Embodiments of the invention include a general purpose distributed computation infrastructure that can be used to perform efficient (in-memory), scalable, failure-resilient, atomic, flow-controlled, long-running state-less and state-full distributed computations. Guarantees provided by a distributed computation infrastructure can build upon existent guarantees of an underlying distributed fabric in order to hide the complexities of fault-tolerance, enable large scale highly available processing, allow for efficient resource utilization, and facilitate generic development of stateful and stateless computations. A distributed computation infrastructure can also provide a substrate on which existent distributed computation models can be enhanced to become failure-resilient.