Application Live-Migration and Automated Recovery in Distributed Clusters
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current solutions for managing multiple applications on a cluster of servers fail to effectively address redundancy and load-balancing issues in high-latency networked systems, particularly due to single points of failure, performance overhead, geographical limitations, and manual intervention requirements.
Innovation Solution
A server configured to operate in either a master or slave role, with logic for replicating data across multiple servers and autonomously switching roles based on events such as failures or load changes, using snapshot replication and heartbeats to maintain data consistency and balance load across the cluster.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If virtualisation is used to replicate server state to a second physical host, then application recovery capability is improved, but performance overhead increases and geographical flexibility is reduced
Solution Approach 1:
The system segments the replication process by separating the master server that generates data changes from slave servers that receive and store replicas. This segmentation allows the master server to continue operating at full performance while replication occurs asynchronously in the background, eliminating the performance overhead associated with synchronous virtualization-based replication.
Solution Approach 2:
The system performs preliminary actions by continuously maintaining replica copies of application data on slave servers before failures occur. When a master server fails, these pre-positioned replicas enable immediate failover without requiring real-time synchronization, thus improving reliability while avoiding the continuous performance penalty of synchronous replication.
2Device complexity
If manual intervention is used to move applications between servers, then system complexity is reduced, but recovery time increases and productivity decreases
Solution Approach 1:
The system implements automated feedback mechanisms where slave servers continuously monitor the health and availability of the master server. When a failure is detected, the feedback loop triggers automatic failover procedures, selecting an appropriate slave server to take over the master's role without requiring manual intervention, thus reducing recovery time while maintaining manageable system complexity through standardized protocols.
Solution Approach 2:
The system enables self-service by empowering slave servers to autonomously detect master server failures and initiate failover procedures. The slave servers automatically update their status, assume the master role, and continue serving applications, eliminating the need for manual intervention and significantly reducing recovery time while keeping system complexity within acceptable bounds through automated decision-making logic.
3Productivity
If a shared storage backend is used in a classical cluster, then load distribution is improved, but single points of failure increase and device complexity increases
Solution Approach 1:
The system extracts the storage dependency from the cluster architecture by implementing file system replication at the application layer rather than relying on a shared storage backend. Each slave server maintains its own local copy of the replicated file system, eliminating the shared storage infrastructure and thereby removing the single points of failure associated with SAN hardware, switches, and storage controllers while preserving load distribution capabilities.
4Manufacturing precision
If synchronous replication is used to ensure data consistency, then data integrity is improved, but network bandwidth consumption increases and speed decreases
Solution Approach 1:
The system employs periodic action by implementing asynchronous replication where slave servers receive and apply data changes at discrete intervals rather than requiring synchronous confirmation for each operation. This approach maintains data consistency over time while allowing the master server to continue operating at full speed, as replication occurs in the background without blocking primary operations, thus resolving the contradiction between data consistency and replication speed.
Data Source
AI summary
A method and apparatus for distribution of applications amongst a number of servers, ensuring that changes to application data on a master for that application are asynchronously replicated to a number of slaves for that application. Servers may be located in geographically diverse locations; the invention permits data replication over high-latency and lossy network connections and failure-tolerance under hardware and network failure conditions. Access to applications is mediated by a distributed protocol handler which allows any request for any application to be addressed to any server, and which, when working in tandem with the replication system, pauses connections momentarily to allow seamless, consistent live-migration of applications and their state between servers. Additionally, a system which controls the aforementioned live-migration based on dynamic measurement of load generated by each application and the topological preferences of each application, in order to automatically keep servers at an optimum utilisation level.