Scalable Fault Tolerant System Using FT Groups
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing fault-tolerant distributed processing systems face challenges in scalability, as failover mechanisms require re-configuration when new servers are added, leading to burdensome and slowed system operations.
Innovation Solution
A fault-tolerant system is designed with FT groups where applications are ranked as primary and mirrors, with a central monitoring and change-over control sub-system using FT messages to quickly switch roles without loss of functionality, allowing for seamless addition of new instances and groups.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If duplicate servers are used for fault tolerance, then system reliability is improved, but system complexity and operational burden increase
Solution Approach 1:
The system segments fault tolerance into functional groups where applications are organized by function rather than requiring complete server duplication. Each FT group contains a primary instance and mirror instances for specific applications, allowing selective redundancy without duplicating entire server configurations.
Solution Approach 2:
Mirror instances are designed to be multi-functional, capable of taking over any primary instance within the same FT group. This universal design allows a single mirror to serve multiple potential primary roles, reducing the need for dedicated duplicate servers for each application.
2Difficulty of detecting and measuring
If each server tracks status of other servers for failover, then fault detection capability is improved, but system operations slow down
Solution Approach 1:
The FT Monitor acts as an intermediary component that centralizes the status tracking and failover detection functions. Instead of each server independently monitoring others (creating multiple monitoring paths), the FT Monitor consolidates this function, receiving status information and managing failover decisions centrally, which reduces communication overhead and speeds up operations.
3Adaptability or versatility
If failover mechanisms are re-configured for each new server addition, then adaptability is improved, but scalability is reduced
Solution Approach 1:
The FT Monitor dynamically manages failover configurations without requiring static pre-configuration for every possible server combination. When new servers are added, the FT Monitor automatically incorporates them into existing FT groups or creates new groups as needed, allowing the system to adapt to changing configurations without manual reconfiguration of failover mechanisms.
Solution Approach 2:
The system allows dynamic modification of FT group parameters and server assignments. New servers can be added to existing FT groups by changing group membership parameters, or new FT groups can be created with different application compositions, enabling flexible scaling without fundamental system reconfiguration.
Data Source
AI summary
A fault tolerant system for a highly scalable distributed processing system in which fault tolerant groups (hereafter referred to as “FT groups”) are formed based on the functionalities of applications. A given FT group includes instances of the same applications and also different applications that perform the same function. The instances within the FT group are ranked as a primary and mirrors, which descend in rank. When the primary instance in the FT group crashes, that is, fails or becomes otherwise unavailable, the system utilizes the next ranking available (running) instance in the FT group as the primary instance. The change over to the next ranking mirror application is performed quickly and without loss of functionality through an exchange of “FT messages” under the control of a central monitoring and change-over control sub-system.