Front End Backend Replicated Storage for Data Availability
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large-scale distributed systems face challenges in maintaining data availability and reliability due to centralized storage, which can lead to single points of failure and are often application-specific, limiting scalability and disaster recovery capabilities.
Innovation Solution
Implementing a front-end and backend replicated storage system where primary data copies are synchronized across multiple front-end server nodes and backed up on a data-agnostic backend node, allowing for seamless failover and application-agnostic data management.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If centralized storage is used to improve data availability and reliability, then system reliability is improved, but device complexity increases and scalability is limited
Solution Approach 1:
The storage system is segmented into multiple independent components: front-end server nodes for read operations, backend server nodes for write operations and data storage, and a replication manager. This segmentation allows each component to function independently, improving reliability while managing complexity through modular architecture.
Solution Approach 2:
The storage system is designed to be application-agnostic and format-agnostic, serving multiple applications with different data formats through a unified interface. The backend server nodes can store data in various formats without requiring application-specific storage software, providing multi-functionality that improves scalability.
2Speed
If application-specific storage formats are used to optimize performance, then access speed is improved, but adaptability deteriorates as multiple applications cannot share the same storage infrastructure
Solution Approach 1:
The storage system uses a universal interface that accepts data from multiple applications in different formats and stores it in a normalized format on backend server nodes. The replication manager handles format conversion and data replication, allowing the system to serve multiple applications simultaneously without requiring application-specific storage software.
Solution Approach 2:
The replication manager acts as an intermediary between front-end server nodes and backend server nodes, handling data format conversion and replication. This intermediary layer abstracts the complexity of format conversion from individual applications, enabling multi-application compatibility while maintaining performance through optimized data access paths.
3Reliability
If synchronous replication to secondary copies is implemented to ensure data consistency, then data reliability is improved, but loss of time increases due to wait time for replication completion
Solution Approach 1:
The system implements partial synchronous replication where write operations must be confirmed on a quorum of front-end nodes before returning to the client, but full synchronous replication to all secondary copies is not required. This partial action approach ensures data consistency for the minimum necessary reliability while reducing write operation latency.
Solution Approach 2:
The system performs preliminary replication actions by immediately replicating data to secondary front-end nodes and backend server nodes in the background after the primary write operation is confirmed. This preliminary action ensures data consistency is maintained without blocking the primary write operation, reducing perceived latency.
4Ease of manufacture
If homogenous storage software is used across all server nodes to simplify deployment, then ease of manufacture is improved, but adaptability deteriorates as the system cannot handle different data formats from multiple applications
Solution Approach 1:
The system uses homogenous storage software on all server nodes for simplified deployment, combined with a replication manager that provides format conversion capabilities. This universal approach allows the same software base to serve multiple applications with different data formats, maintaining ease of deployment while achieving adaptability.
Solution Approach 2:
The replication manager serves as an intermediary layer that handles data format conversion and compatibility issues between different applications and the homogenous storage software. This allows the storage system to maintain homogenous software across all nodes while still supporting multiple data formats through the intermediary's translation capabilities.
Data Source
AI summary
An existing primary data copy can be maintained on an existing primary front end server node. One or more existing secondary data copies can be maintained on one or more existing secondary front end server nodes to minor the existing primary data copy (such as by making synchronous changes to the secondary data copies). One or more existing backup data copies can be maintained on an existing backend server node to mirror the existing primary data copy (such as by making asynchronous changes to the one or more backup data copies). The existing backend server node can be accessible by one or more of the front end nodes. In response to detecting a failure of the existing primary data copy, one of the one or more secondary data copies can be designated as a new primary data copy in place of the existing primary data copy.