Disconnected Ingest Fallback Policy for Distributed Storage Zones
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Data storage systems face challenges in maintaining system throughput when replication policies cannot be immediately met, leading to potential delays or failures in storing and accessing data due to unavailability or inefficiency of storage zones.
Innovation Solution
The implementation of a data storage system with multiple storage zones that utilize a fallback policy to ensure data availability by evaluating the availability of storage zones before storing data and using an alternate zone if the specified policy cannot be met, allowing continuous operation and data access without notifying the client of policy unmet conditions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the system strictly enforces replication policies before storing data, then data resiliency is improved, but system throughput deteriorates due to delays when storage zones are unavailable
Solution Approach 1:
The system performs preliminary evaluation of storage zone availability before data ingestion, but decouples the enforcement of replication policies from the immediate data storage operation. Data is stored in the first available zone without waiting for replication to complete, while replication tasks are queued and executed asynchronously when zones become available.
Solution Approach 2:
The system dynamically adjusts replication enforcement based on real-time zone availability. When zones are unavailable, the system temporarily relaxes policy enforcement to maintain throughput, and automatically resumes replication when zones become available, creating a flexible, adaptive replication strategy.
2Manufacturing precision
If the system waits for storage zones to be available before storing data, then replication policy compliance is improved, but data access speed deteriorates
Solution Approach 1:
The system prepares replication tasks in advance by evaluating zone availability and creating a queue of pending replication operations. When data is stored in an available zone, the system immediately initiates background replication tasks rather than blocking the write operation, enabling fast data access while maintaining eventual policy compliance.
Solution Approach 2:
The system maintains continuous data availability by allowing writes to proceed immediately to available zones, while replication operations continue asynchronously in the background. This ensures that data access is not interrupted by replication delays, and replication compliance is achieved continuously over time rather than being blocked.
3Loss of information
If the system notifies clients when replication policies are not met, then policy transparency is improved, but system complexity increases
Solution Approach 1:
The system automatically manages replication policy enforcement without requiring client intervention or notification. The replication manager autonomously evaluates zone availability, queues replication tasks, and executes them when zones become available, handling all policy compliance matters internally without involving the client system.
Data Source
AI summary
A data storage system allowing for ingest of data when certain storage is unavailable is described herein. The storage system includes zones that are independent and autonomous from each other. The zones include nodes that are independent and autonomous. The nodes include storage devices. When data is to be stored in the data storage system according to a specified storage policy and the specified storage policy cannot be achieved, the data is stored according to a fallback storage policy. This allows a client to be able to continue executing without having to wait for a storage anomaly to be corrected or pass. After the data is stored according to a fallback storage policy, the data is at a later time stored according to the specified storage policy.