Hybrid Data Storage Clustering for Demand Spikes
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Modern network computing and data storage systems face challenges in handling unpredictable demand spikes, as existing database services optimized for certain types of increased demand may not be capable of handling different access patterns, such as varying transaction rates.
Innovation Solution
The implementation of a hybridized data storage system that uses redundancy coding and data transfer devices to store and manage data, allowing for scalable and durable storage by provisioning data transfer devices externally and utilizing redundancy coding techniques like erasure coding to ensure data availability and durability across multiple systems.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If database services are optimized for certain types of increased demand (e.g., payload size), then performance for that specific demand type is improved, but the system becomes incapable of handling different access patterns (e.g., transaction rates)
Solution Approach 1:
The system segments storage resources into multiple independent storage devices that can be dynamically allocated. Each storage device operates independently but can be assigned to different database services based on demand patterns, allowing the system to handle multiple access patterns simultaneously without sacrificing optimization for any single pattern.
Solution Approach 2:
Storage devices are designed with multi-functionality, capable of serving different database services and handling various types of demand spikes. The system can dynamically reassign storage resources between different access patterns (e.g., transaction rates, payload sizes) based on real-time requirements, making the infrastructure universally applicable to diverse workloads.
2Adaptability or versatility
If data is stored on external data transfer devices, then scalability and flexibility are improved, but data availability and durability may be compromised during device cycling or failures
Solution Approach 1:
The system implements local quality by creating geographic clusters of storage devices where data is replicated within each cluster. This ensures that data has different quality characteristics at different locations - locally replicated for high availability and durability, yet distributed across multiple external devices for scalability and flexibility.
Solution Approach 2:
The system provides beforehand cushioning through redundant data replication and error correction codes. Before any device cycling or failure occurs, multiple copies of critical data are maintained across different devices within the same geographic cluster, cushioning against potential data loss or unavailability during external device operations.
3Reliability
If geographic clustering of storage devices is implemented, then data durability and availability are improved through redundancy, but system complexity increases
Solution Approach 1:
The system merges multiple geographic clusters into a unified storage infrastructure that presents a single logical interface to database services. By combining the redundancy benefits of geographic clustering with a simplified unified interface, the system achieves high data durability and availability without exposing the underlying complexity to users or applications.
Data Source
AI summary
A data transfer device is used to augment the capabilities of a data storage system. The data transfer device may be capable of persistently storing data for an indeterminate amount of time, and may be configured to store a portion of a bundle of redundancy coded shards that span between the data transfer device and a data storage system configured to store the remainder of the bundle. Data stored on the data transfer device may be read from and written directly to the data transfer device without transfer of data to the data storage system. If the data transfer device is not available, the remaining shards of the bundle may provide a regenerated, original form of the data.