Distributed Storage Data Splitting and Zone Replication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Managing large volumes of data in distributed storage systems efficiently, particularly in handling data access requests from diverse geographical locations, is a constant challenge due to the complexity of data replication and storage scalability.
Innovation Solution
A distributed storage system that divides data into splits, each associated with an account, and manages these splits within groups, allowing for dynamic creation of new splits when existing ones reach capacity, and replicates data according to account-specific policies across multiple zones, ensuring efficient data access and storage scalability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is divided into splits and stored in a distributed storage system, then data scalability and access efficiency are improved, but system complexity and management overhead increase
Solution Approach 1:
The patent divides data into multiple splits, where each split contains a portion of the data for an account. This segmentation allows parallel access to different splits by different servers, improving data access efficiency while maintaining manageable data units that can be independently replicated and stored across the distributed system.
Solution Approach 2:
The patent introduces a coordinator server as an intermediary that manages the complexity of data organization, replication policies, and split creation. This intermediary handles the bookkeeping of which splits belong to which accounts and manages replication across zones, shielding other system components from the underlying complexity while enabling efficient data access.
2Reliability
If data is replicated across multiple zones according to account-specific policies, then data reliability and availability are improved, but storage overhead and system resource consumption increase
Solution Approach 1:
The patent implements account-specific replication policies where different accounts can have different replication factors and zone distributions. This allows critical data to be replicated more extensively for high availability while less critical data uses minimal replication, optimizing the balance between reliability and storage overhead based on local data importance.
Solution Approach 2:
The system creates multiple replicas of data splits across different zones, but only to the extent required by each account's replication policy. This partial replication approach ensures sufficient availability for each account without unnecessarily duplicating data across all possible zones, thereby controlling storage overhead.
3Adaptability or versatility
If new splits are created dynamically when existing splits reach capacity, then storage scalability is improved, but data management complexity and operational overhead increase
Solution Approach 1:
The system pre-allocates splits with predetermined capacity limits when accounts are created or data is initially stored. This preliminary organization allows the system to anticipate when splits will reach capacity and automatically create new splits before data overflow occurs, enabling seamless scalability without manual intervention.
Solution Approach 2:
The coordinator server automatically monitors split capacity and creates new splits when existing ones reach their limits. This self-service mechanism handles the complexity of dynamic split creation, capacity management, and data redistribution without requiring manual operational intervention, thereby maintaining ease of operation while achieving storage scalability.
4Loss of information
If data is organized into groups and splits with associated metadata, then data traceability and management control are improved, but system complexity and processing overhead increase
Solution Approach 1:
The patent implements a nested hierarchical structure where accounts contain multiple splits, splits are organized into groups, and each level maintains metadata about its contents. This nested organization enables efficient traceability by allowing the system to navigate from account to splits to groups using stored metadata, while the hierarchical structure prevents the need for complex flat-data management.
Solution Approach 2:
The system maintains metadata that provides feedback about data location, capacity, and replication status. This feedback mechanism enables the coordinator server to make informed decisions about data management operations while keeping the metadata structure relatively simple and organized, balancing traceability requirements with system complexity.
Data Source
AI summary
A distributed storage system is provided. The distributed storage system includes multiple front-end servers and zones for managing data for clients. Data within the distributed storage system is associated with a plurality of accounts and divided into a plurality of groups, each group including a plurality of splits, each split being associated with a respective account, and each group having multiple tablets and each tablet managed by a respective tablet server of the distributed storage system. Data associated with different accounts may be replicated within the distributed storage system using different data replication policies. There is no limit to the amount of data for an account by adding new splits to the distributed storage system. In response to a client request for a particular account's data, a front-end server communicates such request to a particular zone that has the client-requested data and returns the client-requested data to the requesting client.