Distributed File System Data Segmentation for Storage Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed file systems face challenges with data management, including data sprawl and the risk of a single point of failure due to storage being inaccessible if a compute node fails, which increases costs and burdens network resources.
Innovation Solution
The technology manages data by moving 'cold' data from primary storage to backup storage, allowing primary storage to focus on 'hot' data, using data eviction policies to identify and relocate less frequently accessed data, thus ensuring efficient storage usage and eliminating the single point of failure.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If multiple copies of the distributed data set (replicas) are created and stored in the storage of multiple compute nodes, then data availability is improved, but storage costs and network resource burden increase
Solution Approach 1:
The system segments data into hot data and cold data based on access frequency. Hot data is replicated across multiple compute nodes for high availability, while cold data is stored in a shared storage system to reduce replication costs. This segmentation allows the system to apply different storage strategies to different data types, resolving the contradiction between data availability and storage costs.
Solution Approach 2:
The system applies different storage quality characteristics to different data locations. Compute nodes maintain high-quality replicated storage for hot data to ensure fast access and availability, while the shared storage system provides adequate storage for cold data. This local quality differentiation allows the system to optimize both availability and cost by matching storage characteristics to data access patterns.
2Quantity of substance
If storage is shared between compute nodes for read operations, then replica sprawl is avoided, but the storage becomes a single point of failure
Solution Approach 1:
The system segments the storage architecture into compute node private storage and shared storage. Private storage is used for hot data replication to ensure availability, while shared storage is used for cold data to avoid replica sprawl. This segmentation allows the system to leverage the strengths of both storage types without combining their weaknesses.
Solution Approach 2:
The system creates an asymmetric storage architecture where hot data and cold data are stored differently. Hot data is replicated across compute nodes with asymmetric read/write capabilities, while cold data is stored in shared storage with different access characteristics. This asymmetry allows the system to optimize for both efficiency and reliability in appropriate contexts.
3Ease of operation
If data is replicated across all compute nodes, then data accessibility is improved, but network resources are burdened by managing replica changes
Solution Approach 1:
The system extracts cold data from the replicated storage structure and places it in shared storage. This extraction eliminates the need to manage replica changes for cold data across the network, reducing network resource burden while maintaining data accessibility through the shared storage system. Only hot data remains in the replicated structure, minimizing replication management overhead.
Data Source
AI summary
Technology is disclosed for managing data in a distributed processing system (“the technology”). In various embodiments, the technology pushes “cold” data from a primary storage of the distributed processing system to a backup storage thereby maximizing the usage of the space on the primary storage to store “hot” data on which most data processing activities are performed in the distributed processing system. The cold data is retrieved from the backup storage into the primary storage on demand, for example, upon receiving an access request from a client. While the primary storage stores the data in a format specific to the distributed processing system, the backup storage stores the data in a different format, for example, format corresponding to the type of backup storage.