Lightweight Distributed Storage via Hash-Based Routing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional distributed storage systems face latency issues due to complex mechanisms and redundancy, with master nodes acting as bottlenecks and causing data duplication and inefficient storage.
Innovation Solution
A lightweight distributed storage system that uses key-based and user-defined storage paths to simplify data storage and retrieval by generating storage keys through hash functions, allowing direct RDMA access to storage servers and integrating user nomination for flexible data placement.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If a master node is used for load balancing in a distributed storage system, then data distribution can be managed centrally, but the system latency increases and a single point of failure is created
Solution Approach 1:
The patent extracts the master node from the storage system architecture entirely. Instead of having a centralized master node for load balancing, the system uses decentralized hash-based routing where each client directly calculates storage paths using hash functions on file metadata, eliminating the intermediary master node and its associated latency and single point of failure issues
Solution Approach 2:
The system enables clients to perform their own load balancing and data placement independently. Each client calculates the storage path for data files using hash functions on file metadata (name, size, type), allowing clients to autonomously determine target storage devices without requiring centralized coordination, thus achieving self-service data distribution
2Reliability
If periodic snapshots are maintained to ensure data integrity, then data reliability is improved, but the system complexity and storage overhead increase
Solution Approach 1:
The patent implements versioned storage where old versions of data files are automatically discarded after a configurable retention period. The hash-based routing system naturally handles versioning by calculating storage paths based on current file metadata, allowing old versions to be identified and discarded when new versions are stored, maintaining data integrity through version control without complex snapshot mechanisms
Solution Approach 2:
Instead of maintaining full periodic snapshots of the entire storage system, the patent uses selective version retention where only the necessary number of historical versions are kept based on configurable policies. The hash-based system allows partial retention of data versions, keeping only what is needed for data integrity and recovery while discarding excessive historical data that is no longer required
3Reliability
If multiple replicas of data files are stored on different storage servers, then data availability is improved, but data duplication occurs and storage efficiency decreases
Solution Approach 1:
The patent changes the routing parameter from centralized master node directives to decentralized hash function outputs. By using hash functions on file metadata (name, size, type) to determine storage paths, the system achieves uniform data distribution across storage devices, ensuring data availability through automatic distribution while eliminating the need for complex replication management and reducing storage overhead through efficient space utilization
Data Source
AI summary
Embodiments described herein provide a system for facilitating a lightweight distributed storage system. During operation, the system generates a data segment from a data file associated with a write operation and determines a storage key for the data segment. The storage key indicates a calculated storage path associated with a replica of the data segment. The system then determines whether a user-defined storage path is specified by a user for the data segment. If a user-defined storage path has not been specified by the user for the data segment, the system generates a data packet comprising the data segment, wherein a destination for the data packet corresponds to a target disk specified by the calculated storage path.