DHT-Based Distributed File System for Read-Write Consistency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing distributed content delivery networks (CDNs) like BitTorrent are limited to handling immutable content and do not support both reads and writes, necessitating different distributed network architectures for scenarios requiring data modifications.
Innovation Solution
A distributed hash table-based distributed file system (DHTFS) that uses a redundancy scheme with R1 < R2 < R3 to store and synchronize content across multiple computer systems, allowing for write operations and maintaining data consistency through hash-based message exchange, enabling efficient data replication and access.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a distributed content delivery network (CDN) like BitTorrent is used for content distribution, then read scalability is improved, but the system cannot support write operations and is limited to immutable content
Solution Approach 1:
The patent transforms the CDN system into a multi-functional platform that can handle both read and write operations. By introducing a distributed file system layer over the CDN infrastructure, the system gains universality to serve diverse workloads including content delivery, file storage, and collaborative editing, resolving the limitation of being restricted to immutable content only
Solution Approach 2:
The patent segments the system into distinct functional layers: the underlying CDN layer handles content distribution and read operations, while a new distributed file system layer handles write operations and data consistency. This segmentation allows each layer to optimize for its specific function while working together to provide comprehensive read-write capabilities
2Reliability
If content is replicated across multiple nodes in a distributed system, then data availability and read scalability are improved, but maintaining data consistency during write operations becomes complex
Solution Approach 1:
The patent introduces a distributed file system layer as an intermediary between clients and the CDN nodes. This intermediary layer manages write operations, handles data consistency, and coordinates updates across replicated nodes using version vectors and conflict resolution mechanisms, thereby reducing synchronization complexity while maintaining data availability
Solution Approach 2:
The system implements feedback mechanisms through version vectors and acknowledgment protocols that track the state of replicated data across nodes. When write operations occur, the system receives feedback about the current state of replicas and uses this information to coordinate updates and resolve conflicts, maintaining consistency without excessive complexity
3Reliability
If a fixed number of redundant copies are stored in a distributed system, then data reliability is improved, but the system cannot naturally balance serving capacity with increases in popularity
Solution Approach 1:
The patent implements dynamic redundancy management where the number of replicas and their distribution across nodes can adapt based on workload demands and data popularity. The system dynamically adjusts replication factors and rebalances data across the distributed CDN infrastructure, enabling serving capacity to scale naturally with popularity while maintaining appropriate data reliability
Data Source
AI summary
Content items in a distributed system are defined by a respective key, and each such content item is copied to R1 computer systems of the distributed system which have unique identifiers closest to a value of the respective key, where R1 is less than R2 which is less than R3, at least R2 number of the computer systems have copies of any respective one of the content items for all of the content items, and none of the computer systems farther than R3 in an address space of the distributed system have a copy of the subject content item. Modifications of individual content items are synchronized across all instances thereof responsive to a put operation and/or at periodic intervals.