Distributed File System Torus Network Erasure Coding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed file systems based on torus interconnection networks face challenges in efficiently supporting erasure coding for exabyte-level storage, particularly in terms of node expandability and varying network loads, which affect data fault tolerance and storage efficiency.
Innovation Solution
A distributed file system architecture that designates a center node for encoding data and determines storage nodes for data and parity blocks in a torus network, using a metadata server to optimize node allocation based on criteria such as disk usage, CPU usage, and adjacency to ensure efficient data distribution and recovery.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a torus interconnection network is used to reduce network construction cost and improve node expandability, then node expandability and cost-effectiveness are improved, but transmission loads between nodes become unbalanced and communication complexity increases
Solution Approach 1:
The patent applies local quality by assigning different roles to different nodes based on their positions in the torus network. Center nodes are designated to perform encoding operations while storage nodes handle data storage, creating localized functional specialization that balances the overall network load and reduces communication complexity
Solution Approach 2:
The patent introduces center nodes as intermediary elements between clients and storage nodes. These center nodes act as mediators that perform erasure coding operations locally, reducing the need for direct communication between all pairs of nodes and thereby simplifying the communication topology
2Reliability
If erasure coding is implemented in a torus network for data fault tolerance, then data reliability is improved, but transmission loads between nodes vary and performance degrades
Solution Approach 1:
The patent segments the torus network into center nodes and storage nodes, distributing the erasure coding workload across multiple center nodes. This segmentation prevents any single node from becoming a bottleneck and maintains encoding performance while achieving data fault tolerance
Solution Approach 2:
The patent exploits the two-dimensional torus topology by strategically selecting center nodes and storage nodes based on their spatial relationships. By utilizing the dimensional structure of the network, the patent optimizes data placement and encoding operations to minimize transmission distances and balance loads
3Productivity
If a fat-tree interconnection network is used to ensure balanced network loads, then load balancing is improved, but network construction cost increases and node expandability is limited
Solution Approach 1:
The patent inverts the conventional approach by not relying on the network topology to provide load balancing, but instead implementing load balancing through software-based node selection and data placement strategies in the distributed file system, allowing the use of more scalable torus topology
Data Source
AI summary
A distributed file system, based on a torus network, includes a center node and one or more storage nodes. The center node encodes data when the data is received from a client. The one or more storage nodes receive data blocks or parity blocks from the center node and store the data blocks or parity blocks.