Peer-to-Peer Master Node Architecture for Distributed File System Reliability
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current distributed file systems face challenges in efficiently managing data across multiple nodes and maintaining system reliability due to the lack of effective task distribution and load balancing mechanisms, particularly in peer-to-peer architectures.
Innovation Solution
A distributed file system architecture that includes multiple data nodes and a director unit with master nodes configured in a peer-to-peer architecture, allowing for task distribution, load balancing, and continuous communication of task and system status among master nodes, enabling scalable and resilient data management.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If a single master node is used to coordinate access to files in a distributed file system, then the system structure is simple and easy to manage, but the system reliability decreases and processing efficiency is limited when the master node becomes a bottleneck or fails
Solution Approach 1:
The single master node is segmented into multiple master nodes that work in parallel. Each master node can independently coordinate access to files, eliminating the single point of failure and bottleneck. The patent implements this by creating a cluster of master nodes that share the coordination workload and can take over for each other if one fails.
Solution Approach 2:
The system changes the parameter of master node quantity from one to multiple. This parameter change transforms the system from having a single coordination point to having multiple coordination points, thereby improving reliability and processing capacity while maintaining manageable complexity through standardized node behavior.
2Ease of operation
If tasks are concentrated on a single master node, then task management is straightforward, but processing efficiency decreases and the master node may become overloaded
Solution Approach 1:
The task management function is segmented across multiple master nodes. Each master node receives and processes a portion of the total task load, distributing the processing efficiency burden. Tasks are automatically routed to appropriate master nodes based on current system state and node capacity.
Solution Approach 2:
The system implements dynamic task distribution where master nodes can dynamically accept or reject tasks based on their current load and capacity. This dynamic behavior allows the system to adapt to changing conditions and maintain high processing efficiency without overwhelming individual nodes.
3Reliability
If multiple copies of files are stored in the distributed file system, then reliability is improved through replication, but storage space increases and data management complexity increases
Solution Approach 1:
Different data blocks are replicated across different data nodes based on local requirements and access patterns. Not all files are replicated to all nodes, but rather replication is applied selectively based on local quality needs, access frequency, and failure risk assessment for specific data blocks.
Solution Approach 2:
The system changes the replication parameter from uniform replication of all files to selective replication of specific data blocks. This parameter change allows the system to maintain reliability for critical data while minimizing unnecessary storage consumption for less critical data.
4Device complexity
If the distributed file system uses a traditional master-slave architecture, then coordination is centralized and simple, but scalability is limited and the system cannot efficiently handle large numbers of data nodes
Solution Approach 1:
The centralized coordination architecture is segmented into a distributed coordination network of multiple master nodes. Each master node maintains coordination information for a subset of data nodes, allowing the system to scale by adding both master and data nodes in a balanced manner without overwhelming any single coordination point.
Solution Approach 2:
The system transitions from a single-dimension centralized coordination model to a multi-dimensional distributed coordination model. Multiple master nodes operate in parallel across different dimensions of the system, each handling specific coordination tasks, thereby enabling scalability while maintaining coordination simplicity through modular design.
Data Source
AI summary
In some examples, a distributed file system is described. The distributed file system may include multiple data nodes and a director unit. The multiple data nodes may each include one or more data blocks. The director unit may include multiple master nodes configured in a peer-to-peer distributed architecture and operably coupled to the multiple data nodes. Each of the master nodes may be configured to receive a task related to managing data with respect to the distributed file system; to manage a distribution of the task among one or more of the plurality of master nodes; and to communicate a task status and a status of at least a part of the distributed file system to each of the other master nodes.