Decentralized File Mapping for Distributed Node Resilience
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional distributed file systems face issues with scalability, resilience, and IP address management, particularly due to centralized task managers and reliance on Dynamic Host Configuration Protocol (DHCP), which can lead to single points of failure and difficulties in configuring IP addresses for virtual machines.
Innovation Solution
Implementing a decentralized distributed system using a consensus algorithm like Paxos for fault-tolerant task management, allowing tasks to be launched from any node, and using MAC addresses to assign IP addresses to virtual machines, eliminating the need for DHCP and enhancing resilience and scalability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If a centralized task manager is used to coordinate tasks on all nodes, then task execution order is known and manageable, but the system reliability deteriorates because the task manager becomes a single point of failure
Solution Approach 1:
The patent segments the centralized task manager functionality into distributed task management capabilities across multiple nodes. Each node maintains its own task queue and can independently execute tasks, eliminating the single point of failure while preserving coordinated task execution through inter-node communication and shared task queue data structures.
2Extent of automation
If DHCP is used to assign IP addresses to all systems in the network, then IP address assignment is automated, but the system complexity increases and virtual machine isolation becomes difficult
Solution Approach 1:
The patent extracts IP address assignment from the centralized DHCP server and implements it as a local function on each node. Each node generates and assigns IP addresses to virtual machines independently using local configuration files, eliminating the need for complex DHCP server configuration and improving virtual machine isolation while maintaining automated IP assignment.
3Loss of information
If out-of-band mechanisms are used for data location services, then data location can be obtained, but the system complexity increases and protocol compatibility is reduced
Solution Approach 1:
The patent merges the data location service into the existing file system protocol stack. Instead of using separate out-of-band mechanisms, the system retrieves data location information through standard file system operations and protocol exchanges between nodes, reducing system complexity while maintaining full protocol compatibility and NFS interoperability.
Data Source
AI summary
In some examples, a distributed system may include a plurality of nodes with a file stored in a file system across the plurality of nodes. One of the nodes may receive a request for location data corresponding to a part of the file. For instance, the location data may be included in a mapping file, and the request may specify an offset of the mapping file. The mapping file may include locations of nodes storing parts of the file in the file system across the plurality of nodes, and the mapping file may have a plurality of parts corresponding to the parts of the file. In response to the request, the node may access the mapping file and send location data corresponding to the offset of the mapping file. The location data may indicate at least one of the nodes storing the part of the file.