Distributed Cluster File System Push Node Architecture
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional distributed cluster file systems face bottlenecks during file transmission due to limited load capacity of interface machines and potential instability in the database, leading to I/O bottlenecks and system failures.
Innovation Solution
A distributed cluster file system architecture that includes a coordination node cluster, a file storage node cluster, and a push node cluster with a master and slave push node configuration, where the master push node generates a file transmission path and directs slave push nodes to transmit files to destination node clusters, distributing the load and ensuring continuity even if a node fails.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If multiple destination nodes request files from a single interface machine, then file transmission can be initiated, but the interface machine becomes an I/O bottleneck due to limited load capacity
Solution Approach 1:
The system segments the interface machine's file transmission function by introducing multiple push nodes (master and slave) that can independently handle file transmission to different destination nodes. This distributes the I/O load across multiple nodes rather than concentrating it on a single interface machine, resolving the bottleneck and stability issues.
Solution Approach 2:
The push nodes act as intermediaries between the storage nodes and destination nodes. Instead of destination nodes directly requesting files from storage nodes through a single interface machine, the push nodes receive transmission paths from the master push node and independently complete file transfers, distributing the intermediary function across multiple nodes.
2Ease of operation
If a single database stores file information, then centralized management is achieved, but the system becomes vulnerable to complete failure when the database host fails
Solution Approach 1:
The system segments the centralized database function by distributing file information storage across multiple coordination nodes. Each coordination node stores file information locally, eliminating the single point of failure while maintaining centralized management capabilities through the coordinated action of multiple nodes.
Solution Approach 2:
The system changes the architectural parameter from centralized database storage to distributed storage across coordination nodes. This parameter change transforms the system from vulnerable to resilient, as file information remains accessible through other coordination nodes if one node fails.
3Device complexity
If file transmission is handled by a single interface machine, then simplified architecture is maintained, but the load capacity is limited
Solution Approach 1:
The interface machine's monolithic function is segmented into multiple push nodes with specialized roles (master and slave). The master push node manages coordination and generates transmission paths, while slave push nodes execute actual file transfers, distributing the workload and increasing overall throughput.
Solution Approach 2:
The system transitions from a single-dimension architecture (one interface machine) to a multi-dimensional architecture with multiple push nodes operating in parallel. This dimensional expansion allows simultaneous file transmissions to multiple destination nodes, dramatically increasing throughput while maintaining manageable complexity through clear role differentiation.
Data Source
AI summary
A file transmission method includes: obtaining file information of a to-be-transmitted file from a distributed coordination node cluster; determining that at least one slave push node stores the to-be-transmitted file according to the file information of the to-be-transmitted file and heartbeat information of the at least one slave push node; wherein the heartbeat information of the at least one slave push node contains file information stored in the at least one slave push node, and the to-be-transmitted file is retrieved by the at least one slave push node from a file storage node cluster; generating a file transmission path, wherein the file transmission path comprises at least one destination node cluster; sending the file transmission path to the at least one slave push node, so that the at least one slave push node sends the to-be-transmitted file to the at least one destination node cluster through the file transmission path.