Dual Node Parallel Flush for Distributed Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed data storage systems, the inefficiency arises when only the primary node flushes data, requiring the secondary node to read from disk, which can lead to unnecessary disk reads and underutilization of nodes, especially when the secondary node has written data that needs to be flushed.
Innovation Solution
Implement a method where both nodes identify which pages they have written and allocate the flush operation in parallel based on factors such as ownership, load balancing, and availability, allowing the secondary node to handle its own flush operations if available, thereby reducing disk reads and optimizing workload distribution.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If only the primary node performs flush operations, then data consistency is maintained, but disk read operations increase and node utilization decreases
Solution Approach 1:
The flush operation is segmented and distributed to specific nodes based on page ownership. Instead of centralizing all flush operations on the primary node, the system divides the workload by identifying which node originally wrote each page and directing the flush operation to that node, thereby segmenting the centralized flush task into distributed sub-tasks.
Solution Approach 2:
Nodes perform flush operations for their own written pages without requiring the primary node to read from disk. When a node has written pages that need flushing, it can autonomously perform the flush operation for those pages, eliminating the need for the primary node to service every flush request and reducing unnecessary disk reads.
2Productivity
If the secondary node flushes its own written pages, then disk reads are reduced, but coordination complexity increases
Solution Approach 1:
The system performs preliminary identification of page ownership before initiating flush operations. By determining which node wrote each page in advance and tracking this information, the system prepares the necessary routing information beforehand, simplifying the coordination during actual flush operations and reducing runtime complexity.
Solution Approach 2:
The system implements feedback mechanisms where nodes report which pages they have written and need flushing. This feedback loop allows the primary node to track page ownership and direct flush operations appropriately, managing coordination complexity through structured information exchange rather than complex real-time decision-making.
3Productivity
If parallel flush operations are implemented across multiple nodes, then system throughput increases, but synchronization requirements increase
Solution Approach 1:
The parallel flush capability is achieved by segmenting the flush workload across multiple nodes based on page ownership. Each node handles flush operations for the pages it originally wrote, creating independent parallel flush streams that do not require complex synchronization since each node operates autonomously on its own data segments.
Data Source
AI summary
A method, computer program product, and computer system for identifying a first node that has written a first page of a plurality of pages to be flushed. A second node that has written a second page of the plurality of pages to be flushed may be identified. It may be determined whether the first page of the plurality of pages is to be flushed by one of the first node and the second node and whether the second page of the plurality of pages is to be flushed by one of the first node and the second node based upon, at least in part, one or more factors. The first node may allocate the first page of the plurality of pages and the second page of the plurality of pages to be flushed in parallel by one of the first node and the second node based upon, at least in part, the one or more factors.