Distributed Data Node Performance Index Selection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current approaches to data processing in distributed storage systems fail to fully utilize the resources of data nodes, leading to inefficient processing due to reliance on the weakest node, resulting in increased time costs and resource wastage.
Innovation Solution
A method is introduced where the performance indices of data nodes storing different copies of a data block are compared, and the node with better performance is selected for processing, using a MAP algorithm for processing and a REDUCE algorithm for integrating results, to optimize data block processing efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data processing is performed on the weakest data node to ensure all copies can be processed, then system reliability is improved, but processing efficiency deteriorates
Solution Approach 1:
The patent applies local quality by allowing different data nodes to process different copies of the same data block based on their individual performance characteristics. Instead of uniformly processing all copies on the weakest node, each copy is routed to a node whose performance matches the processing requirements, optimizing overall system efficiency while maintaining reliability through selective copy processing.
2Reliability
If multiple copies of data blocks are stored on different devices to increase reliability, then fault tolerance is improved, but resource utilization deteriorates
Solution Approach 1:
The patent changes the parameter of node selection from static (always using the weakest node) to dynamic (selecting nodes based on real-time performance indices). By evaluating performance parameters such as processing speed and resource availability, the system selects the most appropriate node for each data block copy, reducing resource wastage while maintaining the redundancy provided by multiple copies.
3Adaptability or versatility
If data blocks are processed on local devices in distributed processing, then system scalability is improved, but processing balance deteriorates
Solution Approach 1:
The patent introduces dynamics into the distributed processing system by continuously evaluating performance indices of data nodes and dynamically assigning data block copies to nodes based on current performance conditions. This dynamic approach allows the system to scale effectively while maintaining processing balance, as nodes can adapt their workload based on real-time performance feedback.
Data Source
AI summary
In an approach for processing a data block, a processor determines a first performance index of a first data node, wherein a first copy of the data block is stored on the first data node, wherein the first copy of the data block is a portion included in a data object. A processor determines a second performance index of a second data node, wherein a second copy of the data block is stored on the second data node. A processor processes the first copy of the data block using a MAP algorithm in response to the first performance index being greater than, according to a predefined threshold, the second performance index. A processor generates a processed result of the data object using a REDUCE algorithm based on, at least, the processed first copy of the data block.