Map-Reduce Hierarchical Data Partitioning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Processing hierarchical data in a map-reduce framework is challenging due to difficulties in achieving full parallelism and independent functioning among maps, especially when input data is not structured, leading to inefficient performance.
Innovation Solution
The method involves determining the cost of partitioning and redefining the job, with options to either partition the data or redefine the job, using a dynamic programming algorithm to minimize processing costs by ensuring each partition size is within a threshold, and introducing intermediate steps to handle large hierarchical data trees.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If hierarchical data is processed in a map-reduce framework, then data processing capability is improved, but parallelism and independent functioning among maps cannot be achieved efficiently
Solution Approach 1:
The patent segments the hierarchical data into multiple independent parts by introducing virtual nodes and partitioning the data tree. Each segment can be processed independently by different map instances, achieving parallelism while maintaining the hierarchical structure. The segmentation is done through virtual node insertion and data partitioning algorithms that ensure independent processing capability.
Solution Approach 2:
The patent introduces virtual nodes as intermediaries between the hierarchical data structure and the map-reduce processing framework. These virtual nodes act as mediators that transform the hierarchical data into a format suitable for parallel processing, enabling independent functioning among maps while preserving the original data relationships.
2Speed
If data is partitioned to enable parallel processing, then processing speed is improved, but data structure complexity increases
Solution Approach 1:
The patent applies segmentation by dividing the hierarchical data into manageable partitions while using virtual nodes to maintain the structural relationships. This allows parallel processing speed improvement without proportionally increasing complexity, as the virtual nodes provide a systematic way to organize the partitioned data.
Solution Approach 2:
The patent makes the data structure dynamic by allowing flexible partitioning and repartitioning of hierarchical data based on processing requirements. The virtual node framework enables dynamic adjustment of data partitions without fundamentally changing the hierarchical structure, balancing speed and complexity.
3Device complexity
If conventional map-reduce framework is used for hierarchical data, then framework simplicity is maintained, but processing performance becomes highly unsatisfactory
Solution Approach 1:
The patent makes the conventional map-reduce framework universal by introducing virtual nodes that can handle various hierarchical data structures. The same framework can process different types of hierarchical data (JSON, XML, etc.) through the virtual node abstraction, improving performance without requiring framework changes.
Solution Approach 2:
The virtual nodes serve as intermediaries that bridge the simple map-reduce framework and complex hierarchical data. This allows the framework to maintain its simplicity while achieving improved performance through the intermediary layer that handles the complexity of hierarchical data processing.
Data Source
AI summary
Methods and arrangements for processing hierarchical data in a map-reduce framework. Hierarchical data is accepted, and a map-reduce job is performed on the hierarchical data. This performing of a map-reduce job includes determining a cost of partitioning the data, determining a cost of redefining the job and thereupon selectively performing at least one step taken from the group consisting of: partitioning the data and redefining the job.