Map-Reduce Merge Subsystem for Relational Dataset Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional MapReduce systems are limited in processing relationships between multiple datasets simultaneously, lacking the capability to efficiently merge and process multiple relational datasets in a scalable, fault-tolerant, and cost-effective manner.
Innovation Solution
The method involves creating multiple map-reduce subsystems for each dataset, with a merge function that collects and merges intermediate data from these subsystems based on user-defined conditions, enabling the processing of relationships between datasets using a cluster-based storage system, and allowing flexible placement of merge functions among subsystems.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional MapReduce systems process multiple datasets, then processing capability is improved, but system complexity increases due to lack of merge functionality
Solution Approach 1:
The patent introduces a merge function that combines intermediate data from multiple map-reduce subsystems processing different datasets. This allows the system to handle multiple datasets simultaneously while maintaining manageable complexity through a standardized merging mechanism that integrates seamlessly with existing map-reduce operations.
Solution Approach 2:
The merge function serves multiple purposes: it combines intermediate data from different datasets, supports user-defined merge conditions, and integrates with the existing map-reduce architecture. This multi-functional approach improves processing capability without proportionally increasing system complexity.
2Adaptability or versatility
If multiple map-reduce subsystems are created for each dataset, then relational processing is improved, but computational resources increase
Solution Approach 1:
By merging intermediate data from multiple map-reduce subsystems, the patent enables relational processing between datasets without requiring separate complete processing chains. The merge function consolidates results from multiple subsystems, reducing the need for redundant computational resources while maintaining adaptability for various relational operations.
3Adaptability or versatility
If merge functions are flexibly placed among subsystems, then processing flexibility is improved, but system control complexity increases
Solution Approach 1:
The merge function acts as an intermediary between multiple map-reduce subsystems, allowing flexible placement and configuration without directly complicating the core subsystems. This mediator approach enables processing flexibility while containing control complexity within the standardized merge function interface.
Data Source
AI summary
A method of processing relationships of at least two datasets is provided. For each of the datasets, a map-reduce subsystem is provided such that the data of that dataset is mapped to corresponding intermediate data for that dataset. The intermediate data for that dataset is reduced to a set of reduced intermediate data for that dataset. Data corresponding to the sets of reduced intermediate data are merged, in accordance with a merge condition. In some examples, data being merged may include the output of one or more other mergers. That is, generally, merge functions may be flexibly placed among various map-reduce subsystems and, as such, the basic map-reduce architecture may be advantageously modified to process multiple relational datasets using, for example, clusters of computing devices.