Map Reduce Cross Product Generation via Key-Based Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Generating cross products from large datasets using the map reduce framework is inefficient, consuming significant computational, memory, and storage resources, and is impractical due to the generation of a single large cross product dataset, which can lead to skewed data processing and idle resources.
Innovation Solution
The approach involves generating multiple cross products based on a join field, where key-value pairs with identical keys are grouped and processed separately, controlling the number of key-value pairs sent to reducers to distribute computational load efficiently, thereby reducing resource consumption and improving processing speed.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a single large cross product dataset is generated from large datasets using map reduce framework, then the complete cross product is obtained, but computational resources, memory resources, and storage resources are consumed significantly
Solution Approach 1:
The patent divides the large dataset into multiple subsets based on a join field (e.g., geographic region, time period, or category). Each subset is processed separately to generate multiple smaller cross products instead of one large cross product. This segmentation reduces memory requirements and computational overhead while maintaining the ability to obtain complete cross product results by combining the subsets.
2Reliability
If a single large cross product dataset is generated, then the complete cross product is obtained, but the processing speed is slow due to skewed data processing and idle resources
Solution Approach 1:
By segmenting the data into multiple subsets and processing them in parallel across different map reduce jobs, the system utilizes all available computational resources efficiently. Each subset is processed independently, eliminating data skew issues where some reducers would be overloaded while others remain idle. The overall processing speed increases while maintaining complete cross product results.
3Productivity
If multiple cross products are generated based on join field with grouped key-value pairs, then resource consumption is reduced and processing speed is improved, but the complexity of data distribution and load balancing increases
Solution Approach 1:
The system automatically performs load balancing by having the map phase dynamically distribute key-value pairs to reducers based on the join field values. Each reducer processes only the subsets of data relevant to its assigned join field values, eliminating the need for manual load balancing configuration. This self-service approach simplifies the overall system complexity while maintaining high processing efficiency.
Data Source
AI summary
A request to perform a set of map reduce jobs that generate a plurality of cross products from a dataset based on a join field is received. The join field indicates that each of the plurality of cross products is to be generated from a corresponding subset of the dataset. Responsive to receiving the request to perform the set of map reduce jobs, the set of map reduce jobs on the dataset to generate the plurality of cross products is performed. The final output data of the set of map reduce jobs is stored. The final output data comprises the plurality of cross products.