Map Reduce Join Optimization via Partitioned Metadata
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current map reduce frameworks face inefficiencies in performing join operations, particularly when dealing with large metadata tables, as existing methods like Map-Side join and Semi Join require multiple map-reduce jobs, leading to high memory consumption, processor time usage, and network traffic, making them impractical for large datasets.
Innovation Solution
A computer-implemented method and system that collects application-specific refining data in a first map phase, resets the data stream, and generates resultant data elements in subsequent map phases, reducing the need for multiple map-reduce jobs by reusing mappers and minimizing network shuffling, thereby optimizing join operations across distributed computing systems.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If Map-Side join is used to join large factual table with metadata table, then join operation can be performed efficiently, but it fails when the smaller table (metadata table) is too large to be loaded in memory
Solution Approach 1:
The patent segments the metadata table into multiple smaller tables based on partitioning criteria. This allows the join operation to process manageable portions of data at a time, avoiding the need to load the entire metadata table into memory while maintaining efficient join performance through distributed processing across multiple nodes.
Solution Approach 2:
The patent introduces a partitioning dimension to the traditional two-table join structure. By organizing metadata tables into multiple partitioned tables with specific partition keys, the system transforms the memory-intensive single-load approach into a distributed multi-table approach, enabling efficient joins without requiring all data to reside in memory simultaneously.
2Productivity
If Semi Join uses three map-reduce jobs to perform join operation, then join can be performed on large datasets, but it involves shuffling large volume of data across network which is time consuming
Solution Approach 1:
The patent merges the functionality of multiple map-reduce jobs into a single coordinated operation. By integrating the join logic directly into the map phase and using partitioned metadata tables, the system eliminates the need for separate reduce jobs and intermediate data shuffling, performing the join operation in one pass through the distributed system.
Solution Approach 2:
The patent performs preliminary partitioning and organization of metadata tables before the join operation begins. This pre-processing arrangement ensures that data is already organized in a way that enables efficient direct joining without requiring subsequent shuffling or multiple passes through the data, significantly reducing network traffic and processing time.
3Productivity
If Semi Join loads identified rows in memory for Map-Side join, then join operation can be performed, but when the number of identified rows is large, it would be infeasible to load all identified rows in memory
Solution Approach 1:
The patent segments the metadata table into multiple partitioned tables, each containing a manageable subset of rows. This segmentation allows the system to load and process only the necessary partitions into memory at any given time, avoiding the need to load all identified rows simultaneously while maintaining the ability to perform comprehensive join operations across the entire dataset.
4Productivity
If Per Split Semi Join is used to process segments of larger table, then join operation can be performed on distributed data, but a tuple of smaller table having join key in both splits will be repeated causing redundancies and explosion in size of resulting look up tables
Solution Approach 1:
The patent applies local quality by assigning specific partition keys to specific metadata table partitions. This ensures that each partition handles a distinct subset of join keys, eliminating overlaps and duplications. The join operation then processes each partition independently with well-defined boundaries, preventing the redundancy problem that occurs when partitions share overlapping key spaces.
Data Source
AI summary
The present invention provides a computer implemented method for performing an application specific operation on a dataset over a distributed computing system. The computer implemented method includes collecting a set of application specific refining data present in a data stream of the dataset in a first map phase by the distributed computing system, resetting the data stream of the dataset to a beginning position by the distributed computing system, and generating one or more resultant data elements using the set of application specific refining data in a second map phase, by the distributed computing system.