Parallel Database Query Partitioning for MapReduce Efficiency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for accessing relational data from Hadoop's MapReduce implementations are inefficient, particularly in parallel database systems, due to the need for multiple queries and excessive sorting, which can lead to performance issues and scalability problems.
Innovation Solution
The proposed solution involves using vertical and horizontal partitioning techniques to allow efficient access of relational data from MapReduce programs to parallel database systems, where queries are submitted once and distributed among multiple mappers or access module processors, reducing the need for multiple queries and sorting.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If multiple queries are submitted to access relational data from Hadoop MapReduce implementations, then data access completeness is improved, but query processing time and system overhead increase
Solution Approach 1:
The patent combines multiple query operations into a single query execution. The system executes one query against the parallel database system and then distributes the result set across multiple mappers through partitioning, eliminating the need to submit multiple separate queries while maintaining complete data access.
Solution Approach 2:
The patent segments the query result set into multiple partitions that are distributed across different mappers. Each mapper receives a unique portion of the results identified through partitioning strategies (vertical or horizontal), allowing parallel processing of complete data without requiring multiple query submissions.
2Manufacturing precision
If excessive sorting operations are performed during data access, then data organization and query accuracy are improved, but processing efficiency and scalability deteriorate
Solution Approach 1:
The patent performs sorting and data organization as a preliminary action during the initial query execution phase. The parallel database system sorts and organizes the complete result set before distribution, so that subsequent mapper operations work with pre-organized data, eliminating the need for repeated sorting operations and improving overall processing efficiency.
Solution Approach 2:
The patent extracts the sorting and data organization operations from the individual mapper processes and consolidates them into the central query execution phase. This separation allows mappers to focus solely on processing their assigned partitions without performing redundant sorting operations.
3Productivity
If queries are distributed among multiple mappers without partitioning, then parallel processing capability is improved, but query result consistency and data distribution efficiency worsen
Solution Approach 1:
The patent implements systematic partitioning of query results into distinct segments assigned to different mappers. Through vertical partitioning (column-based) or horizontal partitioning (row-based), each mapper receives a well-defined unique portion of the results, ensuring consistent and non-overlapping data distribution that maintains query result integrity while enabling parallel processing.
Data Source
AI summary
Techniques for accessing a parallel database system via an external program using vertical and/or horizontal partitioning are provided. An external program to a database management system (DBMS) configures external mappers to process a specific portion of query results on specific access module processors of the DBMS that are to house query results. The query is submitted by the external program to the DBMS and the DBMS is directed to organize the query results in a vertical or horizontal manner. Each external mapper accesses its portion of the query results for processing in parallel on its designated AMP or set of AMPS to process the query results.