MapReduce Query Splitting for Database Fragment Indexing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing MapReduce implementations face inefficiencies when integrating with databases that contain encoded textual objects, leading to redundancy and increased network traffic due to the need for each task to execute the same query and scan overlapping data sets.
Innovation Solution
The method involves dividing a query into split characterization queries, distributing them to worker nodes that store partitions of the distributed database with encoded textual objects and pre-defined indices, and executing these queries to obtain preliminary information about query results, including fragment counts, database partition identification, and host names, thereby minimizing redundant work and optimizing data processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If each task executes the same query and scans overlapping data sets to fetch rows assigned to the task, then data processing can be distributed across multiple tasks, but redundancy is created and network traffic increases
Solution Approach 1:
The system performs preliminary actions by pre-computing indices that characterize encoded textual object fragments and storing them in the distributed database partitions. When MapReduce tasks need to process data, they can directly query these pre-computed indices to determine which specific fragments match the query criteria, rather than scanning entire overlapping data sets. This preliminary preparation eliminates redundant scanning and reduces network traffic by allowing tasks to fetch only the necessary data fragments.
Solution Approach 2:
The invention segments the database into distributed partitions, each containing encoded textual objects and pre-computed indices characterizing fragments within that partition. By dividing the database this way and maintaining local indices at each partition, the system enables tasks to query only relevant segments rather than scanning entire data sets across the network, thereby reducing redundancy while maintaining distributed processing capability.
2Productivity
If MapReduce is used to process distributed databases with encoded textual objects, then parallel processing can be achieved, but existing implementations fail to efficiently integrate with such databases
Solution Approach 1:
The system creates a universal interface layer between MapReduce and databases with encoded textual objects. By implementing a standardized method where MapReduce tasks can query pre-computed indices to obtain fragment information, the solution enables efficient integration across different database implementations while maintaining parallel processing capabilities. This universal approach allows MapReduce to work effectively with encoded textual object databases without requiring implementation-specific optimizations.
Solution Approach 2:
The invention introduces an intermediary mechanism in the form of pre-computed indices that characterize encoded textual object fragments. These indices act as a mediator between MapReduce tasks and the encoded textual object database, allowing tasks to efficiently query and identify relevant fragments without directly accessing or scanning the encoded objects themselves. This intermediary layer enables efficient integration while preserving the benefits of parallel processing.
3Productivity
If large amounts of data are moved across the network for processing, then distributed processing can be performed, but network traffic increases and processing time is consumed
Solution Approach 1:
The system extracts only the essential information needed for query processing by pre-computing and storing indices that characterize encoded textual object fragments. Instead of moving large amounts of encoded textual data across the network, the system extracts and transfers only the compact index information that allows tasks to identify and fetch specific fragments. This extraction approach dramatically reduces network traffic and associated processing time while enabling distributed processing.
Data Source
AI summary
A method of processing a query in a distributed database implemented across a set of nodes includes receiving a query. The query is divided into split characterization queries. The split characterization queries are distributed to worker nodes. Each worker node stores a partition of the distributed database with encoded textual objects and pre-defined indices characterizing encoded textual object fragments. The split characterization queries are executed at the worker nodes to obtain preliminary information about query results. Executing the split characterization queries includes matching query fragments associated with the split characterization queries with encoded textual object fragments of the pre-defined indices to produce fragment matches representative of the size of the query results. For each split characterization query the preliminary information about query results includes a fragment count, a database partition identification, and a database host name.