Distributed Data Storage via Local Key-Value Sorting
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The conventional MapReduce framework is resource-intensive and time-consuming due to high network traffic and data transfer overhead when processing structured data, leading to slower job completion and increased network bandwidth consumption.
Innovation Solution
The file generation system processes structured data in an enhanced map phase, segregating it into subsets, generating and sorting intermediate key-value pairs locally, and storing them directly into a distributed database without network transfer, thereby eliminating network overhead and optimizing resource usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If the conventional MapReduce framework is used to process structured data, then data processing capability is improved, but network bandwidth consumption increases and job completion time increases
Solution Approach 1:
The patent segments the MapReduce processing into two distinct phases: a map phase that generates intermediate key-value pairs locally, and a reduce phase that receives pre-sorted data. This segmentation allows the map phase to operate independently without network communication, reducing network bandwidth consumption while maintaining data processing capability.
Solution Approach 2:
The patent implements preliminary sorting of intermediate key-value pairs during the map phase before data is transferred to the reduce phase. This preliminary action eliminates the need for shuffle sorting during the reduce phase, significantly reducing network bandwidth consumption and improving overall processing efficiency.
2Productivity
If the conventional MapReduce framework is used to process structured data, then data processing capability is improved, but job completion time increases
Solution Approach 1:
The patent performs preliminary sorting of intermediate key-value pairs during the map phase, before data transfer to the reduce phase. This eliminates the time-consuming shuffle sorting operation during the reduce phase, directly reducing job completion time while maintaining full data processing capability.
Solution Approach 2:
The patent extracts the sorting operation from the reduce phase and places it in the map phase. This extraction allows the reduce phase to focus solely on aggregation operations, reducing its computational overhead and accelerating job completion while preserving data processing capability.
3Adaptability or versatility
If intermediate key-value pairs are transferred over the network in MapReduce, then distributed processing is achieved, but network overhead increases
Solution Approach 1:
The patent segments the data processing workflow into map and reduce phases with clearly defined responsibilities. The map phase generates and sorts intermediate key-value pairs locally, while the reduce phase receives already-sorted data. This segmentation minimizes network overhead by transferring only essential data between phases, maintaining distributed processing capability.
Solution Approach 2:
The patent changes the parameter of data organization by sorting intermediate key-value pairs by key during the map phase rather than during the reduce phase. This parameter change in the sorting timing and location reduces network overhead by ensuring that only sorted, essential data is transferred across the network.
Data Source
AI summary
A file generation system for storage of structured data onto a distributed database includes an intermediate data generation module to generate a set of intermediate key-value pairs for each of one or more records in at least one subset of the structured data. A key-value pair includes a key and a value corresponding to the key, where the key is a unique identifier of the value. A file generation system further includes an intermediate data sorting module to sort the set of intermediate key-value pairs to generate a plurality of output files. Each of the plurality of output files includes at least one key-value pair. Further, the file generation system includes a file storing module to store the plurality of output files in the distributed database, where the plurality of output files are representative of the structured data.