Map-Reduce Small File Processing via Identifier-Based Merging
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Processing large numbers of small files in a computer cluster using the map-reduce framework is inefficient, leading to bottlenecks, slow processing times, and high overhead, especially when the number of files exceeds the number of nodes performing the map operation.
Innovation Solution
Generating output files based on file identifiers, which allows distributing these identifiers across nodes, eliminating the need for nodes to download and read individual files, and merging data into larger output files, thereby reducing processing overhead and increasing efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If nodes download and read individual small files one at a time, then file processing can be performed, but processing overhead increases and processing speed decreases
Solution Approach 1:
The patent merges multiple small files into a single combined file by collecting file identifiers, downloading all files to a coordinator node, concatenating their contents, and storing the merged result. This eliminates the overhead of processing each small file individually in subsequent map-reduce operations, as the merged file is treated as a single processing unit, thereby improving processing speed and reducing system overhead.
Solution Approach 2:
The patent performs preliminary merging of small files into a larger combined file before the main map-reduce processing begins. This preliminary action consolidates the data structure, so that when map-reduce operations are executed, they operate on fewer, larger files rather than numerous small files, reducing the overhead of file handling and improving overall processing efficiency.
2Quantity of substance
If the number of files exceeds the number of nodes, then more data can be processed, but bottlenecks occur and processing time increases
Solution Approach 1:
By merging multiple small files into a single combined file, the patent reduces the total number of file processing units from thousands of small files to a single or few large files. This allows the distributed file system to process the data more efficiently, as the number of files is no longer excessive relative to the number of available nodes, eliminating bottlenecks and reducing processing time.
3Ease of operation
If multiple small files are processed individually through map-reduce, then data processing can be performed, but resource usage increases
Solution Approach 1:
The patent merges small files into a larger file to reduce the total number of map-reduce operations required. Instead of launching separate map-reduce jobs for each small file, the system performs one map-reduce operation on the merged file, significantly reducing computational resource usage, memory allocation, and energy consumption while maintaining the same data processing capability.
Data Source
AI summary
A system and method includes increasing processing efficiency of small files. The method includes generating a file including a plurality of file identifiers that identify a plurality of files in a file system. The method includes causing an execution of a map operation based on the file. The map operation distributes the plurality of file identifiers in a plurality of groups across a plurality of nodes of a computer cluster. The method includes causing an execution of a reduce operation on the plurality of groups of the plurality of file identifiers by the plurality of nodes of the computer cluster. The reduce operation for a first group of the plurality of groups includes reading data of first files of the plurality of files from the file system based on the first group of file identifiers and merging the data of the first files into an output file.