Adaptive Map Task Input Split Sizing in MapReduce
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The MapReduce architecture faces inefficiencies due to non-optimal settings of systematic parameters, leading to repeated disk read/write processes caused by mismatched input split sizes and memory sizes for Map tasks, resulting in low processing efficiency and resource waste.
Innovation Solution
A method and apparatus for resource management that determine the ratio of input and output data amounts, average record size, and allocate appropriate memory size to Map tasks, thereby optimizing input split sizes to match memory availability, reducing disk spills and improving processing efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If input split size is set to a fixed large value (e.g., 1000 MB), then the number of Map tasks is reduced, but the output data amount may exceed memory size causing repeated disk read/write operations
Solution Approach 1:
The patent applies dynamics by making the input split size adaptive rather than fixed. The system dynamically adjusts the input split size based on the actual memory size of Map tasks and the output data amount. When memory is sufficient, larger split sizes are used to reduce the number of Map tasks. When memory is limited, the system reduces split size to prevent output data from exceeding memory capacity, thereby avoiding repeated disk operations. This dynamic adjustment resolves the contradiction between reducing task overhead and preventing disk I/O bottlenecks.
Solution Approach 2:
The patent changes the parameter of input split size based on other system parameters such as memory size and output data amount. Instead of using a hardcoded split size, the system calculates the optimal split size by considering the relationship between input data amount, output data amount, and available memory. This parameter change approach allows the system to adapt to different hardware configurations and workloads, resolving the contradiction between efficient resource utilization and avoiding disk spills.
2Device complexity
If input split size is increased to reduce the number of Map tasks, then task management overhead is reduced, but memory may be insufficient to hold all output data causing spills to disk
Solution Approach 1:
The patent changes the input split size parameter dynamically based on memory size and output data characteristics. By calculating the expected output data amount and comparing it with available memory, the system adjusts the split size to ensure that output data can be held in memory without spilling to disk. This parameter adjustment resolves the contradiction between reducing the number of tasks and ensuring memory sufficiency.
Solution Approach 2:
The system implements feedback by monitoring the relationship between input split size, output data amount, and memory usage. Based on this feedback, the system adjusts future split size decisions to prevent memory overflow. This feedback mechanism ensures that the number of Map tasks is optimized while maintaining reliable memory usage, avoiding both excessive task overhead and disk spills.
3Ease of operation
If systematic parameters are set empirically by user experience, then configuration is simple, but the parameters are not optimal for specific node processing capacity and resource situations
Solution Approach 1:
The patent applies self-service by enabling the system to automatically determine optimal input split sizes without requiring manual configuration by users. The system calculates the optimal split size based on its own monitoring of memory size, output data amount, and processing characteristics. This self-service approach eliminates the need for empirical user configuration while achieving optimal processing efficiency, resolving the contradiction between ease of operation and productivity.
Solution Approach 2:
The system uses feedback from actual processing performance and resource usage to automatically adjust parameters. Instead of relying on static empirical settings, the system continuously monitors and adapts the input split size based on real-time conditions. This feedback-driven automatic configuration achieves both ease of operation (no manual tuning needed) and high productivity (optimal parameters for each situation).
Data Source
AI summary
A method for resource management in MapReduce architecture. The method includes: determining a ratio r of an input data amount of a Map task and an output data amount of the Map task and an average size R of a record in Map output results; determining a memory size Memory_size that can be allocated to the Map task corresponding to a Map slot; according to the determined r, R and Memory_size, determining an input split size appropriate for the Map task; and allocating an input split with the input split size in a MapReduce job to be processed to the Map task occupying the Map slot. An apparatus for same is also provided.