MapReduce Task Parallelism for Distributed Storage Overhead
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed data storage systems face inefficiencies in read and write operations due to overhead associated with task creation and termination, and inefficient processing and network resource usage, particularly when nodes access non-local storage.
Innovation Solution
The techniques involve efficient computing task and thread usage by handling multiple operations in parallel, reducing overhead through the use of data block distribution maps to manage data blocks across nodes, and employing multiple threads within tasks to minimize processing and network resources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional MapReduce implementations are used for reading and writing files to distributed storage, then data can be stored across multiple servers in parallel, but overhead associated with task creation and termination reduces efficiency
Solution Approach 1:
The patent combines multiple read or write operations into a single task, allowing the task to handle multiple operations before termination. This merging approach reduces the frequency of task creation and termination events, thereby reducing overhead and improving overall read/write efficiency in the distributed storage system.
2Productivity
If nodes access non-local storage in distributed data storage systems, then data can be distributed across multiple nodes, but processing and network resource usage increases
Solution Approach 1:
The patent employs data block distribution maps that are pre-computed and stored, identifying which data blocks are locally stored by each node before actual read/write operations occur. This preliminary action allows the system to quickly determine local data locations without requiring real-time network queries or complex processing during data access operations.
3Device complexity
If single operations are handled by individual tasks in MapReduce, then task structure is simple, but overhead from frequent task creation and termination increases
Solution Approach 1:
The patent designs tasks with multi-functionality, enabling each task to perform multiple read or write operations across different data blocks. This universal task structure maintains relative simplicity while significantly improving operation throughput by reducing the total number of tasks that need to be created and managed in the distributed system.
Data Source
AI summary
Described herein are techniques for reading data from a distributed storage system and for writing data to a distributed storage system. The disclosed techniques make use of efficient computing task and thread usage to minimize or reduce overhead and improve read or write efficiency. For example, read or write tasks may handle multiple read or write operations instead of just a single operation, which may reduce overhead associated with task creation and termination. Additionally, operations within a single task may be processed in parallel. For example, the disclosed techniques provide MapReduce implementations useful in Apache Hadoop that perform better than previous MapReduce implementations.