Chord DHT MapReduce Double-Layered Ring Load Balancing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Hadoop distributed file systems face performance degradation due to central management and uneven data distribution, leading to load imbalance and reduced cache hit rates during MapReduce tasks.
Innovation Solution
A chord distributed hash table based MapReduce system with a double-layered ring structure, where data is managed across multiple servers with in-memory caches, dynamically adjusting hash key ranges based on access frequency to optimize task scheduling and cache utilization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If a central file system manager is used to manage all data in Hadoop, then data management is centralized and simplified, but the manager manages an excessively large amount of data, causing performance degradation
Solution Approach 1:
The patent divides the centralized file system into multiple distributed file systems, each managed by a separate file system manager. Each manager handles only the data within its own namespace, segmenting the previously monolithic management task into smaller, manageable units that can operate in parallel without overwhelming any single manager.
2Ease of manufacture
If files are divided and distributed without considering content, then data is evenly distributed across servers, but necessary data may be stored in only a specific server causing load imbalance
Solution Approach 1:
The patent enables each file system to maintain its own local data distribution characteristics based on access patterns and requirements. Instead of forcing uniform distribution across all servers, each distributed file system can optimize its local storage strategy, allowing frequently accessed data to be replicated or positioned strategically while maintaining overall system balance.
3Speed
If MapReduce tasks are executed only on servers storing specific required data, then data locality is optimized, but load balance cannot be achieved and performance degrades
Solution Approach 1:
The patent makes each server capable of executing MapReduce tasks for multiple distributed file systems, not just its own. A server can process tasks across different file systems by accessing data through the distributed file system interfaces, enabling load balancing while maintaining data locality benefits. This multi-functionality allows flexible task allocation regardless of which specific file system the data belongs to.
Data Source
AI summary
A chord distributed hash table based MapReduce system includes multiple servers and a job scheduler. The multiple servers include file systems and in-memory caches storing data based on a chord distributed hash table. The job scheduler manages the data stored in the file systems and the in-memory caches in a double-layered ring structure, when receiving a data access request for a specific file from an outside. The job scheduler allocates MapReduce tasks to the servers that store the file for which the data access request has been received among the multiple servers, and outputs a result value obtained by performing the MapReduce tasks in response to the data access request.