Job Management Server Staging Buffer Load Distribution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In parallel processing systems, maintaining the load of metadata servers at a proper ratio to their throughput is challenging, leading to potential bottlenecks and uneven access distribution, which can result in performance issues and difficulty in controlling load concentrations.
Innovation Solution
The system employs a job management server that calculates execution end times for jobs, acquires load information from metadata-process execution nodes, and generates schedule data to assign staging tasks to staging buffer servers when the load exceeds a threshold, ensuring that metadata processing is distributed effectively across nodes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If staging tasks are assigned to metadata servers, then file access speed is improved, but metadata server load increases and may exceed throughput capacity
Solution Approach 1:
The system segments metadata server functions by introducing dedicated staging buffer servers that handle staging tasks separately from metadata processing. This divides the original monolithic metadata server into specialized components, allowing staging operations to be performed without overwhelming the metadata processing capacity.
Solution Approach 2:
Staging buffer servers act as intermediary components between the file system and metadata servers. These intermediaries handle the staging tasks that would otherwise burden the metadata servers, protecting them from excessive load while still enabling fast file access through dedicated staging infrastructure.
2Productivity
If more nodes are added to handle staging tasks, then staging capacity is improved, but system complexity and load management difficulty increase
Solution Approach 1:
Staging buffer servers are designed with multi-functionality, serving both as staging execution nodes and as part of the overall file system infrastructure. This universal design allows the same nodes to perform multiple roles, increasing staging capacity without proportionally increasing system complexity.
Solution Approach 2:
The job management server implements load management that monitors the status of metadata servers and staging buffer servers, using feedback information to dynamically assign staging tasks. This feedback mechanism coordinates multiple nodes efficiently, managing complexity through intelligent control rather than simple scaling.
3Loss of time
If staging tasks are performed immediately, then file access responsiveness is improved, but load concentration on metadata servers occurs
Solution Approach 1:
The system performs preliminary actions by pre-allocating staging buffer servers and preparing staging infrastructure before metadata server load becomes excessive. Job scheduling considers future load patterns, assigning staging tasks in advance to appropriate buffer servers, preventing load concentration while maintaining responsiveness.
Solution Approach 2:
Staging buffer servers serve as intermediaries that decouple the timing of staging operations from metadata server availability. They buffer staging tasks, allowing immediate file access preparation without directly impacting metadata server load, thus resolving the timing conflict between responsiveness and load distribution.
Data Source
AI summary
An information processing apparatus includes: a processor performs a scheduling process of scheduling a job for nodes and including: calculating, when one node executes a first job, a job execution end time when execution of the first job is completed by referring an execution history in which an execution time of a job is recorded; acquiring, from a load management node that manages a load of a metadata-process execution node which performing metadata processing to access metadata of a file among the nodes, the load of the metadata-process execution node at the job execution end time; and generating, when the load is equal to or more than a threshold, schedule data to cause a staging execution node which performs the metadata processing produced by staging, at the job execution end time, the metadata processing based on staging to a file having an execution result of the first job.