Cluster Resource Planning via Proposal Database for Spark Hadoop
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current cluster computing architectures, such as Spark and Hadoop, lack a method for users to dynamically adjust resource allocation based on specific task conditions and expected work times, leading to suboptimal performance under hardware limitations.
Innovation Solution
A resource planning method and system that uses a proposal database to determine optimal resource allocation by capturing setting values and algorithm features, establishing training models, and adjusting worker node resources in real time based on task conditions and expected work times, with the ability to train and refine proposals for improved performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If resource allocation is based on priority setting and weighting under hardware resource limitation, then resource allocation can be managed, but the allocation is not optimal for specific task conditions and expected work times
Solution Approach 1:
The system performs preliminary actions by establishing a proposal database that stores pre-calculated resource allocation proposals for different task conditions and expected work times. When a user submits a resource planning request, the system queries this database to find the most suitable pre-prepared proposal, avoiding the need to perform complex optimization calculations in real-time. This resolves the contradiction by preparing resource allocation schemes in advance, improving responsiveness while maintaining allocation quality.
Solution Approach 2:
The system dynamically adjusts resource allocation by querying the proposal database with specific task conditions and expected work times, then selecting the most appropriate pre-calculated proposal. This dynamic query and selection process allows the system to adapt to different task requirements without performing full optimization calculations each time, resolving the contradiction between dynamic adaptability and computational complexity.
2Productivity
If more hardware resources are provided to the Spark platform or Hadoop platform, then computing capability can be improved, but there is no guidance for users on how much resources to provide
Solution Approach 1:
The system enables self-service by automatically querying the proposal database and selecting the optimal resource allocation proposal based on the user's task conditions and expected work time requirements. The system then provides this optimized proposal to the user, eliminating the need for users to manually determine how much hardware resources to provide. This resolves the contradiction by making the system intelligent and autonomous in resource planning, improving both computing capability optimization and user ease of operation.
Solution Approach 2:
The system implements feedback by using the query results from the proposal database to provide specific, data-driven resource allocation recommendations to users. The feedback loop allows the system to learn from task conditions and expected work times, then provide increasingly accurate resource planning guidance. This resolves the contradiction by transforming the system from passive resource allocation to active, intelligent recommendation, improving both computing capability and user ease of operation.
Data Source
AI summary
A resource planning method, system, and apparatus for a cluster computing architecture are provided. The resource planning apparatus establishes at least one training model based on a training platform and corresponding setting values and algorithm features, such that a master node apparatus operates based on each training model to obtain operating time, and the corresponding operating time of each training model is stored in a proposal database. Afterwards, a proposal is obtained from the proposal database according to a task condition and an expected total work time, such that the master node apparatus decides a resource allocation of the cluster computing architecture.