Dynamic Machine Configuration Matrix for Big Data Jobs
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Enterprises face high costs in maintaining physical machines for big data processing and encounter reliability issues when using interruptible cloud services, as they may be interrupted by other parties paying a higher price, necessitating a method to dynamically identify cost-effective and reliable machine configurations.
Innovation Solution
A system and method that dynamically generates an equivalence matrix to identify optimal and equivalent machine configurations, using historical data and regression algorithms to predict costs and interruption rates, ensuring job completion without interruption by allocating cloud-based machines from third-party services.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If interruptible machines are used for data processing, then cost is reduced, but reliability deteriorates due to potential interruption by other parties
Solution Approach 1:
The system dynamically adjusts machine configuration selection based on real-time equivalence matrix data, allowing the job to adaptively switch between different machine configurations (same capacity, different interruption risks) to maintain reliability while optimizing cost
Solution Approach 2:
The system changes the parameter of machine configuration selection by using an equivalence matrix that identifies multiple configurations with identical processing capacity but different cost and reliability characteristics, allowing flexible parameter adjustment to balance cost and reliability
2Reliability
If optimal machine configuration is sought, then job completion reliability is improved, but device complexity increases due to dynamic generation and evaluation of multiple configurations
Solution Approach 1:
The system creates virtual copies of machine configurations in the equivalence matrix, allowing evaluation of multiple hypothetical configurations without physically provisioning them, thus reducing actual system complexity while maintaining reliability optimization capability
Solution Approach 2:
The system performs preliminary generation of the equivalence matrix and identification of optimal configurations before job execution, allowing the complex evaluation to be done in advance rather than during runtime, reducing operational complexity
Data Source
AI summary
A tangible, non-transitory, machine-readable medium, including machine-readable instructions that, when executed by one or more processors, cause the one or more processors to identify an optimal machine configuration and a number of optimal machines to complete a job. The one or more processors may further dynamically generate a matrix of available equivalent machine configurations and a corresponding number of machines for each available equivalent machine configuration and provide a preferred machine configuration and a number of machines having the preferred machine configuration for completing the job, where the preferred machine configuration comprises the optimal machine configuration or a machine configuration from the dynamically generated matrix.