Metric-Learned Compute-Storage Pairing for Faster Data Transfer
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional methods for pairing compute nodes with storage nodes in distributed computing systems rely on physical distance, leading to sub-optimal pairings that inefficiently utilize computing resources due to neglecting other performance factors.
Innovation Solution
Implementing a machine-learned node pairing optimization model that processes historical performance metrics using metric learning to predict optimal compute-storage pairings based on learned distances in an embedding space, rather than physical geography.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If physical distance is used for pairing compute nodes with storage nodes, then pairing simplicity is improved, but data transfer performance deteriorates
Solution Approach 1:
The patent transforms the pairing criterion from physical distance to a learned metric based on historical performance data. The machine learning model processes multiple performance parameters (data transfer rates, latency, error rates) to generate an optimized pairing metric that replaces simple physical distance measurement, thereby improving data transfer performance while maintaining pairing simplicity through automated model-based decisions.
Solution Approach 2:
The patent introduces a machine learning model as an intermediary between compute nodes and storage nodes. This model acts as a mediator that processes historical performance data and generates optimized pairing recommendations, replacing the direct physical distance-based pairing approach and enabling performance-optimized pairings without complex manual intervention.
2Device complexity
If physical distance-based pairing is used, then implementation complexity is reduced, but resource utilization efficiency deteriorates
Solution Approach 1:
The patent implements preliminary action by pre-processing historical performance data and training the machine learning model in advance. The model learns optimal pairing patterns from historical data before actual node pairing decisions are made. This preliminary training phase enables the system to make efficient, optimized pairing decisions during operation without complex real-time calculations, thereby improving resource utilization while maintaining manageable implementation complexity.
Solution Approach 2:
The system performs self-service by automatically analyzing historical performance metrics and generating optimized pairings without requiring manual intervention. The machine learning model autonomously processes performance data, identifies patterns, and recommends pairings that optimize resource utilization, eliminating the need for complex manual configuration while improving efficiency.
3Productivity
If comprehensive performance metrics are processed, then pairing optimization is improved, but computational overhead increases
Solution Approach 1:
The patent applies preliminary action by pre-training the machine learning model on historical performance data before deployment. During operation, the trained model makes rapid pairing decisions based on learned patterns, avoiding the need for computationally intensive real-time analysis of all performance metrics. This shifts the computational burden to the offline training phase, enabling optimized pairings with minimal runtime overhead.
Solution Approach 2:
The patent transforms multiple performance parameters into a single optimized pairing score through the machine learning model. Instead of processing and comparing multiple individual metrics in real-time, the model has already learned the optimal weightings and relationships between parameters during training, converting complex multi-parameter analysis into efficient single-score comparisons during operation.
Data Source
AI summary
Historical performance metrics are obtained for a first compute node located in a first physical geographic location and a first storage node located in a second physical geographic location different than the first physical geographic location. The historical performance metrics are processed with a machine-learned node pairing optimization model to obtain a training output indicative of a predicted data transfer performance for a compute-storage pairing comprising the first compute node and the first storage node. A training process is performed to train the machine-learned node pairing optimization model based at least in part on the training output indicative of the predicted data transfer performance for the compute-storage pairing.