Storage Query Planning for Predictable AI Training Bandwidth
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In data processing sessions, especially AI training and inferencing, storage systems often fail to provide deterministic and predictable performance, leading to unpredictable session completion times and underutilization of resources due to uneven bandwidth allocation across compute resources.
Innovation Solution
A storage query planning method that allocates and schedules storage, network, and compute resources based on dataset requirements, estimating available bandwidth and predicting Quality-of-Service (QoS) to ensure consistent and efficient resource utilization across multiple concurrent processing sessions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If storage resources are allocated without query planning, then resource allocation is simple and fast, but performance is unpredictable and resources are underutilized
Solution Approach 1:
The system performs preliminary actions by allocating and reserving storage resources before processing sessions begin. The storage query manager creates storage query plans that pre-allocate bandwidth and map storage nodes to compute nodes in advance, ensuring predictable performance when sessions execute without needing to negotiate resources during runtime.
Solution Approach 2:
The storage query manager acts as an intermediary between storage resources and processing sessions. It receives storage queries from compute nodes, determines optimal resource allocation, and returns resource maps with assigned storage nodes and bandwidth allocations, mediating the resource allocation process to ensure predictability while managing complexity centrally.
2Productivity
If bandwidth is allocated evenly across all compute resources, then allocation is simple, but storage and network resources are underutilized due to uneven access patterns
Solution Approach 1:
The system applies local quality by allocating bandwidth differently to different compute nodes based on their specific needs and access patterns. The storage query manager evaluates each storage query individually and assigns bandwidth allocations tailored to the particular dataset and compute node, rather than applying uniform allocation across all resources.
Solution Approach 2:
The system changes the bandwidth parameter dynamically based on query characteristics. The storage query manager adjusts bandwidth allocations as parameters in the resource maps, modifying the amount of storage and network bandwidth assigned to different compute nodes according to their specific dataset access requirements and performance needs.
3Loss of time
If storage resources are allocated dynamically during processing sessions, then flexibility is high, but session completion times become unpredictable
Solution Approach 1:
The system performs preliminary resource allocation before processing sessions begin. By creating storage query plans that pre-assign storage nodes and bandwidth allocations, the system eliminates runtime negotiation and ensures consistent session completion times while maintaining the flexibility to allocate resources optimally for each session's specific requirements.
4Reliability
If multiple concurrent processing sessions share storage resources without planning, then resource sharing is efficient, but performance becomes unpredictable due to resource contention
Solution Approach 1:
The storage query manager serves as an intermediary that coordinates resource sharing among multiple concurrent sessions. It receives storage queries from multiple compute nodes, evaluates their resource requirements, and allocates storage bandwidth and nodes in a manner that prevents contention and ensures predictable performance for each session while managing overall resource utilization.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A method may include receiving a request for storage resources to access a dataset for a processing session, allocating, based on the dataset, one or more storage nodes for the processing session, and mapping one or more of the storage nodes to one or more compute nodes for the processing session through one or more network paths. The method may further include returning a resource map of the one or more storage nodes and the one or more compute nodes. The method may further include estimating an available storage bandwidth for the processing session. The method may further include estimating an available client bandwidth. The method may further include allocating a bandwidth to a connection between at least one of the one or more storage nodes and at least one of the one or more compute nodes through one of the network paths.