Time-Synchronized Storage Delivery for Multi-AI Training IO
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current AI model training processes face challenges in efficiently managing large data sets and synchronizing data delivery across multiple AI processing units, leading to inefficiencies in training operations.
Innovation Solution
The implementation of a method that assigns timestamps to input/output commands based on bandwidth allocation for each AI model training operation, allowing for time-synchronized storage delivery by identifying AI model processing units through global flow identifiers and managing data transmission through a network interface, ensuring prioritization and efficient data exchange.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is delivered to multiple AI processing units without time synchronization, then data transmission can proceed in parallel, but training efficiency deteriorates due to unsynchronized data delivery causing idle waiting time
Solution Approach 1:
The system performs preliminary actions by calculating and assigning timestamps to IO commands before execution. The storage device receives IO commands, calculates appropriate timestamps based on bandwidth allocations and training operation priorities, and queues commands with their assigned timestamps. This preliminary timestamp assignment ensures that when data is retrieved and transmitted to AI processing units, the synchronization is already established, preventing idle waiting time and maintaining high training efficiency.
2Productivity
If bandwidth is allocated to different AI model training operations, then resource utilization improves, but system complexity increases due to the need for timestamp calculation and synchronization management
Solution Approach 1:
The storage device performs self-service by autonomously calculating timestamps for IO commands based on pre-configured bandwidth allocations and training operation priorities. When the storage device receives an IO command, it automatically determines the appropriate timestamp using the stored bandwidth information and command priority, without requiring external coordination or complex synchronization protocols. This self-service approach simplifies the overall system architecture while maintaining effective resource utilization and time synchronization.
3Reliability
If timestamps are assigned to IO commands based on bandwidth allocation, then data delivery synchronization improves, but processing overhead increases due to timestamp calculation and management
Solution Approach 1:
The system performs preliminary actions by pre-storing bandwidth allocation information and training operation priority levels in the storage device before IO operations begin. When IO commands are received, the storage device quickly retrieves the relevant bandwidth and priority information from its stored data structures and calculates timestamps using simple lookup and arithmetic operations. This preliminary preparation significantly reduces the processing overhead during actual IO operations while maintaining reliable data delivery synchronization.
Data Source
AI summary
A method includes receiving, at a first computing device, a first input/output (IO) command from a first artificial intelligence processing unit (AI PU), the first IO command associated with a first AI model training operation. The method further includes receiving, at the first computing device, a second IO command from a second AI PU, the second IO command associated with a second AI model training operation. The method further includes assigning a first timestamp to the first IO command based on a first bandwidth assigned to the first AI model training operation. The method further includes assigning a second timestamp to the second IO command based on a second bandwidth assigned to the second AI model training operation.