Parallel Iterator for ML Data Retrieval Bottlenecks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional memory sub-systems for machine learning frameworks retrieve data in a single-threaded manner, leading to performance degradation due to storage bottlenecks, where processors can process data faster than memory devices can provide it, especially when dealing with large datasets.
Innovation Solution
Implementing a parallel iterator that logically partitions data and initiates multiple I/O threads to retrieve data in parallel, leveraging multiple I/O queues in memory devices to improve throughput and prevent bottlenecks by matching processing speeds with storage and memory subsystems.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If single-threaded data retrieval is used, then system simplicity is maintained, but data retrieval performance degrades due to storage bottlenecks
Solution Approach 1:
The patent divides the data retrieval operation into multiple parallel threads, where each thread handles a portion of the data retrieval independently. This segmentation allows simultaneous access to multiple I/O queues in the memory device, thereby improving data retrieval performance without requiring fundamental changes to the memory subsystem architecture.
Solution Approach 2:
The patent introduces parallelism as a new dimension for data retrieval by utilizing multiple I/O queues that operate simultaneously. This transforms the single-threaded sequential access model into a multi-threaded parallel access model, effectively adding a temporal dimension to the data retrieval process and overcoming the storage bottleneck.
2Speed
If processors process data faster than memory devices can provide it, then processing speed is improved, but storage bottlenecks occur
Solution Approach 1:
The patent initiates multiple I/O threads in advance to retrieve data from the memory device before the processor needs it for processing. By pre-fetching data through parallel threads, the system ensures that data is ready in the buffer when the processor requires it, eliminating waiting time and preventing storage bottlenecks while maintaining high processing speed.
Solution Approach 2:
The patent maintains continuous data flow to the processor by using multiple parallel I/O threads that continuously retrieve data from the memory device. This continuous action ensures that the processor never idle due to data unavailability, matching the processing speed with data throughput and eliminating storage bottlenecks.
Data Source
AI summary
A request to retrieve data from a memory device of a memory sub-system can be received from a machine learning (ML) framework executing on a host system, where the data comprises a plurality of logical partitions. A set of parallel I/O threads can be initiated to retrieve the data from the memory device, where each I/O thread of the set of parallel I/O threads retrieves a different portion of the data from a different corresponding logical partition and stores the different portion of the data in a I/O buffer of a set of I/O buffers corresponding to the set of I/O threads in parallel. The different portion of the data can be successively provided from each I/O buffer to the ML framework, where the set of parallel I/O threads is to continually retrieve the data from the memory device until all of the data from the logical partitions has been provided to the ML framework.