Bulk Copy Request Partitioning for Storage Prefetch Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data copy methods in storage environments consume significant host computer resources and can lead to performance issues due to out-of-order requests when dealing with large requests, particularly when multiple requests complete around the same time on different cores, confusing prefetchers and affecting performance.
Innovation Solution
A method that partitions a copy request into multiple requests, issues a first number of these requests, detects completion of a second number of requests (greater than one but less than the first number), and then issues a third number of requests upon detecting completion of the second number, optimizing data transfer and prefetching by ensuring sequential order.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If data is copied from source to target through host computer buffering, then data transfer can be performed, but host computer resources (memory, bandwidth) are significantly consumed
Solution Approach 1:
The patent extracts the data transfer function from the host computer by implementing offloaded data transfer (ODX) at the storage system level. The storage system directly copies data between source and target devices without buffering through the host, removing the host from the data path while maintaining transfer capability. This resolves the contradiction by eliminating host resource consumption without sacrificing data transfer functionality.
Solution Approach 2:
The storage system acts as an intermediary between source and target devices, performing the copy operation autonomously. Instead of the host mediating every data movement, the storage system's controller directly manages the data transfer between devices, using tokens to identify and relocate data. This intermediary approach enables efficient data transfer while preserving host resources.
2Productivity
If multiple copy requests are issued in parallel on different processor cores, then throughput is improved, but requests complete out of order which confuses prefetchers and degrades performance
Solution Approach 1:
The patent segments the parallel request processing into batches, where each batch is assigned to a specific processor core. By dividing the workload into discrete batches handled sequentially by different cores, the system maintains both parallel processing capability and request order integrity. This segmentation resolves the contradiction by allowing throughput improvement through parallelism while preserving reliability through ordered batch processing.
Solution Approach 2:
The system performs preliminary actions by pre-assigning batches of requests to specific processor cores before execution. This preliminary organization ensures that when requests are processed in parallel, their completion order is predictable and maintains sequential logic. The prefetcher can anticipate request patterns because the batching strategy is established in advance, resolving the order reliability issue while maintaining throughput.
3Quantity of substance
If large copy requests are partitioned into multiple smaller requests, then resource consumption is reduced, but the complexity of managing request sequences increases
Solution Approach 1:
The patent implements feedback mechanisms where the storage system monitors request completion status and dynamically adjusts the issuance of subsequent requests. When a batch of requests is issued, the system waits for their completion before issuing the next batch, creating a feedback loop that maintains order without requiring complex management. This feedback-based approach resolves the contradiction by simplifying request management while handling partitioned data effectively.
Data Source
AI summary
Techniques are described for performing a copy operation. A first request is received to perform a copy operation to copy source data from a source area to a target area. The first request is partitioned into a plurality of other requests. Each of the plurality of other requests being a request to copy a portion of source data located in the source area to a corresponding location in the target area. A first number of the other requests is issued. Responsive to the issued requests, a copy operation is performed to copy a portion of the source data from the source area to the target area. Completion of a second number of the issued other requests is detected. The second number corresponds to a fraction of the first number such that the second number is greater than one and less than the first number. Upon detecting completion of the second number, a third number of the other requests is issued to copy another portion of the source data from the source area to the target area. The third number corresponding to the second number of the issued other requests detected as complete.