Archive Node Optimized Schedule for Batch Analytics Data Retrieval
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current batch analytics methods face inefficiencies and high overhead due to the slow retrieval times of archived data from archive tiers, which require data to be copied to analytics tiers for processing, leading to significant delays and storage costs.
Innovation Solution
An optimized schedule is generated by an archive node computing device for retrieving archived files from archive storage devices, allowing direct access and prefetching based on the archive data layout, enabling parallel retrieval and reducing the need for data copying.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If archived data is stored on archive tier storage media (tapes/spun down disks), then storage cost is reduced and long term retention is enabled, but retrieval speed decreases and time to first byte increases significantly
Solution Approach 1:
The system performs preliminary actions by pre-fetching archived files to the analytics tier before they are actually needed for processing. The analytics platform communicates the order of file access to the archive tier, which then prefetches files in advance and stages them in the analytics tier. This eliminates the need for slow real-time retrieval during job execution, resolving the contradiction between cost-effective archiving and fast access.
2Ease of operation
If archived data is copied to analytics tier before processing (ingest-then-compute), then data can be processed directly, but significant delay is introduced and additional storage space is required on analytics tier
Solution Approach 1:
The system performs preliminary actions by pre-fetching archived files to the analytics tier before they are actually needed for processing. The analytics platform communicates the order of file access to the archive tier, which then prefetches files in advance and stages them in the analytics tier. This eliminates the need for slow real-time retrieval during job execution, resolving the contradiction between cost-effective archiving and fast access.
Solution Approach 2:
The system uses selective copying by copying only the specific archived files needed for a job to the analytics tier, rather than copying entire datasets. The analytics platform requests only the necessary archived files from the archive tier, and these files are copied/staged in the analytics tier for processing. This minimizes the storage overhead and time delay associated with the ingest-then-compute approach.
3Ease of operation
If archived data is accessed through NFS mounting from archive tier, then direct access is enabled, but high time to first byte prevents significant performance improvement
Solution Approach 1:
The system performs preliminary actions by pre-fetching archived files to the analytics tier before they are actually needed for processing. The analytics platform communicates the order of file access to the archive tier, which then prefetches files in advance and stages them in the analytics tier. This eliminates the need for slow real-time retrieval during job execution, resolving the contradiction between cost-effective archiving and fast access.
Data Source
AI summary
A method, non-transitory computer readable medium, and archive node computing device that receives an indication of each of a plurality of archived files required to service a job from one of a plurality of compute node computing devices of an analytics tier. An optimized schedule for retrieving the archived files from one or more archive storage devices of an archive tier is generated. The optimized schedule is provided to the one of the plurality of compute node computing devices. Requests for the archived files received from the one of the plurality of compute node computing device and at least one other of the plurality of compute node computing devices, wherein the requests are sent according to the optimized schedule.