Consolidating Small Files for Magnetic Tape Migration
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data storage systems face inefficiencies when migrating small files to magnetic recording tape, resulting in high latency due to the need for frequent starting and stopping of the tape, which is costly and resource-intensive.
Innovation Solution
A method that performs queries to find files meeting predefined criteria, such as file size, and uses query expansion profiles to consolidate files into a single consolidation file, which is then written to magnetic tape, reducing individual write operations and minimizing latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If small files are migrated individually to magnetic tape, then data migration can be performed, but latency increases due to frequent starting and stopping of the tape
Solution Approach 1:
The patent merges multiple small files into a single consolidated file before migration to magnetic tape. The system identifies multiple small files meeting predefined criteria, combines them into one consolidation file, and performs a single tape write operation. This eliminates the need to repeatedly start and stop the tape for each individual file, thereby reducing latency while maintaining data migration capability.
2Productivity
If multiple individual write operations are performed for small files, then all files can be migrated, but operational costs and resource consumption increase
Solution Approach 1:
The system consolidates multiple small files into a single file before migration, reducing the number of tape write operations from multiple individual writes to one consolidated write. This merging approach maintains complete data migration while significantly reducing operational costs and resource consumption associated with repeated tape operations.
Solution Approach 2:
The system performs preliminary consolidation of small files into a single file before the actual tape migration process. By pre-processing and combining files on the storage system before initiating tape write operations, the system minimizes the number of expensive tape operations required, thereby reducing operational costs while ensuring complete data migration.
3Loss of time
If files are consolidated into a single write operation, then latency is reduced, but file selection and consolidation complexity increases
Solution Approach 1:
The system segments the file selection and consolidation process into distinct, manageable components: a query module that identifies small files meeting predefined criteria, a consolidation module that combines selected files, and a migration module that performs the tape write. This segmentation reduces complexity by creating modular, independent functions that can be executed sequentially with clear interfaces between them.
Solution Approach 2:
The system performs preliminary file identification and selection using predefined criteria before the consolidation process begins. By establishing clear selection rules in advance and pre-identifying candidate files, the system simplifies the subsequent consolidation step and reduces the complexity of real-time decision-making during the migration process.
Data Source
AI summary
A computer-implemented method according to one embodiment includes performing a query for finding files that meet at least one predefined criterion. The at least one predefined criterion includes a file size that is less than a predefined fraction of a predetermined threshold file size. Performing the query includes sequentially performing additional querying in response to finding a first file that meets the at least one predefined criterion. As files are found in the additional querying, for each sequence of additional querying, the method includes determining whether a total storage size of files found during performance of the query exceeds the predetermined threshold file size. In response to a determination that the total storage size of the files found during performance of the query exceeds the predetermined threshold file size, the files are consolidated into a consolidation file. The method further includes writing the consolidation file to a magnetic recording tape.