Single Instancing Chunk Folder for File System Bottlenecks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data management systems face challenges in efficiently storing large numbers of files across multiple computing systems, leading to excessive file creation on storage devices, which can overwhelm file systems and reduce performance.
Innovation Solution
The implementation of a single instancing system that uses a chunk folder with three container files - one for single instanced data, one for non-single instanced data, and an index file to track locations, reducing the number of files stored by referencing existing data objects instead of duplicating them.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If each backup or copy job creates a separate folder with its own files, then data organization and retrieval are simplified, but the number of files on the storage device becomes excessively large (millions of files)
Solution Approach 1:
The patent merges multiple backup jobs into a single folder structure, where each folder contains container files that store data from multiple jobs. Instead of creating separate folders for each job, the system combines data from numerous jobs into shared container files, dramatically reducing the total number of files while maintaining organized access through the container file structure.
Solution Approach 2:
The patent uses container files that can store multiple copies of the same data objects from different backup jobs. Rather than creating separate file instances for each copy, the system uses a single container file to hold multiple references or copies of data objects, reducing file proliferation while preserving data integrity and organization.
2Quantity of substance
If file systems store millions of files, then capacity is utilized, but performance degrades and file systems may become incapable of handling the load
Solution Approach 1:
By merging multiple backup jobs into shared container files within single folders, the system reduces the total file count from millions to a manageable number. This consolidation maintains storage capacity utilization while dramatically improving file system performance by reducing the overhead associated with managing an excessive number of individual files.
3Productivity
If single instancing is implemented to reduce file numbers, then file system performance improves, but data organization and tracking become more complex
Solution Approach 1:
The patent segments the data tracking structure into organized folders, where each folder contains container files and associated metadata. This segmentation provides a hierarchical organization that simplifies tracking compared to a flat structure, making the system more manageable while still reducing file proliferation through the container file approach.
Solution Approach 2:
The container files act as intermediaries between the backup jobs and the storage system. Each container file manages multiple data objects from different jobs, providing a structured layer that simplifies tracking and organization. The container file structure serves as a mediator that reduces the complexity of directly managing millions of individual files while maintaining performance benefits.
Data Source
AI summary
Described in detail herein are systems and methods for managing single instancing data. Using a single instance database and other constructs (e.g. sparse files), data density on archival media (e.g. magnetic tape) is improved, and the number of files per storage operation is reduced. According to one aspect of a method for managing single instancing data, for each storage operation, a chunk folder is created on a storage device that stores single instancing data. The chunk folder contains three files: 1) a file that contains data objects that have been single instanced; 2) a file that contains data objects that have not been eligible for single instancing; and 3) a metadata file used to track the location of data objects within the other files. A second storage operation subsequent to a first storage operation contains references to data objects in the chunk folder created by the first storage operation instead of the data objects themselves.