Single Instance Database Chunk Folder File System Limits
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data management systems face challenges in efficiently storing large numbers of files across multiple computing systems, leading to excessive file creation on storage devices, which can exceed the capacity limits of file systems and result in performance degradation.
Innovation Solution
The implementation of a single instance database and sparse files to reduce data density on archival media, where a chunk folder contains a container file for single instanced data, a container file for non-single instanced data, and an index file to track data locations, allowing multiple data objects to be stored in a small number of files, thereby reducing the overall number of files stored on the file system.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If a separate folder is created for each backup or copy job, then data organization and retrieval are simplified, but the number of files on the storage device increases excessively, potentially exceeding file system capacity limits
Solution Approach 1:
The patent merges multiple backup jobs into a single folder structure, where each folder contains multiple data objects from different backup operations. Instead of creating separate folders for each backup job, the system consolidates data objects into shared folders, dramatically reducing the total number of folders and files on the storage device while maintaining organized access through metadata tracking.
2Reliability
If millions of files are stored on the storage device, then comprehensive backup coverage is achieved, but file system performance degrades and capacity limits are exceeded
Solution Approach 1:
The patent uses metadata copying and referencing instead of physical file duplication. Each data object is stored once in a shared folder, and multiple backup jobs reference the same physical files through metadata entries. This approach achieves comprehensive backup coverage across multiple jobs while maintaining low file system overhead, as the actual data files are not replicated but logically referenced by multiple backup operations.
3Quantity of substance
If data density on archival media is increased, then storage capacity is optimized, but the ability to prune and recover space efficiently is reduced
Solution Approach 1:
The patent segments data storage into discrete data objects with individual metadata tracking, allowing selective pruning of specific files while maintaining others. Each data object is independently addressable through metadata, enabling efficient space recovery operations where individual files can be identified, evaluated for retention, and removed without affecting other data objects. This segmentation maintains high storage density while preserving granular control for space management.
Data Source
AI summary
Described in detail herein are systems and methods for managing single instancing data. Using a single instance database and other constructs (e.g. sparse files), data density on archival media (e.g. magnetic tape) is improved, and the number of files per storage operation is reduced. According to one aspect of a method for managing single instancing data, for each storage operation, a chunk folder is created on a storage device that stores single instancing data. The chunk folder contains three files: 1) a file that contains data objects that have been single instanced; 2) a file that contains data objects that have not been eligible for single instancing; and 3) a metadata file used to track the location of data objects within the other files. A second storage operation subsequent to a first storage operation contains references to data objects in the chunk folder created by the first storage operation instead of the data objects themselves.