File System Delta Compression for Redundant Data Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data processing systems face challenges in efficiently storing large volumes of highly redundant unstructured data files, such as genomic sequence data, due to limited storage capacity and transmission bandwidth, especially in mobile devices and wide-area transfers, where existing compression techniques do not adequately reduce file sizes.
Innovation Solution
A method and system that associate similar data files with reference data files, using a file association module to group and compress data files via delta compression, storing only the delta between new and existing data, thereby reducing storage needs and maintaining file system integrity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If traditional compression techniques are used, then some storage space is saved, but compression factors are insufficient for highly redundant data
Solution Approach 1:
The patent creates a reference copy of the base file and stores only the delta (differences) between the new file and the reference. This copying approach achieves compression factors of 90-99% by storing minimal difference data rather than complete file copies, resolving the contradiction between storage space savings and compression effectiveness.
Solution Approach 2:
The system changes the storage parameter from storing complete file copies to storing only delta differences. By transforming the storage representation from full copies to differential data, the system achieves dramatically higher compression factors while maintaining data accessibility through reference to the base file.
2Reliability
If data is stored in original form, then data accessibility is maintained, but storage capacity is wasted
Solution Approach 1:
The system maintains data accessibility by preserving the reference base file in its original form while storing new files as deltas. This allows the reference to remain fully accessible while compressed delta versions save storage capacity, resolving the contradiction between maintaining data accessibility and conserving storage resources.
3Reliability
If complete file copies are stored, then data integrity is ensured, but transmission bandwidth is consumed
Solution Approach 1:
The patent transmits only the delta (difference) data between files rather than complete copies. By copying the reference file locally and transmitting only the minimal delta information, the system ensures data integrity through reference to the complete base file while dramatically reducing transmission bandwidth consumption.
4Quantity of substance
If delta compression is applied, then storage space is minimized, but file system complexity increases
Solution Approach 1:
The system manages file system complexity by maintaining a reference copy mechanism where the base file is stored in its original form and new files are represented as deltas with references to the base. This copying approach simplifies the file system structure compared to storing all files completely, as the reference mechanism provides a natural organizational hierarchy.
Data Source
AI summary
A mechanism is provided for storing data files in a file system. The file system provides a plurality of reference data files, where each reference data file in the plurality of data files represents a group of similar data files. The mechanism creates a new data file and associated the new data file with one reference data file in the plurality of data files thus defining an associated reference data file of the plurality of reference data files. The mechanism informs the file system about the association of the new data file with the associated reference data file. The mechanism compresses the new data file using the associated reference data file thereby forming a compressed data file. The mechanism stores the compressed data file together with information about the association of the new data file with the associated reference data file.