Dictionary-Based Data Compression for Archiving
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional data backup systems face inefficiencies due to unnecessary redundancy, high storage requirements, and bandwidth issues, particularly in managing large datasets, as they often duplicate unchanged data and require transferring entire datasets for backup and archiving, leading to increased costs and access challenges.
Innovation Solution
The implementation of adaptive content factoring, which uses a heuristic method to optimize data storage by identifying and storing unique data once, employing dictionary-based compression and metadata to reconstruct data versions, and utilizing hash codes to identify changes, thereby reducing redundant storage and bandwidth needs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional backup systems duplicate all digital data, then data retention requirements are met, but storage requirements and processing bandwidth increase unnecessarily
Solution Approach 1:
The patent extracts only the unique portions of data that have changed between backup sets, rather than duplicating entire datasets. By identifying and storing only the differential content, the system meets retention requirements while dramatically reducing storage requirements.
Solution Approach 2:
The patent segments data into identifiable portions or blocks, allowing comparison between backup sets to determine what has changed. This segmentation enables the system to process and store only the necessary differential data rather than treating entire datasets as monolithic units.
2Reliability
If traditional backup systems duplicate all digital data, then data retention requirements are met, but processing and bandwidth requirements increase
Solution Approach 1:
The system extracts only the changed portions of data for processing and transmission, rather than processing entire datasets. This extraction approach maintains data retention compliance while reducing processing and bandwidth requirements to minimal levels.
3Reliability
If traditional backup systems store multiple backup tapes, then data archiving is achieved, but access time and operational complexity increase
Solution Approach 1:
The patent merges multiple backup sets into a single consolidated storage structure by storing differential data in sequence. This consolidation allows the system to retrieve archived data from a single location rather than searching through multiple separate tapes, dramatically improving access time and operational simplicity.
4Reliability
If traditional backup systems duplicate unchanged data, then data integrity is maintained, but redundancy increases storage needs
Solution Approach 1:
The system takes out only the unique changed portions from each backup set, storing them sequentially while maintaining references to preserve data integrity. This approach eliminates redundant storage of unchanged data while ensuring that complete data sets can be reconstructed when needed.
Data Source
AI summary
Provided are systems and methods for use in data archiving. In one arrangement, compression techniques are provided wherein an earlier version of a data set (e.g., file folder, etc) is utilized as a dictionary of a compression engine to compress a subsequent version of the data set. This compression identifies changes between data sets and allows for storing these differences without duplicating many common portions of the data sets. For a given version of a data set, new information is stored along with metadata used to reconstruct the version from each individual segment saved at different points in time. In this regard, the earlier data set and one or more references to stored segments of a subsequent data set may be utilized to reconstruct the subsequent data set.