Hierarchical Datapack Storage for File Deduplication and Transfer
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data storage and transfer systems, such as cloud-based revision control systems, suffer from inefficiencies due to significant amounts of duplicate data, leading to increased storage requirements and slower transfer speeds, especially when dealing with multiple versions of files.
Innovation Solution
Implementing a method that partitions datafiles into chunks using natural chunking algorithms and SHA-256 checksums, groups these chunks into 'datapacks', and stores them in a hierarchical tree structure to reduce duplicate data and optimize storage and transfer.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If complete versions of both files are stored, then data integrity is maintained, but storage space increases significantly due to duplicate data
Solution Approach 1:
The patent divides files into smaller chunks using content-defined chunking algorithms. Each chunk is independently hashed and stored, allowing the system to maintain data integrity through checksum verification while storing only unique chunks. This segmentation enables efficient deduplication where identical chunks across different files are stored only once.
Solution Approach 2:
The patent uses cryptographic hashing (SHA-256) to create unique identifiers for each chunk. Instead of storing complete file copies, the system stores chunk references and uses the hash function to verify data integrity. This allows the system to maintain reliability through cryptographic verification while dramatically reducing storage requirements through deduplication.
2Quantity of substance
If lots of small files are transferred to cloud storage, then data can be stored, but transfer performance deteriorates due to the large number of individual file operations
Solution Approach 1:
The patent groups multiple small chunks into larger datapack files for cloud storage. Instead of uploading numerous small files individually, the system combines related chunks into consolidated datapacks, reducing the number of I/O operations and improving transfer performance while maintaining the ability to retrieve individual chunks when needed.
Solution Approach 2:
The patent creates an intermediate hierarchical structure where chunks are organized into datapacks. This segmentation at the datapack level (rather than individual chunk level) optimizes cloud transfer by creating a manageable number of larger files, while the underlying chunk structure remains intact for efficient retrieval and deduplication.
3Quantity of substance
If a file is partitioned into many small chunks, then storage efficiency improves through deduplication, but system complexity increases due to managing numerous small data units
Solution Approach 1:
The patent implements a nested hierarchical structure where chunks are grouped into datapacks, which are then stored in the cloud. This nesting reduces system complexity by creating manageable intermediate layers: individual chunks are hashed and deduplicated, then organized into datapacks for efficient cloud storage. The hierarchy allows the system to manage complexity at each level rather than dealing with all chunks individually.
Solution Approach 2:
The patent introduces datapacks as an intermediary structure between individual chunks and cloud storage. This intermediary layer simplifies cloud I/O operations while maintaining the fine-grained chunk structure needed for deduplication. The datapact acts as a mediator that batches chunks for storage but preserves individual chunk identities for retrieval and deduplication operations.
4Quantity of substance
If sequential transfer of large files is used, then data can be transferred, but transfer time increases and fault tolerance decreases
Solution Approach 1:
The patent divides large files into smaller chunks that can be transferred in parallel. Each chunk is independently processed and transferred, allowing multiple transfer operations to occur simultaneously. This segmentation enables parallel transfer mechanisms that reduce total transfer time and improve fault tolerance, as failed transfers can be retried independently without affecting other chunks.
Solution Approach 2:
The patent combines parallel chunk transfers into a unified datapack structure. Multiple chunks are transferred in parallel and then merged into datapacks for cloud storage. This merging approach maintains the benefits of parallel transfer (reduced time, improved fault tolerance) while organizing the data into a manageable storage structure.
Data Source
AI summary
Embodiments store a datafile. One such embodiment partitions a datafile into chunks. A data structure is constructed that represents the datafile. The data structure includes a hierarchical tree representing the chunks. In turn, respective chunk identifiers (IDs) are generated that correspond to the chunks. Next, non-duplicate chunk(s) are identified from among the chunks based on the generated respective chunk IDs. Based on the constructed data structure and the identified non-duplicate chunk(s), datapack(s) are constructed. The constructed datapack(s) are then stored in memory.