Chunked Datapack Storage for Deduplicated File Version Transfer
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data storage and transfer systems, such as cloud-based revision control systems, suffer from inefficiencies due to significant amounts of duplicate data, leading to increased storage requirements and slower transfer speeds, particularly when handling multiple versions of files.
Innovation Solution
Implementing a method that partitions datafiles into chunks using natural chunking algorithms and SHA-256 checksums, groups these chunks into 'datapacks', and stores them in a hierarchical tree structure, allowing for deduplication and efficient transfer.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If complete versions of both files are stored, then data integrity is maintained, but storage space is doubled to 2GB with significant duplicate data
Solution Approach 1:
The patent divides files into smaller chunks using content-defined chunking algorithms. Instead of storing complete file versions, only the changed chunks are stored. For example, a 1GB file and a slightly modified version are split into chunks, and only the differing chunks are retained, dramatically reducing storage requirements while maintaining data integrity through version control metadata.
Solution Approach 2:
The patent implements a hierarchical storage structure where chunks are organized in nested containers (datapacks, repositories). This allows efficient storage of chunk metadata and enables the system to store only the necessary portions of files at each hierarchical level, reducing overall storage consumption while preserving complete file reconstruction capability.
2Adaptability or versatility
If lots of small files (chunks) are transferred to cloud storage, then fine-grained control is achieved, but transfer performance degrades due to numerous small file operations
Solution Approach 1:
The patent combines multiple small chunk files into larger aggregated files called datapacks during the transfer process. This merging operation reduces the number of individual file transfer operations to the cloud, improving transfer performance while maintaining the fine-grained chunk structure within each datapack for efficient storage and retrieval.
Solution Approach 2:
The patent introduces a new organizational dimension by grouping chunks into datapacks with metadata structures. This adds a hierarchical layer between individual chunks and cloud storage, allowing the system to maintain fine-grained control at the chunk level while transferring data in optimized larger units, thus resolving the performance contradiction.
3Device complexity
If sequential transfer of large files is used, then simplicity is maintained, but transfer time increases and fault tolerance is reduced
Solution Approach 1:
The patent segments large files into smaller chunks that can be transferred independently and in parallel. This segmentation enables multiple transfer operations to occur simultaneously, reducing total transfer time and providing fault tolerance since failed chunks can be retried independently without affecting the entire file transfer.
Solution Approach 2:
The patent performs preliminary chunking and organization of files into transferable units before the actual transfer process. This preliminary segmentation allows the transfer mechanism to operate on pre-prepared chunks, enabling parallel transfer operations and improving overall transfer efficiency while maintaining relatively simple transfer logic.
4Quantity of substance
If delta-based versioning is used, then storage efficiency improves, but checkout runtime increases with more versions
Solution Approach 1:
The patent segments files into chunks and stores only the changed chunks between versions. This chunk-level versioning allows the system to efficiently store multiple versions by retaining only the differing portions, while checkout operations can quickly reconstruct files by retrieving base chunks and applying stored deltas, maintaining fast performance even with many versions.
Solution Approach 2:
The patent applies different storage strategies to different parts of the version control system. Frequently accessed base chunks are optimized for quick retrieval, while version-specific delta chunks are stored efficiently for minimal storage overhead. This local optimization allows fast checkout operations regardless of the total number of versions stored.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Embodiments store a datafile. One such embodiment partitions a datafile into chunks. A data structure is constructed that represents the datafile. The data structure includes a hierarchical tree representing the chunks. In turn, respective chunk identifiers (IDs) are generated that correspond to the chunks. Next, non-duplicate chunk(s) are identified from among the chunks based on the generated respective chunk IDs. Based on the constructed data structure and the identified non-duplicate chunk(s), datapack(s) are constructed. The constructed datapack(s) are then stored in memory.