Chunk-Based Data Store Using Hash-Based Directed Acyclic Graph
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data store architectures face challenges in scaling efficiently to handle large amounts of data, leading to reduced performance and storage limitations, particularly in backup and enterprise storage systems.
Innovation Solution
A chunk-based, content-addressable data store architecture that divides files into chunks, using cryptographic hashes to store and retrieve data efficiently, with a hash-based directed acyclic graph (HDAG) structure that allows for efficient storage and retrieval of file chunks, enabling sharing of common chunks and reducing storage and network bandwidth requirements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If traditional data store architectures are used to store large amounts of data, then the system can handle basic storage operations, but the system performance degrades and storage capacity is limited as data volume increases
Solution Approach 1:
The patent divides files into smaller chunks and stores them as individual units in the data store. Each chunk is assigned a unique identifier (hash) and can be independently managed. This segmentation allows the system to handle large volumes of data by processing and retrieving individual chunks rather than entire files, thereby maintaining system performance while increasing effective storage capacity.
2Quantity of substance
If files are stored as complete units in the data store, then retrieval operations are simple, but storage space is wasted when multiple files share common data segments
Solution Approach 1:
Files are segmented into chunks that can be independently stored and shared. When multiple files contain identical or similar data segments, those chunks are stored only once in the data store with a single unique identifier. Multiple files can reference the same chunk identifier, eliminating redundant storage and improving storage efficiency.
Solution Approach 2:
The patent merges common data segments from multiple files into a single stored chunk. By identifying identical content across different files through hash comparison, the system consolidates duplicate data into one instance that can be shared by multiple parent files, thereby reducing total storage requirements while maintaining data integrity.
3Loss of energy
If the data store stores entire files rather than chunks, then data retrieval is straightforward, but network bandwidth is wasted when only portions of files need to be transferred
Solution Approach 1:
The data store operates at the chunk level rather than the file level, enabling selective retrieval of individual chunks or specific combinations of chunks. When a user needs only a portion of a file, the system retrieves only the required chunks identified by their hash values, significantly reducing network bandwidth consumption compared to transferring entire files.
4Productivity
If the data store uses a simple storage architecture, then implementation is easier, but the system cannot efficiently handle large directory structures and identify changes
Solution Approach 1:
The system pre-computes and stores hash values for all chunks and maintains an index structure that maps chunk identifiers to their locations and relationships. This preliminary organization of data through hashing and indexing enables efficient change detection by allowing the system to quickly compare current state hashes against stored hashes, identifying only the chunks that have changed without scanning entire directory structures.
Data Source
AI summary
To provide a data store, nodes representing respective chunks of files are stored in a predefined structure that defines relationships among the nodes, where the files are divided into the chunks. The nodes are collected into plural groups stored in persistent storage, where some of the nodes are collected into a particular one of the groups according to a locality relationship of the some of the nodes.