Immutable Datastore Using Merkle Trees for Low-Latency Large Data Access
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional databases fail to adequately protect against internal threats and tampering, as they often rely on mutable audit logs and lack robust security measures for large data sets, particularly in low-latency applications like blockchain-based systems which are inefficient for storing and accessing large binary objects.
Innovation Solution
The implementation of a tamper-evident, immutable data structure using a directed acyclic graph with cryptographic hash pointers, where data is stored directly in Merkle Trees, preventing modification detection and reducing latency through direct data access within the tree, and data is fragmented across multiple blockchain-backed storage structures for enhanced security and robustness.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional mutable databases are used for storing large data sets, then ease of operation and flexibility are maintained, but security against tampering and internal threats deteriorates
Solution Approach 1:
The patent segments large data sets into smaller blocks and organizes them within a Merkle Tree structure. Each block is hashed and linked to parent nodes, creating a hierarchical segmentation that enables efficient verification without requiring the entire data structure to be validated. This segmentation approach maintains security while managing complexity for large data sets.
Solution Approach 2:
The patent implements a nested structure where Merkle Trees are embedded within blockchain blocks, and multiple blocks form a chain. The Merkle root hash is nested within the blockchain block, creating layers of verification. This nesting provides tamper-evident storage while maintaining a manageable structural complexity through hierarchical organization.
2Reliability
If blockchain-based immutable structures are used for storing large binary objects, then security and immutability are improved, but access latency and efficiency deteriorate
Solution Approach 1:
The patent extracts large binary objects from the blockchain chain itself and stores them in external storage systems. Only the cryptographic hash values and essential metadata are kept in the blockchain. This extraction eliminates the performance penalty of storing large objects in-chain while maintaining the integrity verification benefits through hash-based proof of existence and unalterability.
Solution Approach 2:
The patent introduces Merkle Trees as an intermediary structure between the blockchain and external storage. The Merkle Tree enables efficient verification of data integrity without requiring full blockchain validation for each access. This intermediary layer provides fast verification for large data sets while maintaining the security guarantees of the underlying blockchain structure.
3Reliability
If cryptographic hash pointers and Merkle Trees are implemented for tamper detection, then reliability and tamper evidence are improved, but device complexity and computational overhead increase
Solution Approach 1:
The patent implements partial verification by allowing validation of specific data blocks without requiring verification of the entire data structure. Through the Merkle Tree structure, one can verify individual leaves or subsets of data by traversing only the necessary paths to the root. This partial action approach provides tamper detection capability while reducing the computational complexity compared to full structure validation.
4Loss of time
If data is stored directly in Merkle Trees within blockchain blocks, then read latency is reduced through direct access, but the size of individual blockchain blocks increases
Solution Approach 1:
The patent extracts actual data content from blockchain blocks and stores it in external systems, retaining only cryptographic hash values and essential metadata within the blocks. This extraction maintains the ability to verify data integrity through Merkle proofs while preventing blockchain blocks from becoming excessively large, thus balancing read latency improvements with manageable block sizes.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Provided is a process including: receiving a write command requesting that a document associated with the write command be stored in an immutable data structure that prevents an attacker attempting to modify the document from concealing that the document was modified after storing the document in the data structure; forming a tamper-evident, immutable directed acyclic graph having the document in one or more nodes of the graph; and storing the directed acyclic graph.