Log-structured storage for blockchain data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed ledger systems, such as blockchain networks, face high storage costs and inefficiencies due to each node storing a complete copy of the blockchain data, leading to demanding storage requirements as the system scales.
Innovation Solution
A log-structured storage system that uses a merge-tree architecture to store blockchain data in append-only data files, with data log files for substantive data and index log files for storage locations, supporting tiered storage and asynchronous processing to optimize performance and reduce costs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If each node stores a complete copy of blockchain data, then data trustworthiness and independence are improved, but storage cost and storage requirements increase
Solution Approach 1:
The patent segments blockchain data into two distinct components: immutable data (stored on-chain in the blockchain) and mutable data (stored off-chain in a log-structured storage system). This segmentation allows nodes to store only the essential immutable data locally while accessing mutable data remotely, significantly reducing storage requirements while maintaining data integrity and trustworthiness through cryptographic verification.
2Reliability
If each node stores a complete copy of blockchain data, then data independence is improved, but storage efficiency deteriorates
Solution Approach 1:
The patent introduces a log-structured storage system as an intermediary between the blockchain and node applications. This intermediary handles data storage and retrieval operations, allowing nodes to maintain data independence through cryptographic verification while achieving improved storage efficiency by not duplicating entire data sets locally. The log-structured system acts as a shared storage layer that multiple nodes can access efficiently.
3Adaptability or versatility
If random write operations are performed on blockchain data, then data update flexibility is improved, but write performance deteriorates
Solution Approach 1:
The patent inverts the traditional approach by storing immutable data on the blockchain and mutable data in a log-structured storage system. This inversion allows mutable data to be updated efficiently through sequential append operations rather than random writes, dramatically improving write performance. The log-structured system processes writes in a streamlined manner while the blockchain maintains data flexibility through its immutable ledger structure.
4Reliability
If data is stored in traditional blockchain structure, then data immutability is improved, but read performance deteriorates
Solution Approach 1:
The patent segments data into immutable on-chain data and mutable off-chain data stored in a log-structured format. The log-structured storage system optimizes read performance by organizing data in a hierarchical structure with indexes that enable efficient querying and retrieval. This segmentation allows frequently accessed mutable data to be read quickly from the log-structured system while immutable data remains secured on the blockchain.
Data Source
AI summary
Disclosed herein are methods, systems, and apparatus, including computer programs encoded on computer storage devices, for data processing and storage. One of the method includes: maintaining a plurality of tiers of storage devices and one or more tiers of caches by a storage system for storing blockchain data, the plurality of tiers of storage devices including at least a higher-tier storage device and a lower-tier storage device; determining that a blockchain data object in a data log file stored in a lower-tier storage device is an active data object, wherein the blockchain data object is block data, transaction data, or state data; and writing the blockchain data object into a cache of the one or more tiers of caches.