Log-Structured Storage for Blockchain Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed ledger systems, such as blockchain networks, face significant storage challenges due to the need for each node to store a complete copy of the blockchain, leading to high storage demands and costs as the system scales, necessitating more efficient and cost-effective storage solutions.
Innovation Solution
A log-structured storage system that uses a combination of data log files and index log files, employing a merge-tree architecture to optimize storage by converting random write operations into sequential append operations, delaying write flush operations, and supporting tiered storage, data compression, and erasure coding to reduce storage costs and improve performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If each node stores a complete copy of the blockchain, then data reliability and trustworthiness are improved, but storage cost and requirements increase significantly
Solution Approach 1:
The blockchain storage is segmented into two types of log files: data log files storing actual blockchain data and index log files storing mapping information. This segmentation allows nodes to store only necessary data while maintaining reliability through the index structure that enables efficient data retrieval and validation.
Solution Approach 2:
The index log file acts as an intermediary between the data log files and retrieval operations. It stores mapping relationships (block number, offset, size) that enable nodes to locate and access specific data without storing the entire blockchain, thus reducing storage requirements while maintaining data accessibility and reliability.
2Adaptability or versatility
If random write operations are performed, then data can be written flexibly, but write amplification and storage performance deteriorate
Solution Approach 1:
Instead of allowing random writes directly to the data log files, the system inverts the approach by using sequential append-only writes. Data is written sequentially to data log files, and the index log file maintains the mapping relationships, thereby achieving both flexibility and high write performance through sequential operations.
3Adaptability or versatility
If storage capacity is increased to accommodate scaling, then system scalability is improved, but storage cost increases
Solution Approach 1:
The system stores only the necessary data in data log files and uses index log files to maintain mapping relationships. This partial storage approach allows the system to scale efficiently by storing only essential information while using the index structure to enable efficient data retrieval, thereby reducing overall storage costs while maintaining scalability.
Data Source
AI summary
Disclosed herein are methods, systems, and apparatus, including computer programs encoded on computer storage devices, for data processing and storage. One of the methods includes receiving, by a storage system, a plurality of write requests for storing a respective plurality of blockchain data in the storage system. Each of the plurality of the blockchain data includes: a value of one or more of a block, a transaction, or a state of a blockchain network, and a key corresponding to the value. The storage system appends the plurality of the blockchain data to a data log file according to a chronological order of the plurality of the blockchain data, and refrains from sorting the plurality of the blockchain data in the data log file according to respective keys of values in the plurality of the blockchain data.