Log-Structured Storage for Blockchain Data Tiering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed ledger systems (DLSs) face high storage costs and inefficiencies due to each node storing a complete copy of blockchain data, leading to demanding storage requirements as the system scales.
Innovation Solution
A log-structured storage system that uses a tiered storage approach with append-only data files and index log files, supporting data tiering, compression, erasure coding, and asynchronous processing to optimize storage and retrieval operations, and convert random write operations into sequential append operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If each node stores a complete copy of blockchain data, then data trustworthiness and independence are improved, but storage cost and storage requirements increase
Solution Approach 1:
The patent segments blockchain data into two categories: hot data (recent blocks) and cold data (historical blocks). Nodes store only hot data locally, while cold data is stored in a centralized archive. This segmentation allows nodes to maintain trustworthiness for recent transactions while reducing storage requirements for historical data.
Solution Approach 2:
The patent introduces a centralized archive as an intermediary component that stores cold data. Nodes can query this archive for historical data when needed, eliminating the need for each node to store complete copies while maintaining data accessibility and trustworthiness through the intermediary archive system.
2Adaptability or versatility
If complete blockchain data is stored at each node, then data independence is improved, but storage cost worsens
Solution Approach 1:
The patent divides data storage responsibilities between nodes and the centralized archive. Nodes maintain independent access to hot data they need for current operations, while historical data is managed centrally. This segmentation reduces per-node storage costs while maintaining functional independence.
Solution Approach 2:
Instead of requiring each node to store complete copies of all blockchain data, the patent implements a hybrid approach where nodes store copies of recent data locally and access historical data through the centralized archive, reducing overall storage costs while maintaining data availability.
3Ease of operation
If random write operations are performed, then data retrieval flexibility is improved, but storage performance deteriorates
Solution Approach 1:
The patent segments data into hot and cold categories with different access patterns. Hot data that requires frequent random access is stored locally at nodes, while cold data is archived centrally. This segmentation allows random write flexibility for active data while optimizing storage performance for historical data.
Solution Approach 2:
The patent employs append-only log files for hot data that can be efficiently overwritten or replaced. New blocks are appended to existing logs, and old logs can be archived or discarded, providing efficient storage performance while maintaining flexibility for active data operations.
Data Source
Figure 1~2
Figure 3
Figure 4
AI summary
Disclosed herein are methods, systems, and apparatus, including computer programs encoded on computer storage devices, for data processing and storage. One of the systems includes a first tier storage device with a first performance characteristic and a second tier storage device with a second performance characteristic inferior to the first performance characteristic. The first tier storage device stores a first data log file that includes first blockchain data generated by a blockchain network. The second tier storage device stores a second data log file that includes second blockchain data generated by the blockchain network at an earlier time than the first blockchain data.