Log-Structured Storage for Blockchain Data Tiering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed ledger systems (DLSs) face high storage costs and inefficiencies due to the need for each node to store a complete copy of blockchain data, leading to demanding storage requirements as the system scales.
Innovation Solution
A log-structured storage system that employs a tiered storage approach, using append-only data files and index log files to store blockchain data, allowing for efficient data tiering, compression, and erasure coding, while supporting multi-level data tiering across various storage devices, including cloud disks, NAS, and object storage services.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If each node stores a complete copy of blockchain data, then data trustworthiness and independence are improved, but storage cost and storage requirements worsen
Solution Approach 1:
The storage system is segmented into multiple levels: hot storage for frequently accessed blockchain data, warm storage for less frequently accessed data, and cold storage for archival data. This segmentation allows nodes to store only essential data locally while archiving less critical data elsewhere, reducing storage requirements while maintaining data trustworthiness through selective local retention.
Solution Approach 2:
An intermediary storage coordination mechanism is introduced that manages data distribution across multiple storage levels and nodes. This intermediary layer coordinates data retrieval and verification, allowing nodes to maintain trustworthiness through the coordination system rather than requiring complete local copies of all data, thereby reducing individual node storage requirements.
2Ease of operation
If complete blockchain data is stored at each node, then data independence is improved, but storage cost worsens
Solution Approach 1:
Data independence is maintained through segmentation by implementing distributed hash tables and peer-to-peer data retrieval mechanisms that allow nodes to independently access and verify data from other nodes or storage levels, eliminating the need for complete local data copies while preserving operational independence.
Solution Approach 2:
The storage system implements multi-functionality by creating a universal data access layer that can retrieve and verify data from multiple sources (local storage, network nodes, archival storage) through a unified interface. This allows nodes to maintain data independence through universal access capabilities rather than complete local data retention, reducing storage costs.
3Productivity
If multi-level storage system is implemented, then storage efficiency is improved, but system complexity worsens
Solution Approach 1:
The multi-level storage system implements self-service through automated data classification, migration, and retrieval mechanisms. The system automatically determines which data should be stored at which level based on access patterns and data characteristics, and autonomously manages data movement between levels without requiring complex manual intervention or centralized control, thereby improving storage efficiency while managing system complexity.
Solution Approach 2:
System complexity is managed through parameter changes by implementing dynamic thresholds and policies for data classification and migration. The system adjusts storage parameters such as data retention periods, access frequency thresholds, and migration triggers based on observed usage patterns, allowing the multi-level storage system to adapt automatically without requiring complex fixed-rule management, thus improving efficiency while controlling complexity.
Data Source
Figure 1~2
Figure 3
Figure 4
AI summary
Disclosed herein are methods, systems, and apparatus, including computer programs encoded on computer storage devices, for data processing. One of the methods includes maintaining, by a storage system, a plurality of storage devices that include at least a first tier storage device and a second tier storage device. The storage system receives a write request of a ledger data, determines whether a type of the ledger data is block data, and, in response to determining that the type of the ledger data is block data, writes the data into the second tier storage device.