Log-Structured Storage for Blockchain Data Tiering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed ledger systems (DLSs) face challenges in efficiently managing storage due to the need for each node to store a complete backup of blockchain data, leading to high storage costs and inefficiencies as the system scales.
Innovation Solution
A log-structured storage system that uses a tiered storage approach with append-only data files and index log files, supporting multi-level data tiering and various storage devices, and employs techniques like data compression, erasure coding, and snapshotting to optimize storage and performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If each node stores a complete backup of blockchain data, then data reliability and trustworthiness are improved, but storage costs and system complexity increase
Solution Approach 1:
The storage system is segmented into multiple tiers: hot storage for frequently accessed blockchain data, warm storage for less frequently accessed data, and cold storage for archival data. This segmentation allows nodes to store only essential data locally while maintaining reliability through distributed storage across the network.
Solution Approach 2:
The patent introduces a hierarchical storage dimension, moving from a single-level complete backup approach to a multi-tier storage architecture. This adds temporal and access-frequency dimensions to storage management, enabling nodes to balance reliability requirements with storage efficiency.
2Reliability
If each node stores a complete backup of blockchain data, then data trustworthiness is improved, but storage costs increase
Solution Approach 1:
Different nodes are assigned different storage responsibilities based on their role and capacity. Full nodes store complete blockchain data for maximum trustworthiness, while light nodes store only essential verification data. The system provides tailored storage solutions matching each node's quality requirements.
Solution Approach 2:
The patent changes storage parameters dynamically based on data access patterns. Frequently accessed blockchain data is maintained in full copies across multiple nodes, while historical data is archived with reduced redundancy. This parameter adjustment optimizes the balance between trustworthiness and storage capacity.
3Reliability
If complete blockchain data is stored at each node, then data availability is improved, but storage efficiency deteriorates
Solution Approach 1:
The system performs preliminary data preparation by pre-processing blockchain data into different formats and tiers before storage. Data is organized and indexed in advance, enabling efficient retrieval without requiring complete raw data to be stored at every node, thus improving storage efficiency while maintaining availability.
Data Source
Figure 1~2
Figure 3
Figure 4
AI summary
Disclosed herein are methods, systems, and apparatus, including computer programs encoded on computer storage devices, for data processing. One of the methods includes determining, by a storage system from an index log file, information of a data log file stored in a storage device. The data log file includes one or more of block data, transaction data, or state data generated by a blockchain network. The storage system determines whether the data log file needs a rewritten placement. In response to determining that the data log file needs a rewritten placement, the storage system determines a source storage location of the data log file, reads the data log file from the source storage location, rewrites the data log file to a destination storage location; and generates a destination index log file that indicates the destination storage location corresponding to the data log file.