Log-Structured Storage for Blockchain Data Management
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed ledger systems, such as blockchain networks, face significant storage challenges due to the need for each node to store a complete copy of the blockchain, leading to high storage costs and inefficiencies as the system scales.
Innovation Solution
A log-structured storage system that uses a combination of data log files and index log files, adopting a merge-tree architecture to optimize storage by converting random write operations into sequential append operations, supporting tiering, data compression, and erasure coding, and managing data across multiple storage devices to reduce storage costs and improve performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If each node stores a complete copy of blockchain data, then data trustworthiness and independence are improved, but storage costs and system scalability deteriorate
Solution Approach 1:
The patent segments blockchain data into two distinct types: immutable data (block headers, transaction hashes, merkle roots) stored on the blockchain, and mutable data (transaction details, state changes) stored in a log-structured storage system. This segmentation allows nodes to store only essential immutable data while accessing mutable data from centralized or distributed log storage, significantly reducing per-node storage requirements while maintaining data integrity and trustworthiness through cryptographic verification.
2Reliability
If each node stores a complete copy of blockchain data, then data independence is improved, but storage efficiency deteriorates
Solution Approach 1:
The patent introduces a log-structured storage system as an intermediary between the blockchain and node applications. This intermediary handles data storage and retrieval operations, providing nodes with efficient access to mutable data without requiring them to store complete blockchain copies. The log-structured storage acts as a mediator that maintains data independence for nodes while dramatically improving storage efficiency through centralized or distributed log management.
3Ease of operation
If random write operations are performed on storage systems, then data accessibility is improved, but write amplification and performance deteriorate
Solution Approach 1:
The patent implements a dynamic storage architecture where the log-structured storage system automatically adapts its internal structure based on access patterns and data characteristics. The system dynamically organizes data into log segments, manages segment merging and compaction operations, and optimizes storage layouts to balance random read accessibility with sequential write performance, thereby reducing write amplification while maintaining data accessibility.
Data Source
AI summary
Disclosed herein are methods, systems, and apparatus, including computer programs encoded on computer storage devices, for data processing and storage. One of the systems includes a multi-tier storage subsystem including multiple tiers of storage devices, an input/output (I/O) subsystem that is configured to perform a write operation of blockchain data into a data log file, and a back-end data management subsystem that is configured to perform a management operation of the data log file stored in the one of the multiple tiers of storage devices.