Blockchain Indexing System Decoupling Storage and Compute
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The decentralized nature of blockchains poses challenges for indexing data, as new chains and protocols constantly emerge, requiring a compatible indexing platform that can handle chain reorganizations and data availability issues, with conventional approaches being ineffective due to the unique characteristics of blockchain nodes.
Innovation Solution
A bifurcated indexing system with a dynamic compute engine is introduced, decoupling storage and compute layers to allow for flexible data processing and storage schema modifications, using a sticky master node approach for efficient data extraction and implementing mono-increasing sequence records for reorganization immunity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a common platform for indexing blockchain data is developed, then data management efficiency is improved, but compatibility with new chains and protocols becomes difficult to maintain
Solution Approach 1:
The indexing system is divided into separate modular components: a data ingestion layer that handles raw blockchain data, a processing layer with configurable indexing rules, and a storage layer. This segmentation allows each module to be independently configured for different chains and protocols without affecting the entire system, resolving the contradiction between efficiency and adaptability.
Solution Approach 2:
The system employs dynamic configuration files and runtime-loaded protocols that allow indexing behavior to adapt to new chains and protocols without requiring system-wide redeployment. The configurable parameters enable the same platform to efficiently index diverse blockchain types by loading appropriate configuration sets.
2Loss of information
If the entire blockchain index is redone to add new fields, then data completeness is improved, but processing time and computational resources increase significantly
Solution Approach 1:
The system extracts only the specific new fields required by the application from the blockchain data, rather than re-indexing the entire blockchain. This selective extraction approach maintains data completeness for required fields while avoiding the computational overhead of processing all existing data again.
Solution Approach 2:
The system performs preliminary parsing and validation of blockchain data during the initial ingestion phase, organizing data into a structured format that facilitates incremental updates. When new fields are needed, the pre-organized data structure allows for efficient insertion without requiring complete re-processing.
3Adaptability or versatility
If data is extracted from blockchain nodes on an ad hoc basis, then data flexibility is improved, but reliability and speed deteriorate
Solution Approach 1:
The system introduces an intermediary data cache layer between the blockchain nodes and the indexing application. This cache layer maintains a reliable, pre-fetched copy of blockchain data that can be served to multiple applications simultaneously, ensuring data reliability while allowing flexible access patterns. The intermediary layer handles node communication centrally, improving both reliability and speed.
4Productivity
If conventional load-balanced nodes are used for data processing, then processing capacity is improved, but consistency with blockchain state is lost
Solution Approach 1:
The system employs blockchain-aware indexing nodes that perform multiple functions: they maintain the blockchain state locally to ensure consistency, process and index data in parallel to achieve high capacity, and provide standardized interfaces for applications. This multi-functionality allows nodes to simultaneously deliver high processing capacity and state consistency.
Data Source
AI summary
Systems and methods for supporting both batch processing and streaming data applications, to load and process data incrementally, while providing a near-constantly materialized dataset based on raw blockchain data, are described. For example, the system may receive first on-chain data in a first format via a first input stream, wherein the first on-chain data originates from a blockchain node of a blockchain network. The system may transform the first on-chain data to a second format for storage in a second dataset, wherein the second format comprises an unbounded table.