Bifurcated Blockchain Indexing for Throughput Bottlenecks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The decentralized nature of blockchains poses challenges for indexing data, as new chains and protocols constantly emerge, requiring a compatible indexing platform that can handle chain reorganizations and data availability issues, with conventional approaches being ineffective due to the unique characteristics of blockchain nodes.
Innovation Solution
A bifurcated indexing system with a dynamic compute engine is introduced, decoupling storage and compute layers to allow for flexible data processing and storage schema modifications, using a sticky master node approach for efficient data extraction and a reorganization-immune indexing method with mono-increasing sequence records.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If a single blockchain node is used for data extraction, then system complexity is reduced, but throughput is limited and becomes a bottleneck
Solution Approach 1:
The system segments the blockchain node pool into master nodes and slave nodes with distinct roles. Master nodes handle chain reorganization detection and canonical chain determination, while slave nodes perform parallel data extraction. This segmentation allows multiple nodes to work together without requiring complex coordination, resolving the contradiction between simplicity and throughput.
Solution Approach 2:
The system introduces an intermediary component that coordinates between the single-point application interface and the multi-node extraction system. This intermediary manages the distribution of extraction tasks to slave nodes and aggregates results, maintaining a simple interface for applications while enabling high-throughput parallel extraction from multiple nodes.
2Ease of manufacture
If conventional indexing schemas are used, then implementation is simple, but they cannot handle new blockchains and protocols requiring complete index redone
Solution Approach 1:
The indexing schema is designed to be dynamic and configurable rather than fixed. The system can adapt its indexing structure based on the specific blockchain and protocol being indexed, allowing new blockchains and protocols to be supported without complete redesign. The schema can be modified to accommodate new data formats and requirements while maintaining existing functionality.
Solution Approach 2:
The system implements a universal indexing framework that can handle multiple blockchain types and protocols through a common architecture. By designing the indexing schema to be protocol-agnostic and configurable, the system achieves versatility across different blockchains while maintaining implementation simplicity through standardized interfaces and processes.
3Adaptability or versatility
If ad hoc data extraction from blockchain nodes is performed, then flexibility is maintained, but extraction is unreliable and slow
Solution Approach 1:
The system performs preliminary actions by pre-establishing connections to multiple blockchain nodes and pre-configuring extraction parameters before actual data extraction begins. This preparation ensures that when extraction is needed, reliable and fast access is already in place, improving both reliability and speed while maintaining flexibility through pre-configurable options.
Solution Approach 2:
The system replaces ad hoc, manual data extraction mechanisms with an automated, systematic approach. Instead of performing extraction on-demand from single nodes, the system automatically manages multiple node connections, distributes extraction tasks, and handles reorganization detection systematically, improving reliability and speed while preserving flexibility through configurable parameters.
4Device complexity
If blockchain nodes are arranged in master-slave configuration, then coordination is simplified, but it conflicts with the decentralized master-to-master nature of blockchain
Solution Approach 1:
The system applies local quality by assigning different roles (master or slave) to nodes based on their specific functions in the indexing system rather than requiring all nodes to have identical capabilities. Master nodes specialize in chain reorganization detection while slave nodes specialize in parallel data extraction. This role differentiation simplifies coordination locally while maintaining compatibility with the overall decentralized blockchain architecture.
Data Source
AI summary
Systems and methods for a unified approach that is compatible with all blockchains, protocols, etc. The systems and methods use a bifurcated indexing system with a dynamically selected application service. The systems and methods bifurcate the indexing process into a storage layer and a compute layer.