Blockchain Indexing via Decoupled Compute and Storage Layers
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The decentralized nature of blockchains poses challenges for indexing data, as new chains and protocols constantly emerge, requiring a compatible indexing platform that can handle chain reorganizations and data availability issues, with conventional approaches being inefficient and unreliable.
Innovation Solution
A bifurcated indexing system with a dynamic compute engine decouples storage and compute layers, allowing for flexible data formatting and processing, and uses sticky master nodes for efficient data extraction and batch APIs to handle throughput bottlenecks, while implementing mono-increasing sequence records for reorganization immunity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a common indexing platform is developed to handle blockchain data, then data processing efficiency is improved, but the system must handle constant changes in chains and protocols, increasing system complexity
Solution Approach 1:
The patent segments the indexing system into modular components: a protocol-agnostic core layer that handles basic blockchain data structures, and protocol-specific adapter layers that can be added or removed as needed. This allows the system to process multiple chains and protocols without requiring complete system redesign, resolving the contradiction between handling diversity and maintaining simplicity.
Solution Approach 2:
The system implements dynamic configuration capabilities where new blockchain protocols and chains can be registered and integrated without system downtime or complete reindexing. The indexing platform can adapt its behavior based on the specific protocol being processed, allowing efficient handling of constant changes while maintaining a stable core architecture.
2Reliability
If the entire blockchain index is redone to add new fields, then data completeness is improved, but processing time and computational resources increase significantly
Solution Approach 1:
The patent implements preliminary schema validation and incremental index updates. When new fields are needed, the system validates the schema changes beforehand and applies them incrementally to existing indexes rather than performing complete reindexing. This allows data completeness to be improved while minimizing processing time and resource consumption.
Solution Approach 2:
The system allows dynamic modification of indexing parameters and field configurations without requiring complete reindexing. New fields can be added to the indexing schema, and the system will begin capturing those fields in subsequent blocks while preserving historical data, thus improving data completeness without the time cost of full reindexing.
3Reliability
If archival nodes with complete blockchain state are used, then data availability is improved, but operational costs increase
Solution Approach 1:
The patent creates a universal indexing platform that can serve multiple blockchain protocols and applications from a single infrastructure. By designing protocol-agnostic data structures and shared processing logic, the system achieves data availability for multiple chains without requiring separate archival nodes for each protocol, thereby reducing operational costs while maintaining comprehensive data availability.
4Adaptability or versatility
If data extraction is performed on an ad hoc basis, then flexibility is improved, but reliability and speed decrease
Solution Approach 1:
The patent introduces an intermediary indexing layer between the blockchain nodes and applications. This intermediary maintains continuous, reliable data extraction and transformation pipelines, allowing applications to query data flexibly without requiring direct ad hoc extraction from nodes. The indexing layer ensures data reliability while providing flexible access patterns to applications.
Data Source
AI summary
Systems and methods for a unified approach that is compatible with all blockchains, protocols, etc. The systems and methods use a bifurcated indexing system with a dynamically selected application service. The systems and methods bifurcate the indexing process into a storage layer and a compute layer.