Late-Binding Schema for Distributed Ledger Data Correlation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Analyzing and searching massive quantities of machine data from diverse sources in data centers and networks is challenging due to the vast variety and format of data types, requiring efficient data intake and query systems that can process and store data flexibly and retrieve insights effectively.
Innovation Solution
A data intake and query system utilizing a flexible schema, known as a late-binding schema, which processes and stores machine data as events with timestamps, allowing for field-searchable and semantically-related data extraction, and enables users to refine extraction rules during search time, facilitating the correlation of data across disparate sources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If data from diverse sources is processed and stored using traditional schemas, then data structure is well-defined and query processing is straightforward, but the system cannot flexibly adapt to various data types and formats from different distributed ledger nodes
Solution Approach 1:
The patent implements a dynamic schema approach where extraction rules are not fixed at data ingestion time but are refined and adjusted during search time. The system allows users to modify extraction rules based on the specific query requirements and data characteristics, enabling flexible adaptation to diverse data types from different distributed ledger nodes without requiring complex pre-processing for each data format
Solution Approach 2:
The system changes the parameter of schema binding time from early (at ingestion) to late (at search time). This parameter change allows the extraction rules to be adapted and refined based on actual query needs and data characteristics, providing versatility in handling diverse data formats while maintaining manageable system complexity through on-demand rule refinement
2Quantity of substance
If massive quantities of machine data are collected and stored from multiple sources, then data volume and coverage are increased, but data correlation and insight retrieval become challenging
Solution Approach 1:
The system performs preliminary data collection and storage from multiple distributed ledger nodes, accumulating massive quantities of machine data in a standardized format. This preliminary action enables subsequent correlation analysis by having all necessary data already available in the system, reducing the difficulty of detecting and measuring relationships across diverse data sources when queries are executed
Solution Approach 2:
The patent introduces an intermediary layer (the data intake and query system with late-binding schema) that sits between the diverse data sources and the analysis tools. This intermediary standardizes and stores data from multiple nodes, enabling correlation by providing a unified access point and consistent data structure that facilitates detecting relationships across otherwise disparate sources
3Productivity
If extraction rules are fixed at data ingestion time, then processing efficiency is high, but users cannot refine data extraction based on specific query requirements
Solution Approach 1:
The system transitions from static extraction rules fixed at ingestion time to dynamic rules that can be refined at search time. This allows the extraction process to adapt to specific query requirements while maintaining efficient processing through cached data structures, achieving both productivity and ease of operation through on-demand rule refinement rather than complete re-processing
Data Source
AI summary
Systems and methods are described to associate data from different nodes of a distributed ledger system. The nodes can generate transaction notifications, log data, and/or metrics data. At least some of the data generated by the nodes can be obtained by a data intake and query system via a distributed ledger system monitor. The data from the distributed ledger system can be stored in the data intake and query system and correlated. Based on an association between at least some of the data of the first node and at least some of the data of the second node, the data intake and query system can determine at least a partial history of a transaction in the distributed ledger system, relationships between components of the distributed ledger system, and/or an architecture of the distributed ledger system.