Distributed Ledger Data Correlation Across Logs, Metrics, and Transactions
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Analyzing and searching massive quantities of machine data from diverse sources in computing environments is challenging due to the variety of data types and formats, which can be time-consuming and inefficient with existing data processing methods.
Innovation Solution
A data intake and query system utilizing a flexible schema and late-binding schema to process and store machine data, enabling field-searchable events with extraction rules applied at search time, and supporting correlation across disparate data sources through a common information model.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If existing data processing methods are used to analyze and search machine data from diverse sources, then data processing can be performed, but it is time-consuming and inefficient
Solution Approach 1:
The system applies extraction rules and creates a common information model in advance during data ingestion, rather than processing data formats at query time. This preliminary action transforms diverse data formats into a standardized structure upfront, enabling efficient searching and correlation later without time-consuming format conversions.
Solution Approach 2:
The system changes the parameter of data representation by maintaining raw data alongside extracted field-based representations. This dual representation allows the system to work with standardized parameters (fields and values) for efficient processing while preserving the original data for later retrieval, resolving the contradiction between processing efficiency and data fidelity.
2Adaptability or versatility
If a flexible schema with late-binding schema is used to maintain raw data for later retrieval, then data analysis flexibility is improved, but data storage and processing complexity increases
Solution Approach 1:
The system segments data storage into two distinct components: raw data storage and extracted field storage. This segmentation allows the system to maintain flexibility by preserving raw data while simultaneously enabling efficient processing through extracted fields. The segmentation resolves complexity by organizing data into manageable, purpose-specific segments rather than attempting to handle all data formats uniformly.
Solution Approach 2:
The common information model acts as an intermediary layer between diverse data sources and analysis queries. It provides a standardized interface that mediates between the variability of raw data formats and the consistency required for efficient searching and correlation, reducing processing complexity while maintaining adaptability.
3Ease of operation
If extraction rules are applied at search time to create field-searchable events, then data search capability is improved, but processing time increases
Solution Approach 1:
The system performs extraction rule application as a preliminary action during data ingestion rather than at search time. This creates field-searchable events in advance, allowing the system to maintain excellent search capability while avoiding time-consuming extraction operations during query execution. The preliminary action resolves the contradiction by shifting processing to when it is most efficient.
Data Source
AI summary
Systems and methods are described to correlate different types of data obtained from a distributed ledger system. The distributed ledger system can generate transaction notifications, log data, and/or metrics data. At least some of the data generated by the distributed ledger system can be obtained by a data intake and query system via a distributed ledger system monitor. The data from the distributed ledger system can be stored in the data intake and query system and correlated. Based on the correlation, the data intake and query system can determine a history of a transaction in the distributed ledger system, relationships between components of the distributed ledger system, and/or an architecture of the distributed ledger system.