Data Intake System Correlating Logs and Metrics
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Analyzing and searching massive quantities of machine data from diverse sources, such as system logs, network data, and mobile devices, is challenging due to the vast amounts of different types and formats of data generated by thousands of components, which can be time-consuming and inefficient with existing data intake and query systems.
Innovation Solution
A data intake and query system that integrates metric collection from an instrumentation platform, allowing for the correlation of log data with metric data, enhancing metric data with log information by storing log data as raw machine data and using a late-binding schema for flexible extraction and analysis of field values during search time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If existing data intake and query systems are used to analyze and search machine data from diverse sources, then data processing can be performed, but the analysis becomes time-consuming and inefficient due to vast amounts of different data types and formats
Solution Approach 1:
The system segments machine data into distinct event units with standardized schemas, allowing parallel processing and efficient indexing. Each event is broken down into discrete fields that can be independently queried and analyzed, dramatically improving search efficiency across vast data volumes.
Solution Approach 2:
The system transforms unstructured and semi-structured machine data into standardized event parameters with consistent schemas. By changing the parameter representation from raw heterogeneous data to structured event fields, the system enables rapid filtering, aggregation, and analysis operations.
2Adaptability or versatility
If log data and metric data are kept separate in traditional systems, then data storage is simpler, but correlation and integration between different data types become difficult
Solution Approach 1:
The system merges log data and metric data into a unified event-based framework where both types of data are ingested, standardized, and stored together. This consolidation enables direct correlation between logs and metrics through shared event identifiers and time synchronization, while the unified schema reduces overall system complexity.
Solution Approach 2:
The system creates a universal data model that handles multiple data types (logs, metrics, traces) through a single event schema. This multi-functional framework allows the same processing pipeline to handle diverse data sources, enabling correlation across different data types without requiring separate integration mechanisms.
3Measurement precision
If early data intake systems process all data immediately, then real-time analysis is achieved, but computational resources are overwhelmed by massive data volumes
Solution Approach 1:
The system performs preliminary actions during data ingestion by immediately standardizing, parsing, and indexing data as it arrives. This upfront processing transforms raw data into query-optimized formats, enabling efficient later retrieval and analysis without requiring intensive computational resources during actual query execution.
Solution Approach 2:
The system applies partial processing to data streams by extracting and indexing only the most relevant fields immediately, while maintaining the ability to perform deeper analysis on demand. This selective preprocessing reduces immediate computational overhead while preserving measurement precision for critical fields.
Data Source
AI summary
A data intake and query system establishes a network connection with an instrumented target system. The instrumented target system collects metrics in accordance with an instrumentation platform, whereby the instrumentation platform defines an instrumentation platform query. The metrics are stored in the data intake and query system. A data intake and query system query for a subset of metrics is transmitted. The data intake and query system query replicates the instrumentation platform query of the instrumentation platform. From the data store, log data for the instrumented target system, and correlated with the metrics to obtain correlated results.