Federated Data Enrichment Objects for Machine Data Search
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Analyzing and searching massive quantities of diverse machine data generated from various sources, such as system logs, network packets, and sensors, is challenging due to the vast amount of data and its varying formats, leading to inefficiencies in data retrieval and analysis.
Innovation Solution
An event-based data intake and query system that uses a late-binding schema to process and store machine data, allowing for flexible schema definition and extraction rules application at search time, enabling field-searchability and efficient retrieval of data across disparate data sources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If massive quantities of diverse machine data are stored for later retrieval and analysis, then data flexibility and analysis completeness are improved, but data retrieval efficiency and search performance deteriorate
Solution Approach 1:
The system performs preliminary actions by generating data enrichment objects during data ingestion that contain extracted field values and metadata. These enrichment objects are created in advance and stored alongside the raw data, enabling efficient searching without requiring full data processing at query time. This resolves the contradiction by preparing data for efficient retrieval while maintaining the ability to analyze all raw data later.
Solution Approach 2:
The patent introduces data enrichment objects as an intermediary layer between raw machine data and search queries. These objects contain pre-extracted fields and metadata that enable efficient searching without requiring the system to process all raw data during queries. This mediator structure allows the system to maintain both data flexibility for complete analysis and retrieval efficiency through indexed fields.
2Productivity
If data is pre-processed to extract specified data items for efficient retrieval, then data retrieval efficiency is improved, but data completeness and flexibility deteriorate
Solution Approach 1:
The system creates copies of data in multiple formats: the original raw data is preserved in its entirety, and separate data enrichment objects are generated containing extracted fields and metadata. This copying approach enables efficient retrieval through the enriched objects while maintaining complete data flexibility by preserving the original unprocessed data for any future analysis needs.
Solution Approach 2:
The patent segments data into different components: raw machine data, extracted field values, metadata, and data enrichment objects. This segmentation allows the system to use different representations for different purposes - efficient retrieval using extracted fields while maintaining data flexibility through preserved raw data and configurable extraction rules.
3Productivity
If extraction rules are applied during data ingestion to generate data enrichment objects, then search performance is improved, but processing time and computational resources increase
Solution Approach 1:
Extraction rules are applied in advance during data ingestion to generate data enrichment objects containing extracted fields and metadata. This preliminary processing creates searchable structures before queries are executed, significantly improving search performance. The trade-off of increased processing time during ingestion is acceptable because it enables much faster retrieval operations later.
Solution Approach 2:
The system dynamically configures extraction rules based on data characteristics and search requirements. Extraction rules can be adjusted, added, or modified without reprocessing all historical data, allowing the system to optimize for different search scenarios while maintaining efficient retrieval performance. This dynamic approach balances processing overhead with search performance needs.
Data Source
AI summary
A data intake and query system can generate local data enrichment objects and receive federated data enrichment objects from another data intake and query system. In response to receiving a query, the data intake and query system can determine whether the query is subquery of a federated query. If the query is a subquery, the data intake and query system can use the federated data enrichment objects to execute the query.