Data Model Summaries for Machine Data Search Acceleration
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Analyzing and searching massive quantities of diverse machine data generated from various sources, such as system logs, network packets, and sensors, poses challenges due to its vastness and complexity, requiring efficient data intake and query systems to facilitate real-time operational intelligence.
Innovation Solution
A data intake and query system that utilizes a flexible schema to process and store machine data as events, enabling field-searchability and late-binding schema application during search time, along with a distributed architecture that includes ingestors, message buses, and indexers for scalable data processing and storage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If massive quantities of raw machine data are stored for later retrieval and analysis, then data analysis flexibility and completeness are improved, but data search and analysis performance deteriorate
Solution Approach 1:
The patent applies preliminary action by generating data model summaries during data ingestion before actual analysis queries are executed. The system pre-processes raw machine data into structured summaries that capture essential patterns and relationships, so when queries arrive later, they can be answered efficiently against these pre-computed summaries rather than scanning all raw data. This resolves the contradiction by preparing data structures in advance that enable both comprehensive analysis flexibility and fast query performance.
Solution Approach 2:
The patent creates simplified copies of the raw machine data in the form of data model summaries. These summaries are condensed representations that retain the essential analytical value of the original data while occupying minimal storage space and enabling rapid retrieval. The system maintains both the complete raw data for flexibility and the summarized copies for performance, allowing analysts to query comprehensive data without the performance penalty of searching through all raw machine data.
2Productivity
If data is preprocessed and extracted based on anticipated analysis needs, then retrieval efficiency is improved, but data completeness and flexibility are reduced
Solution Approach 1:
The system performs preliminary action by automatically generating data model summaries during the data ingestion phase, before any specific analysis queries are formulated. This pre-processing creates structured, queryable representations of the raw data that capture essential patterns without requiring analysts to pre-specify their exact needs. The summaries are generated using automated machine learning techniques that identify relevant features and relationships, enabling efficient retrieval while preserving analytical flexibility for future queries.
Solution Approach 2:
The system applies self-service by using automated machine learning algorithms to autonomously analyze the incoming raw machine data and generate appropriate summaries without human intervention. The system self-determines what patterns, relationships, and features are worth capturing in the summaries, eliminating the need for analysts to manually preprocess data or anticipate their future analysis needs. This automated approach maintains data completeness while enabling efficient retrieval.
3Adaptability or versatility
If vast amounts of diverse raw data are retained instead of being discarded during preprocessing, then analytical insights and flexibility are improved, but system complexity and processing requirements increase
Solution Approach 1:
The patent applies segmentation by dividing the data management system into distinct layers: the complete raw machine data is retained for flexibility, while separately generated data model summaries provide the structured, efficient interface for queries. This segmentation allows the system to maintain both the comprehensive raw data and the simplified summary structures without requiring the entire system to handle the full complexity of raw data processing. The summaries act as an intermediate layer that reduces system complexity while preserving analytical flexibility.
Solution Approach 2:
The system creates simplified copies (summaries) of the complex raw machine data. These summaries are compact, structured representations that capture the essential analytical value of the raw data while being much easier to process and query. By working with these copied summaries rather than the original complex raw data, the system reduces processing requirements and complexity while maintaining the ability to derive analytical insights. The raw data remains preserved for cases where maximum flexibility is needed.
Data Source
AI summary
Embodiments described herein facilitate enhancement of data model acceleration, including generating data model summaries and performing searches in an accelerated manner. In one implementation, a set of events are indexed, each of the events having a corresponding index time representing a time at which the event was indexed in an indexer. Index time parameters including an index earliest time indicating a first index time at which to begin generating a data model summary and an index latest time indicating a second index time at which to complete generating the data model summary are obtained. Thereafter, a data model summary is generated. Such a data model summary summarizes events having corresponding index times between the index earliest time and the index latest time. The data model summary is provided to a remote data store that is separate from the indexer at which at least a portion of the events were indexed.