Data Intake System for Minimally Processed Machine Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Analyzing and searching massive quantities of machine-generated data from diverse sources is challenging due to the time-consuming nature of processing different types and formats of data from thousands of components, which existing tools like the SPLUNK® ENTERPRISE system attempt to address by using event-based data intake and query systems with late-binding schemas, but still face inefficiencies in data retrieval and analysis.
Innovation Solution
The implementation of a data intake and query system that processes and stores minimally processed machine data, allowing for flexible schema development and extraction rules application at search time, enabling users to search all data rather than pre-specified sets, and employing acceleration techniques like parallel search operations, keyword indexing, and high-performance analytics stores to speed up query processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data from thousands of diverse sources is collected and stored for later analysis, then data volume and analysis flexibility are improved, but data processing time and system complexity increase
Solution Approach 1:
The system performs preliminary actions by collecting and storing data from thousands of diverse sources in a centralized data lake before analysis is requested. Data is ingested, normalized, and made available for future queries without waiting for specific analysis requests, enabling fast retrieval when needed.
Solution Approach 2:
The system segments data processing into distinct stages: data collection from multiple sources, data normalization and standardization, data storage in a centralized repository, and on-demand analysis. This segmentation allows each stage to be optimized independently and enables parallel processing of multiple data streams.
2Productivity
If data is pre-processed and structured before storage, then data retrieval efficiency is improved, but data analysis flexibility deteriorates
Solution Approach 1:
The system changes the parameter of data structure from fixed pre-processing to dynamic on-demand processing. Data is stored in a standardized format with consistent schemas and metadata, but the actual analysis and transformation occur dynamically based on specific query requirements, allowing both efficient retrieval and flexible analysis.
Solution Approach 2:
The system implements dynamic data processing where the level and type of processing applied to data depends on the specific analysis request. Data can be retrieved in its standardized form for quick access, then dynamically transformed, filtered, or aggregated based on the analytical needs of the query, combining efficiency with flexibility.
3Adaptability or versatility
If all machine data is stored in minimal processing form, then data storage capacity and analysis options are improved, but query processing complexity increases
Solution Approach 1:
The system performs preliminary normalization and standardization actions on data during ingestion, establishing consistent schemas, data types, and metadata structures before storage. This preliminary processing reduces the complexity of subsequent queries by ensuring data is already in a standardized format, while still retaining minimal processing to preserve analysis flexibility.
4Quantity of substance
If diverse data formats from multiple sources are processed, then data comprehensiveness is improved, but processing time and computational resources increase
Solution Approach 1:
The system segments data processing by source type and data format, applying format-specific normalization rules and transformation logic tailored to each data source. This segmentation allows parallel processing of multiple data streams simultaneously while maintaining comprehensive data collection from diverse sources, improving overall processing throughput.
Data Source
AI summary
Embodiments of the present invention are directed to identifying related data, in particular, data associated with different source types. In embodiments, a first source type related to a second source type associated with a search query is identified. Field set pairs are identified from a first data set associated with the first source type and a second data set associated with the second source type. Each field set pair can include one field set associated with the first source type and another field set associated with the second source type. For each field set pair, an extent of similarity is determined between the corresponding field sets. Based on the extent of similarities between the corresponding field sets, at least one pair of related field sets is identified. An indication of the at least one pair of related field sets is provided, for example, for presentation to a user.