Flexible Schema Data Intake System for Machine Log Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Analyzing and searching massive quantities of machine data generated from various sources, such as system logs, network packets, and sensors, is challenging due to the vast amounts of different types and formats of data, which can be time-consuming and inefficient with existing data processing systems.
Innovation Solution
A data intake and query system that uses a flexible schema to process and store machine data as events with timestamps, allowing for field-searchable and minimally processed data storage, enabling late-binding schema extraction rules to be applied during search time, facilitating the use of common field names across disparate data sources and supporting pipelined search queries.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If existing data processing systems are used to analyze and search machine data, then data processing can be performed, but the process becomes time-consuming and inefficient due to the vast amounts of different types and formats of data
Solution Approach 1:
The patent segments machine data into standardized events with consistent schemas, dividing the heterogeneous data into manageable, uniformly structured units that can be processed efficiently. This segmentation allows the system to handle diverse data types through a common processing framework, improving productivity while reducing analysis time.
Solution Approach 2:
The system changes the parameter of data representation by transforming raw machine data into standardized events with defined schemas, timestamps, and field structures. This parameter transformation enables efficient indexing and searching, resolving the contradiction between processing capability and time consumption.
2Adaptability or versatility
If a flexible schema is used to store minimally processed data, then greater flexibility and insight are achieved, but the system complexity increases due to late-binding schema extraction
Solution Approach 1:
The system performs preliminary actions by storing data in a standardized event format with basic schema validation at ingestion time, while deferring complex schema extraction and field identification to search time. This preliminary structuring provides flexibility without immediately incurring the complexity of advanced schema processing.
Solution Approach 2:
The patent introduces an intermediary layer of standardized events that sits between raw data ingestion and complex queries. This intermediary structure with defined schemas and fields acts as a mediator, providing flexibility for various analysis types while hiding the complexity of schema processing from end users and simplifying the overall system architecture.
3Loss of information
If all machine data is stored and made searchable, then comprehensive analysis is enabled, but the data storage and processing requirements increase significantly
Solution Approach 1:
The system extracts only the essential and relevant information from raw machine data, storing it in a compact event format with key fields and schemas. This extraction approach maintains data availability for comprehensive analysis while significantly reducing storage requirements by eliminating redundant and irrelevant information.
Solution Approach 2:
Instead of storing all raw data and filtering during retrieval, the system inverts the approach by pre-processing and structuring data into standardized events during ingestion, then storing only the essential structured information. This inversion enables comprehensive searchability with reduced storage needs, as the structuring happens upfront rather than requiring retention of all raw data.
Data Source
AI summary
Systems and methods for testing a subject system with a software testing process are described. The system receives Boolean states responsive to repeatedly applying a first test case to a subject system. Each Boolean state signifies an outcome of an application of the first test case to a version of a first software feature over a span of time. The system identifies test case outcomes for the first test case that are adjacent in time and different and generates an intermittency value for the first test case. The system determines that the intermittency value for the first test case exceeds an intermittency threshold and alerts an engineering resource. Finally, the system repeats the above operations until the intermittency value for the first test case does not exceed the intermittency threshold.