Event-Based Data Intake System for Machine Log Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Analyzing and searching massive quantities of diverse machine data generated from various sources, such as system logs, network packets, and sensors, is challenging due to the vast amount of data and its varying formats, leading to inefficiencies in data retrieval and analysis.
Innovation Solution
An event-based data intake and query system, like SPLUNK, that collects, indexes, and searches machine data as 'events' using a flexible schema, allowing for late-binding schema application during search time, enabling field-searchability and efficient extraction of insights from raw machine data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If massive quantities of raw machine data are stored for later retrieval and analysis, then data flexibility and analysis completeness are improved, but data retrieval and analysis performance deteriorate
Solution Approach 1:
The patent segments raw machine data into discrete events with specific fields and values. Each event is structured with identifiable components that can be independently indexed and searched. This segmentation transforms unstructured raw data into organized units that maintain flexibility while enabling efficient retrieval through field-based indexing.
Solution Approach 2:
The patent introduces an event-based intermediate representation layer between raw data ingestion and analysis queries. This event structure acts as a mediator that preserves the completeness of raw data while providing a standardized, searchable format. The event schema enables flexible data capture without compromising retrieval performance, as queries operate on the structured event representation rather than raw unstructured data.
2Loss of information
If diverse data from various sources is collected and stored, then insight potential is improved, but data complexity and processing difficulty increase
Solution Approach 1:
The patent implements a universal event schema that can accommodate diverse data sources through a common structure. The event format with flexible fields and values can represent different data types (logs, metrics, traces) uniformly. This universality enables diverse data collection without proportionally increasing processing complexity, as the same event handling mechanisms process all data types.
Solution Approach 2:
The patent uses parameter-based field extraction where data characteristics are defined by configurable parameters rather than fixed structures. Extraction rules and field definitions can be adjusted to match different data sources without changing the core event processing framework. This parameter flexibility allows diverse data ingestion while maintaining consistent processing logic.
Data Source
AI summary
Disclosed herein is a fraud analysis data reduction technique. When reviewing a large set of data for potential fraudulent action there is often too much data for a human to reasonably analyze. A technique to reduce the overall amount of data associates entities that have duplicate values stored in corresponding data elements with one another and removes those entities that do not have at least one duplicate value. The entities with duplicate values are entered into a node graph and analyzed for connected components. The connected components analysis and a duplicate threshold analysis provide usable results to identify fraudulent activity.