Indexing Data via Event Segmentation and Schema Flexibility

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing data intake and query systems face challenges in efficiently processing and analyzing massive quantities of minimally processed machine data, which can lead to increased time and resources in data analysis.

Innovation Solution

The implementation of an event-based data intake and query system that uses a flexible schema to process and store machine data as events, allowing for real-time operational intelligence and flexible data analysis.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If a data intake and query system processes and stores all machine data, then data analysis flexibility is improved, but system resource consumption increases

Engineering Contradiction:

Improvedata analysis flexibilityVSAvoidsystem resource consumption

Core Design Contradiction:

Adaptability or versatilityVSUse of energy by moving object

Solution Approach 1:

The patent segments machine data into discrete events with specific schemas, organizing data into manageable units that can be processed efficiently. Each event is structured with defined fields and data types, allowing the system to handle large volumes of data without overwhelming resource consumption.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The system changes the parameter of data representation by converting raw machine data into standardized events with specific schemas. This transformation allows the data to be stored in a format that enables flexible querying while optimizing storage efficiency and reducing the computational resources needed for data processing.

Inventive Principle:

Principle #35Parameter changes

2Reliability

If the system stores minimally processed machine data, then data integrity is improved, but processing time increases

Engineering Contradiction:

Improvedata integrityVSAvoidprocessing time

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The system performs preliminary actions by structuring and schema-validating machine data as it is ingested, rather than processing it later. Events are created with defined schemas during data intake, which enables faster querying and analysis without compromising data integrity. This upfront structuring reduces the time needed for subsequent data processing operations.

Inventive Principle:

Principle #10Preliminary action

3Loss of information

If the system uses a flexible schema to process all machine data, then data insight quality is improved, but system complexity increases

Engineering Contradiction:

Improvedata insight qualityVSAvoidsystem complexity

Core Design Contradiction:

Loss of informationVSDevice complexity

Solution Approach 1:

The patent implements a universal event schema that can handle multiple types of machine data through a common structure. The schema design allows different data sources and formats to be processed using the same event framework, reducing system complexity while maintaining the ability to extract comprehensive insights from diverse machine data.

Inventive Principle:

Principle #6Universality (Multi-functionality)

Data Source

PatentUS12299508B2Indexing data at a data intake and query system based on a node capacity threshold

Publication Date: 2025.05.13 CISCO TECHNOLOGY INC

AI summary

As an indexer indexes and groups events, it can generate data slices that include events. Based on a slice rollover policy, the indexer can add a particular slice to an aggregate slice. Based on an aggregate slice backup policy, the indexer can store a copy of the aggregate slice to a shared storage system. The aggregate slice can be used for restore purposes in the event the indexer fails or becomes unresponsive.