Partition-Specific Query Rewriting for Faster Machine Data Search

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Analyzing and searching massive quantities of machine data from diverse sources is challenging due to the variety of data types and formats, leading to inefficiencies in data retrieval and analysis.

Innovation Solution

Implementing a data intake and query system that uses a late-binding schema to process and store machine data, allowing flexible schema definition at search time and enabling field-searchable events through extraction rules, with a common information model across disparate data sources.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Speed

If data is pre-processed and only specified data items are extracted for storage, then retrieval efficiency is improved, but data flexibility and analysis capability are reduced

Engineering Contradiction:

Improvedata retrieval speedVSAvoiddata analysis flexibility

Core Design Contradiction:

SpeedVSAdaptability or versatility

Solution Approach 1:

The system performs preliminary indexing and metadata extraction during data ingestion, organizing data by partition keys and value ranges. This preliminary action enables efficient query execution without requiring full data scanning, thus improving retrieval speed while preserving access to all original data for flexible analysis.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent introduces an intermediary indexing layer that sits between raw data storage and query processing. This intermediary structure includes partition metadata and value range indexes that mediate between the stored data and analysis queries, enabling both fast retrieval and flexible analysis by translating diverse queries into efficient data access patterns.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Adaptability or versatility

If massive quantities of raw data are stored for later retrieval, then data analysis flexibility is improved, but search and analysis performance deteriorate

Engineering Contradiction:

Improvedata analysis flexibilityVSAvoiddata search and analysis performance

Core Design Contradiction:

Adaptability or versatilityVSProductivity

Solution Approach 1:

The patent segments the massive raw data into partitioned datasets organized by partition keys and value ranges. This segmentation allows the system to store all raw data for flexible analysis while enabling performance optimization by querying only relevant partitions and value ranges, thus maintaining both flexibility and productivity.

Inventive Principle:

Principle #1Segmentation

3Loss of information

If diverse data types and formats from numerous sources are analyzed, then insight generation is improved, but analysis complexity and time increase

Engineering Contradiction:

Improveinsight generation completenessVSAvoiddata analysis time

Core Design Contradiction:

Loss of informationVSLoss of time

Solution Approach 1:

The patent implements a universal partitioned dataset structure that can accommodate diverse data types and formats from numerous sources through a common schema framework. This universal structure enables the system to handle heterogeneous data uniformly, reducing analysis complexity while preserving complete information for comprehensive insight generation.

Inventive Principle:

Principle #6Universality (Multi-functionality)

Data Source

PatentUS20260111422A1Query modification using partition-specific commands

Publication Date: 2026.04.23 SPLUNK INC

AI summary

Systems and methods are disclosed for modifying a query using partitioned datasets. A query system may receive a query that includes a data field identifier and identifies a set of data to be processed. The system may use the query to identify a partitioned dataset that is associated with the query and identify partitions of the partitioned dataset that include a data field that satisfies the data field identifier. The query system may use the identified partitions to modify the query to include filter criteria that includes a partition identifier for the identified partitions.