Late-Binding Schema for Flexible Machine Data Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current tools lack the ability to efficiently and flexibly search and analyze large sets of raw machine data from diverse sources, as they often discard unprocessed data and require pre-defined schemas, limiting the ability to derive insights from all generated data.
Innovation Solution
A data intake and query system that utilizes a late-binding schema, allowing for flexible data processing and storage of raw machine data, enabling efficient retrieval and analysis by indexing and querying machine data with a flexible schema that can be refined during search time, and utilizing containerized indexing and search nodes for scalable data processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If pre-processing and pre-defined schemas are used to manage data, then data management efficiency is improved, but data flexibility and the ability to analyze all generated data deteriorate
Solution Approach 1:
The patent implements late-binding schemas that are dynamically created and refined during search time rather than being pre-defined. The schema evolution process allows the system to adapt to different data types and structures on-demand, enabling both efficient management of structured data and flexible handling of unstructured raw data without requiring predetermined data models.
2Adaptability or versatility
If raw data is stored without pre-processing, then data flexibility and analysis capability are improved, but data management complexity and storage requirements increase
Solution Approach 1:
The patent segments data into different storage locations based on processing state: hot data in local indexers, warm data in remote shared storage, and cold data in archival storage. This segmentation allows raw data to be stored flexibly while managing complexity through hierarchical organization and automated data lifecycle management that handles different data types and access patterns separately.
Solution Approach 2:
The patent introduces a data intake and query system that acts as an intermediary between raw data sources and analysis tools. This system handles schema evolution, data indexing, and query processing automatically, reducing the complexity of managing raw data while preserving full data flexibility and analysis capability.
3Adaptability or versatility
If massive quantities of diverse raw data are stored, then analysis flexibility is improved, but search and query execution efficiency deteriorates
Solution Approach 1:
The patent performs preliminary actions by creating indexes and metadata structures during data ingestion and storage, even before queries are executed. The late-binding schema process pre-processes data to create searchable structures, and the system pre-loads relevant data into memory during query execution, enabling fast search performance across massive quantities of diverse raw data while maintaining full analysis flexibility.
Data Source
AI summary
Systems and methods are disclosed for processing and executing queries in a data intake and query system. The data intake and query system receives raw machine data at an indexing system, and stores at least a portion of the raw machine data in buckets. Based on a determination that the size of multiple buckets satisfies a threshold size, the data intake and query system converts the buckets to non-editable buckets and stores the data in a remote shared storage system.