Bucket Merging in Data Intake Systems for Storage Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current tools lack the ability to efficiently and flexibly search and analyze large sets of raw machine data from diverse sources, as they often discard unprocessed data and require pre-defined schemas, limiting the ability to derive insights from all generated data.
Innovation Solution
A data intake and query system that utilizes a late-binding schema, allowing for flexible data processing and storage of raw machine data, enabling real-time search and analysis through containerized indexing and query nodes, and efficient data retrieval and indexing techniques.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If raw machine data is stored without pre-processing, then data flexibility and analysis capability are improved, but data volume and storage complexity increase
Solution Approach 1:
The patent segments raw machine data into discrete events with structured fields (time, host, source, sourcetype, index, tenant, etc.). Each event is a standardized unit that can be independently stored, retrieved, and analyzed. This segmentation allows the system to handle large volumes of raw data efficiently while maintaining flexibility for various analysis scenarios.
2Productivity
If pre-defined schemas are used for data processing, then data retrieval efficiency is improved, but adaptability to diverse data sources deteriorates
Solution Approach 1:
The patent implements a universal event schema that can accommodate diverse data sources through standardized fields. The event structure includes generic fields (time, host, source, sourcetype) that can represent any data source type, while allowing custom fields for source-specific data. This universal structure enables efficient retrieval through standardized query mechanisms while maintaining adaptability to diverse data sources.
3Speed
If data is pre-processed based on anticipated needs, then analysis speed is improved, but data completeness and flexibility deteriorate
Solution Approach 1:
The patent performs preliminary structuring of raw data into standardized events during ingestion, organizing data into consistent fields and formats without discarding any information. This preliminary action creates a uniform structure that enables fast retrieval and analysis while preserving all original data. The system maintains completeness by storing the full event data while allowing flexible querying based on any field.
4Adaptability or versatility
If massive quantities of raw data are stored, then analytical flexibility is improved, but system complexity and resource requirements increase
Solution Approach 1:
The patent changes the parameter of data representation by transforming unstructured raw data into structured events with standardized fields. This parameter change enables efficient storage, retrieval, and analysis operations while maintaining analytical flexibility. The standardized event structure with consistent field names and types allows the system to handle large volumes of data without proportionally increasing complexity.
Data Source
AI summary
Systems and methods are disclosed for processing and executing queries in a data intake and query system. An indexing system of the data intake and query system receives data and stores at least a portion of it in buckets, which are then stored in a shared storage system. The indexing system merges multiple buckets to generate merged buckets and uploads the merged buckets to the shared storage system.