Bucket Merging for Data Intake Systems Using Size Thresholds
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current tools lack the ability to efficiently search and analyze large sets of raw machine data from diverse data systems, particularly in a visually intuitive and flexible manner, leading to challenges in deriving insights due to the complexity and volume of data.
Innovation Solution
A data intake and query system that utilizes a flexible schema to process and store machine data as events, allowing for late-binding schema application during search time, enabling efficient retrieval and analysis of all generated data through indexing and querying mechanisms.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If data is stored as small individual buckets, then data organization flexibility is improved, but storage efficiency and query performance deteriorate due to excessive metadata overhead
Solution Approach 1:
The patent merges multiple small data buckets into larger merged buckets when they reach a size threshold. This combining approach reduces the number of individual bucket metadata entries, thereby improving storage efficiency and query performance while maintaining the flexibility of organized data storage through the bucketing structure.
2Measurement precision
If data is stored as small individual buckets, then data retrieval granularity is improved, but system resource consumption deteriorates due to metadata management overhead
Solution Approach 1:
The system merges small buckets into larger merged buckets to reduce metadata management overhead and system resource consumption. The merging operation consolidates multiple small bucket metadata structures into fewer larger bucket structures, thereby reducing the total quantity of metadata and associated system resources while preserving data retrieval capability through updated metadata pointers.
3Productivity
If data buckets are frequently merged, then storage efficiency is improved, but processing time deteriorates due to merge operations
Solution Approach 1:
The patent implements dynamic merge policies that adjust merging behavior based on system conditions and data characteristics. Merges are triggered when buckets reach size thresholds or under specific conditions, balancing storage efficiency improvements against processing time costs. The system dynamically determines when and how to merge buckets rather than using fixed periodic merging.
4Adaptability or versatility
If data is stored without pre-processing, then data flexibility and analysis capability are improved, but data volume and processing complexity deteriorate
Solution Approach 1:
The patent segments data into organized buckets and merged buckets with structured metadata, providing a hierarchical organization that maintains data flexibility for various analysis needs while reducing processing complexity through efficient data structure and metadata management. The segmentation approach allows selective data retrieval without processing entire datasets.
Data Source
AI summary
Systems and methods are disclosed for scalable bucket merging in a data intake and query system. Various components of a bucket manager can be used to monitor recently-created buckets of data in common storage that are associated with a particular tenant and a particular index, apply a comprehensive bucket merge policy to determine groups of buckets that qualify for merging, merge those group of buckets into merged buckets to be stored in the common storage, and update any information associated with the merged buckets and pre-merged buckets. These components may be shared across multiple tenants, and some of these components may be dynamically scalable based on need. This approach may also provide many additional benefits, including improved search performance from merged buckets, efficient resource utilization associated with discriminate merging, and redundancy in case of component failure.