Index File Size Reduction via Event Attribute Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
As data centers and computing environments generate vast amounts of machine-generated data, the increasing size of index files for data storage leads to higher costs and operational challenges, making it difficult to efficiently search and analyze the data.
Innovation Solution
Implementing an event-based data intake and query system like SPLUNK® ENTERPRISE, which uses a late-binding schema to index and store data, allowing for flexible schema development and refinement at search time, and employing techniques such as parallel search operations, keyword indexing, and high-performance analytics stores to optimize index file size and enhance search efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If more data is indexed to improve search coverage, then search accuracy is improved, but index file size increases leading to higher storage costs
Solution Approach 1:
The patent applies local quality by differentiating index file formats based on event source types. Machine-generated events use a compressed binary format while user-generated events use a text-based format. This selective approach optimizes storage efficiency for machine events (which dominate volume) while maintaining readability and searchability for user events, thereby improving search accuracy without proportionally increasing storage costs.
Solution Approach 2:
The patent changes the parameter of index file format from a single uniform format to multiple formats based on event characteristics. By transitioning from purely text-based indexing to a hybrid system with compressed binary format for machine events, the system achieves better compression ratios and storage efficiency while maintaining the ability to accurately search and retrieve events, thus resolving the contradiction between search accuracy and storage size.
2Quantity of substance
If index file size is reduced to lower storage costs, then storage efficiency is improved, but search speed and accuracy may deteriorate
Solution Approach 1:
The patent applies local quality by using compressed binary format specifically for machine-generated events which constitute the majority of data volume. This targeted compression approach achieves significant storage efficiency improvements for the largest data segment while maintaining separate text-based indexing for user-generated events that require frequent textual search operations, thereby preserving search speed for critical user queries.
Solution Approach 2:
The patent segments the index file system into different formats based on event source type. Machine events are indexed in compressed binary format for storage efficiency, while user events maintain text-based format for searchability. This segmentation allows the system to optimize storage for machine events without compromising search performance for user events, effectively resolving the speed-storage efficiency contradiction.
3Ease of operation
If a uniform text-based index format is used for all events, then searchability is maintained, but storage costs increase significantly
Solution Approach 1:
The patent applies local quality by using text-based format selectively only for user-generated events where human readability and searchability are paramount, while using compressed binary format for machine-generated events where storage efficiency is more critical. This differentiated approach maintains searchability for user events while achieving significant storage cost reductions overall by compressing the larger volume of machine events.
Solution Approach 2:
The patent changes the index format parameter from a uniform text-based system to a conditional system that selects format based on event source type. This parameter change enables the system to maintain text-based searchability where needed while applying compression to reduce storage costs for machine events, effectively resolving the contradiction between searchability and storage cost.
4Quantity of substance
If compressed binary format is used for all events, then storage efficiency is maximized, but search flexibility and readability are reduced
Solution Approach 1:
The patent applies local quality by using compressed binary format specifically for machine-generated events where storage efficiency is the primary concern and search flexibility requirements are lower. User-generated events continue to use text-based format where human readability, interpretation, and search flexibility are critical. This localized application of compression maximizes storage efficiency for machine events while preserving search flexibility for user events.
Solution Approach 2:
The patent segments the indexing system into compressed binary format for machine events and text-based format for user events. This segmentation allows the system to maximize storage efficiency for the machine event portion while maintaining search flexibility and readability for the user event portion, effectively resolving the contradiction between storage efficiency and search flexibility.
Data Source
AI summary
Techniques and mechanisms are disclosed to optimize the size of index files to improve use of storage space available to indexers and other components of a data intake and query system. Index files of a data intake and query system may include, among other data, a keyword portion containing mappings between keywords and location references to event data containing the keywords. Optimizing an amount of storage space used by index files may include removing, modifying and/or recreating various components of index files in response to detecting one or more storage conditions related to the event data indexed by the index files. The optimization of index files generally may attempt to manage a tradeoff between an efficiency with which search requests can be processed using the index files and an amount of storage space occupied by the index files.