Data Intake Query System Local Storage Eviction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The challenge lies in efficiently managing and analyzing vast amounts of diverse machine data generated by IT environments, which is exacerbated by the complexity of determining what data to store locally versus remotely, especially as storage capacity becomes more abundant and flexible.
Innovation Solution
A data intake and query system utilizing a flexible schema and late-binding schema to process and store machine data, allowing for efficient retrieval and analysis of all generated data, with features like field-searchability and the ability to refine extraction rules over time, and employing a caching subsystem with a cache manager and search bucket predictor to optimize storage and retrieval.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data is stored in local data stores for faster access, then retrieval speed is improved, but storage capacity is limited and data management complexity increases
Solution Approach 1:
The patent segments data storage into local data stores (for frequently accessed data requiring fast retrieval) and remote data stores (for bulk storage capacity). The system divides data management responsibilities between local and remote components, allowing each to be optimized independently for their specific function.
Solution Approach 2:
The patent introduces a data intake and query system as an intermediary layer between local and remote data stores. This mediator handles data routing, caching decisions, and query coordination, simplifying the complexity of managing distributed storage by providing a unified interface and automated management logic.
2Adaptability or versatility
If all generated machine data is retained for comprehensive analysis, then analytical flexibility is improved, but storage requirements and system complexity increase
Solution Approach 1:
The patent extracts only the essential metadata and indexing information needed for analysis to the local data store, while retaining the complete raw data in remote storage. This allows comprehensive analytical flexibility by keeping all data available while reducing local system complexity and storage requirements.
Solution Approach 2:
The patent adds a temporal and spatial dimension to data management by implementing a hierarchical storage architecture. Data can be accessed from multiple levels (local cache and remote storage), and the system manages data across different time scales (frequently accessed vs. historical data), thereby maintaining analytical flexibility without proportionally increasing system complexity.
3Adaptability or versatility
If minimally processed data is stored for later analysis, then data flexibility is improved, but storage capacity requirements increase
Solution Approach 1:
The patent applies local quality by storing different types of data in different locations with different processing levels. Frequently accessed data and metadata are stored locally in their processed form for quick retrieval, while bulk raw data is stored remotely with minimal processing. This allows the system to maintain data flexibility where needed while optimizing storage capacity utilization.
Data Source
AI summary
Systems and methods are disclosed for making space available in a local storage of a data intake and query system. A cache manager of the data intake and query system may determine an amount of storage space of a local data store that is available for use to perform a query. The cache manager may then use one or more eviction policies associated with content stored at the local data store to purge content items to evict from the local storage. The system may then retrieve content for performing the query from a remote storage and store the retrieved content at the local storage.