Log Data Index Catalog for Independent Search and Index Scaling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing indexing and searching infrastructure for telemetry and log data in distributed systems is tightly coupled, making it difficult to scale independently, leading to poor cost/performance trade-offs and outages due to varying workloads in indexing and searching capabilities.
Innovation Solution
Decouple indexing and searching processes by storing log data in cold storage with catalog pointers and using hot storage for search nodes, allowing independent scaling of each function based on demand.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If log data is stored in hot storage for fast access, then search performance is improved, but storage cost increases and only limited data can be kept
Solution Approach 1:
The patent segments the storage system into hot storage and cold storage components. Hot storage holds only recently indexed log data that is frequently searched, while cold storage archives historical log data. This segmentation allows fast search performance for recent data while maintaining large storage capacity for historical data at lower cost.
Solution Approach 2:
The patent introduces an index catalog as an intermediary layer between the search API and the distributed file system. The index catalog stores metadata about log data locations and enables the search system to quickly identify and retrieve relevant log portions from cold storage without scanning entire datasets, thus maintaining search performance while using cold storage.
2Device complexity
If indexing and searching are tightly coupled, then system simplicity is maintained, but independent scaling is prevented and outages propagate
Solution Approach 1:
The patent segments the log processing system into distinct indexing and searching components that operate independently. The indexing pipeline ingests and indexes log data into cold storage, while the search API queries the index catalog and retrieves data from cold storage. This segmentation allows each component to be scaled independently based on workload demands and prevents outages in one component from propagating to the other.
Solution Approach 2:
The patent extracts the searching function from the indexing pipeline by introducing a separate search API that queries the index catalog. The indexing pipeline focuses solely on ingesting and indexing log data into cold storage, while the search API handles all query operations. This extraction enables independent scaling and failure isolation of the searching component from the indexing component.
3Ease of operation
If all log data is kept in hot storage, then complete data accessibility is maintained, but cost increases significantly
Solution Approach 1:
The patent applies local quality by making recently indexed log data (which is most likely to be searched) readily accessible in hot storage, while storing historical log data in cold storage. The index catalog provides metadata that enables efficient retrieval from cold storage when needed. This approach maintains data accessibility for all log data while optimizing cost by keeping only frequently accessed data in expensive hot storage.
Data Source
AI summary
A method comprises receiving, from one or more search nodes of a distributed system, one or more requests for log data, the one or more search nodes being associated with one or more first storage systems; identifying an indexed portion of the log data stored in a second storage system of one or more second storage systems based on a particular request of the one or more requests, the log data being indexed by one or more indexing nodes independently from the one or more requests being received and processed by the one or more search nodes; changing a first number of the one or more search nodes and updating a second number of the one or more indexing nodes independently.