Archive Cut-Off Routing for Cross-Store Performance Queries
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Modern data centers face challenges in efficiently analyzing massive quantities of heterogeneous performance data due to the unstructured nature of the data and the difficulty in indexing and querying it using traditional database systems, which limits the flexibility and efficiency of data analysis.
Innovation Solution
A search support system that utilizes a late-binding schema and parallel processing to analyze minimally processed performance data at search time, enabling flexible schema development and efficient retrieval of data from multiple data stores, including external archives.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is archived to a second data store, then storage efficiency is improved, but data availability deteriorates due to the limited retention period in the first data store
Solution Approach 1:
The system dynamically determines whether to search the first data store or the second data store based on the current time and the archive cut-off date. This dynamic switching ensures that data is retrieved from the appropriate location, maintaining availability while enabling efficient archiving.
Solution Approach 2:
The search support system acts as an intermediary that transparently manages the complexity of searching across multiple data stores with different retention policies. It automatically determines which data store to query based on the archive cut-off date, shielding users from the underlying data availability challenges.
2Stability of the object's composition
If traditional database systems are used to index and query performance data, then data structure is improved, but flexibility and efficiency deteriorate due to the unstructured nature of performance data
Solution Approach 1:
The system changes the parameter of data structure from rigid pre-defined schemas to flexible late-binding schemas. This allows the system to adapt to the unstructured nature of performance data while maintaining query capabilities, thereby improving data analysis efficiency without sacrificing structural integrity.
Solution Approach 2:
The system performs preliminary processing of performance data by storing it in a searchable format with metadata tags that enable efficient querying later. This preliminary action allows the data to be readily accessible for analysis without requiring extensive pre-processing or rigid structuring.
3Adaptability or versatility
If multiple databases of different search support systems are searched, then comprehensive data coverage is improved, but operation complexity deteriorates due to the need to login to each system separately
Solution Approach 1:
The search support system provides universal access to multiple data stores through a single interface. It can search both the first data store and the second data store (archive) using the same search protocol, eliminating the need for users to login to different systems separately while maintaining comprehensive data coverage.
Solution Approach 2:
The system merges the functionality of searching across multiple data stores with different retention policies into a unified search operation. By combining the first data store and the second data store into a single searchable namespace with transparent routing, it simplifies the user experience while maintaining adaptability.
Data Source
AI summary
In embodiments, a computer-implemented method may entail receiving a search request. A first data store and a second data store, that contains data archived from the first data store, may be identified. Data from the first data store may remain available in the first data store for a limited period of time once archived to the second data store. The first data store storing data in a first format and the second data store storing data in a second format, the first format and the second format being different from one another. Determining that a subset of data that has been archived into the second data store and is to be searched as part of the search request is still available from the first data store, and executing the search request on the subset of data utilizing the first data store. Additional embodiments are described and/or claimed.