Search-Derived Asset Hierarchy for Flexible Machine Data Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Analyzing and searching massive quantities of machine-generated data from diverse sources is challenging due to the complexity and volume of data, requiring efficient processing and presentation methods to derive insights effectively.
Innovation Solution
An event-based data intake and query system, such as the SPLUNK® ENTERPRISE system, uses a late-binding schema to process and store data, allowing flexible extraction of information at search time, enabling the analysis of minimally processed machine data across disparate data sources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a late-binding schema is used to process machine data, then adaptability and flexibility in data analysis are improved, but processing time and computational complexity increase
Solution Approach 1:
The system performs preliminary actions by pre-computing and storing asset tree hierarchies, asset group relationships, and search result metadata before actual data analysis queries are executed. This allows the system to have search results and asset relationships readily available when needed, reducing the time penalty of late-binding schema approaches while maintaining adaptability.
Solution Approach 2:
The system dynamically adjusts its processing approach by caching search results and asset tree data in memory, allowing it to switch between pre-computed cached data and on-demand processing based on query patterns. This dynamic approach optimizes the balance between flexibility and processing speed.
2Speed
If asset tree hierarchy and asset group interface data are cached in memory, then data retrieval speed is improved, but memory resource consumption increases
Solution Approach 1:
The system extracts only the essential and frequently accessed data elements (asset tree hierarchy, asset group relationships, and search result metadata) into memory cache, rather than caching entire datasets. This selective extraction approach provides fast retrieval for critical operations while minimizing memory consumption.
Solution Approach 2:
The system organizes cached data in a nested hierarchical structure where asset tree hierarchies contain nested asset group information, which in turn contains nested search result data. This nested organization allows the system to cache data at multiple levels of granularity, retrieving only the necessary portion from the hierarchy based on query requirements, thus optimizing memory usage.
Data Source
AI summary
An asset monitoring and reporting system (AMRS) implements an interface to establish an asset hierarchy to be monitored and reported against. The interface employs a search query of extant asset data from which definitional aspects of the asset hierarchy can be identified, and therefrom the interface automatically determines control information reflective of the asset hierarchy to direct the ongoing operation of the AMRS. The interface further receives an identification of one or more asset nodes of an asset hierarchy for inclusion in an asset group, receives an identification of a group metric definition of a group metric of the asset group, and reflects in computer storage an asset group definition comprising an association of each of the one or more asset nodes and an association of the group metric definition.