Cross-Tenant Data Analysis via Late-Binding Schema and Inverted Indexing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current tools lack the capability to efficiently and flexibly search and analyze large sets of raw machine data for insights, as they often require pre-processing and discard significant amounts of data, limiting the ability to investigate all generated data.
Innovation Solution
A data intake and query system that utilizes a late-binding schema, allowing for flexible data extraction and storage of raw machine data, enabling field-searchable events with a pipelined search language and inverted indexing for efficient querying and analysis.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If pre-processing is applied to extract specified data items for efficient retrieval, then retrieval efficiency is improved, but data flexibility and completeness are worsened because significant amounts of data are discarded
Solution Approach 1:
The system performs preliminary indexing of all raw data fields without pre-processing or discarding any data. The inverted index structure is built in advance to enable efficient retrieval while maintaining all original data for future flexible querying.
Solution Approach 2:
The search system is designed to handle multiple types of queries (full-text search, field-specific search, aggregation queries) on the same indexed data structure, making it universally applicable to diverse data analysis needs without requiring pre-processing for specific query types.
2Adaptability or versatility
If all raw data is stored for later analysis, then data completeness and flexibility are improved, but search and analysis complexity increases
Solution Approach 1:
The inverted index acts as an intermediary data structure between the raw data storage and the query interface. It transforms the complexity of searching through all raw data into simple lookups of pre-computed index structures, reducing search complexity while maintaining data completeness.
Solution Approach 2:
The system creates a copied and reorganized version of the data in the inverted index structure, where the data is rearranged for efficient retrieval without altering the original raw data. This copy enables fast searching while the original data remains intact for completeness.
3Measurement precision
If traditional search tools are used to search data systems separately, then individual data system analysis is improved, but overall data analysis efficiency is worsened due to piecemeal result collection
Solution Approach 1:
The system merges multiple data sources into a unified search index, allowing simultaneous searching across all data systems. The inverted index consolidates data from multiple sources, enabling analysts to search all data in one operation rather than collecting results piecemeal from separate systems.
Data Source
AI summary
An information technology (IT) and security operations application is described that enables cross-tenant analyses of data to derive insights that can be used to provide actionable information across the application including, for example, action recommendations, threat confidence scores, and other incident data enrichments. The generation and presentation of such information to users of an IT and security operations application can enable analyst teams to more efficiently and accurately respond to various types of incidents in IT environments, thereby improving the overall operation and security of the IT environments. Furthermore, because of the shared use of an IT and security operations application concurrently by any number of separate tenants, such cross-tenant analyses can be performed in near real-time and on an ongoing basis to deliver relevant insights.