Late-Binding Schema for Machine Data Identity Resolution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Analyzing and searching massive quantities of diverse machine data generated from various sources, such as system logs, network packet data, and sensor data, is challenging due to its vastness and complexity, as existing tools typically pre-process and discard data, limiting analysis flexibility and efficiency.
Innovation Solution
An event-based data intake and query system with a late-binding schema that allows flexible schema definition and extraction rules, enabling the storage and analysis of minimally processed machine data, facilitating field-searchable events and real-time operational intelligence across disparate data sources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If pre-processing and extracting specified data items is performed, then retrieval and analysis efficiency of extracted data is improved, but flexibility to analyze all generated data is reduced and data loss occurs
Solution Approach 1:
The system segments data processing into two paths: a fast path for pre-processed extracted data and a complete path for raw data. The search system can selectively query either path or both paths simultaneously, allowing efficient retrieval of commonly accessed extracted data while maintaining the ability to analyze all raw data when needed, thus resolving the contradiction between efficiency and flexibility
Solution Approach 2:
The system performs preliminary extraction and processing of frequently accessed data items to create indexed search results, enabling fast retrieval for common queries. However, the raw data is preserved and can be processed later when needed, allowing the system to balance immediate efficiency needs with future flexibility requirements
2Productivity
If pre-processing and extracting specified data items is performed, then retrieval and analysis efficiency of extracted data is improved, but analysis flexibility and insights from comprehensive data are reduced
Solution Approach 1:
The system introduces an intermediary layer that maintains both extracted processed data and raw data in a unified search architecture. This intermediary structure allows the search system to efficiently query extracted data while also providing access to complete raw data, preventing information loss and enabling comprehensive analysis when required
Solution Approach 2:
The search system is designed with multi-functionality to handle both extracted data queries and raw data queries through a single unified interface. This universal approach allows the same system to provide efficient retrieval for extracted data while also enabling comprehensive analysis of all raw data, eliminating the need to choose between efficiency and completeness
3Adaptability or versatility
If massive quantities of raw data are stored for later retrieval, then analysis flexibility is improved, but search and analysis performance is reduced
Solution Approach 1:
The system segments the data search function into two independent search paths: one for extracted pre-processed data and one for raw data. This segmentation allows the system to serve both flexible analysis of all data and efficient retrieval of extracted data, with each path optimized for its specific purposes
Solution Approach 2:
The system performs partial processing of data by extracting and indexing only the most relevant fields and data items for common queries, while preserving the complete raw data. This partial action approach improves search performance for typical queries while maintaining the ability to perform comprehensive analysis when needed
Data Source
AI summary
An identify resolution system performs actions comprises a set-up process and an identity resolution process that executes asynchronously with respect to the set-up process. the set-up process includes accessing machine data including a plurality of event data objects, each event data object of the plurality of event data objects including timestamped raw machine-generated data indicative of performance or operation of one or more entities in a computer network environment. The identity resolution process ascertains the identity of an entity associated with the computer network environment, based on the association data in the data store, wherein the identity of the entity is not expressed directly in the association data in the data store.