External Query Result Reading via Translation and Parallel Readers
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Analyzing and searching massive quantities of machine data from diverse sources, including system logs, network packet data, sensor data, and application program data, is time-consuming due to the varied formats and types of data, especially with the increasing trend of retaining minimally processed data for later retrieval and analysis.
Innovation Solution
An event-based data intake and query system with a flexible schema that applies extraction rules 'on the fly' at search time, enabling field-searchable events and a late-binding schema to handle diverse data formats, allowing for efficient querying and analysis of machine data across disparate sources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data is pre-processed and extracted before storage, then retrieval efficiency is improved, but data flexibility and analysis capability are reduced
Solution Approach 1:
The system performs preliminary indexing and metadata extraction during data ingestion, organizing data into a searchable structure without discarding raw data. This allows efficient retrieval through indexes while preserving the ability to analyze complete raw data when needed, resolving the contradiction between retrieval speed and analysis flexibility.
Solution Approach 2:
The patent implements a nested data structure where raw data is stored in its complete form, and multiple layers of extracted metadata and indexes are nested within or alongside it. This allows the system to access extracted data quickly for routine queries while enabling deep analysis of the complete nested structure when required.
2Adaptability or versatility
If all raw data is stored without preprocessing, then data flexibility and completeness are improved, but search and analysis performance deteriorate
Solution Approach 1:
The patent segments data into multiple representational layers: complete raw data for comprehensive analysis, extracted metadata for contextual understanding, and indexes for rapid retrieval. This segmentation allows the system to operate at different levels of detail depending on query requirements, maintaining both flexibility and performance.
Solution Approach 2:
Different portions of the data structure are optimized for different purposes: indexes and metadata are optimized for fast search and retrieval operations, while raw data is preserved in its complete form for detailed analysis. This local optimization resolves the contradiction by applying different quality characteristics to different parts of the data system.
3Loss of information
If diverse data formats are retained in original form, then data completeness is improved, but query complexity and processing time increase
Solution Approach 1:
The patent introduces metadata and standardized schemas as intermediary layers between diverse raw data formats and query processing mechanisms. These intermediaries translate and normalize data from various sources into a common structure, enabling efficient querying without losing the completeness or original format of the underlying raw data.
Data Source
AI summary
A computing device can receive a query in a first query language that identifies a set of data to be processed and determine that at least a portion of the set of data resides in an external data system that uses a different query language. The query system can translate the query in the first query language in to a second query language for the external data system. To retrieve results of the translated query, the computing device may determine a quantity of results of generated by the translated query and generate one or more results readers to read the results in parallel. The computing device may further process the results and provide them to a user.