Accelerated Data Model Summary for Search Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Modern data centers face inefficiencies in processing and retrieving massive volumes of machine-generated data, leading to delays in search results due to the need for extensive computational operations and data processing at search time.
Innovation Solution
The implementation of an accelerated data model summary stored in an external data system, which allows for parallel processing and optimized data retrieval using a columnar format, reducing the need for extensive data filtering and improving search performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If raw machine-generated data is processed at search time, then comprehensive search results can be obtained, but processing time and computational resources increase significantly
Solution Approach 1:
The patent applies preliminary action by pre-computing and storing accelerated data model summaries before search operations. The system identifies frequently queried fields from data models, extracts and aggregates their values in advance, and stores them in an external data system. When a search is performed, the system retrieves these pre-computed summaries instead of processing raw data, thereby resolving the contradiction between search result completeness and processing time.
2Measurement precision
If extensive data filtering and processing is performed during search operations, then accurate search results are achieved, but CPU usage and system performance deteriorate
Solution Approach 1:
The patent applies the extraction principle by separating the data filtering and processing operations from the search time. The system extracts frequently queried field values from raw data during data model acceleration, creates simplified summaries containing only these extracted values, and stores them externally. During search operations, only these extracted summaries are processed, eliminating the need to filter and process the complete raw data set, thus resolving the contradiction between search result accuracy and system processing efficiency.
3Adaptability or versatility
If data is stored in traditional format for comprehensive querying, then all fields are accessible, but data retrieval speed decreases
Solution Approach 1:
The patent applies local quality by optimizing data storage according to specific access patterns. Instead of storing all data uniformly, the system identifies fields that are frequently queried in data models and stores their values in an accelerated, optimized format in the external data system. This creates different storage qualities for different data: frequently accessed fields are stored in high-speed optimized format, while other data remains in standard format. This resolves the contradiction between field accessibility and data retrieval speed by providing fast access to locally optimized frequently queried fields.
Data Source
AI summary
Embodiments of the present invention are directed to facilitating data model acceleration in association with an external data system. In accordance with aspects of the present disclosure, at a core engine, a search request associated with a data model is received. The data model generally designates one or more fields, from among a plurality of fields, that are of interest for subsequent searches. Thereafter, it is determined that an accelerated data model summary associated with the data model is stored at an external data system remote from the core engine that received the search request. The accelerated data model summary includes field values associated with the one or more fields designated in the data model. A search for the received search request is initiated using the accelerated data model summary at the external data. A set of search results relevant to the search request is obtained and provided to a user device for display to a user.