Late-Binding Schema for ML Search Latency Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data analytic platforms face challenges in providing efficient search capabilities across multiple programming languages without unacceptable latency, particularly with the increasing complexity of search queries due to the integration of machine learning models and diverse data sources, and lack mechanisms for reusing ML-based pipelines in data science workflows.
Innovation Solution
The implementation of an event-based data intake and query system that uses a late-binding schema to process and index machine data, allowing for flexible schema definition and extraction rules, enabling efficient search and analysis across disparate data sources and supporting the deployment of ML models within search workflows.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If pre-processing data and storing only specified data items are performed, then data retrieval efficiency is improved, but data flexibility and analytical flexibility are reduced
Solution Approach 1:
The patent segments data into two components: a schema definition layer that describes data structure, and actual data storage. This allows efficient retrieval through schema-guided processing while maintaining flexibility to analyze all raw data when needed, resolving the contradiction between retrieval efficiency and analytical flexibility.
Solution Approach 2:
The patent implements dynamic schema binding where the schema is applied at query time rather than fixed during ingestion. This enables the system to adapt its processing behavior based on the specific query requirements, allowing both efficient retrieval for common cases and flexible analysis for ad-hoc queries.
2Adaptability or versatility
If machine learning models are integrated into search queries, then search capability is enhanced, but search latency increases
Solution Approach 1:
The patent pre-processes and indexes data according to the schema before queries arrive. This preliminary organization of data allows ML models to operate on pre-structured information during search, reducing the computational overhead and latency that would otherwise result from processing raw data in real-time.
Solution Approach 2:
The schema acts as an intermediary between the raw data and the ML models. It transforms and structures the data in a way that optimizes for both search efficiency and ML model performance, enabling enhanced search capability without excessive latency by avoiding direct processing of raw data.
3Adaptability or versatility
If programming language translations are performed for search queries, then multi-language support is achieved, but search latency increases
Solution Approach 1:
The patent creates a standardized internal representation (schema) that copies and normalizes data from various programming languages and formats. This unified representation eliminates the need for real-time translation during search, as all queries are processed through the same schema framework, thereby supporting multiple languages without increasing latency.
4Adaptability or versatility
If all machine data is stored for analysis, then analytical flexibility is improved, but storage requirements and system complexity increase
Solution Approach 1:
The patent applies local quality by storing all machine data in its original form but organizing it according to a schema that provides structure only where needed for retrieval. This maintains analytical flexibility to examine any data while reducing system complexity through standardized organization rather than complete processing of all data.
Data Source
AI summary
A computer-implemented method is disclosed that includes operations of parsing a query comprised of a sequence of operators to detect each operator of the sequence of operators, where the sequence of operators includes a machine learning (ML) operator representing a trained ML model. Additionally, a schema of the ML operator is determined through metadata. A filter or a projection is generated based on the schema of the ML operator, where the filter or projection is configured to reduce an amount of data retrieved upon application of the filter of the projection to an operator of the sequence of operators comprising the query. The schema of the ML operator indicates a schema of input data to be provided to the ML operator and a schema of output data to be provided by the ML operator following processing.