Pipelined Search Query ML Integration Visualization

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Current data analytics platforms face challenges in efficiently searching and analyzing vast amounts of diverse data across different programming languages and data sources, particularly with the integration of machine learning models and streaming algorithms, due to lack of frameworks that support cross-language deployment and reuse of ML components in search workflows.

Innovation Solution

The implementation of a data intake and query system that uses a late-binding schema and pipelined command language to process, index, and store data, allowing for flexible extraction and reuse of machine learning components within search queries, and enabling in-line visualization and handling of diverse data formats through configuration files and extraction rules.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If data is pre-processed before storage to reduce data volume, then storage efficiency and retrieval speed improve, but data flexibility and analysis capability deteriorate because only pre-selected data items are retained

Engineering Contradiction:

Improvedata retrieval speedVSAvoiddata analysis flexibility

Core Design Contradiction:

ProductivityVSAdaptability or versatility

Solution Approach 1:

The system performs preliminary indexing and metadata extraction during data ingestion, organizing data into searchable structures without discarding raw data. This allows both fast retrieval through indexes and flexible analysis through access to complete raw data when needed.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system applies different processing levels to different portions of data: highly processed indexes for fast search operations, and raw/unprocessed data for comprehensive analysis. Each data representation serves its specific purpose optimally.

Inventive Principle:

Principle #3Local quality

2Measurement precision

If machine learning models are integrated into search queries to improve analysis capability, then insight quality improves, but search latency and system complexity increase

Engineering Contradiction:

Improveanalysis accuracyVSAvoidsearch latency

Core Design Contradiction:

Measurement precisionVSLoss of time

Solution Approach 1:

The ML pipeline is divided into distinct operators that can be executed independently and in parallel. Each operator processes specific aspects of the data, allowing for optimized execution and reducing overall latency compared to monolithic ML processing.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

Data is pre-processed and indexed before ML analysis, so that when ML operators are invoked, they receive pre-formatted, filtered, or aggregated data that reduces computation time while maintaining analysis accuracy.

Inventive Principle:

Principle #10Preliminary action

3Adaptability or versatility

If multiple programming languages are supported for data analytics to improve versatility, then platform adaptability improves, but search performance and system complexity deteriorate due to language translation overhead

Engineering Contradiction:

Improveprogramming language supportVSAvoidsearch performance

Core Design Contradiction:

Adaptability or versatilityVSProductivity

Solution Approach 1:

The system employs a universal query language (SPL) that serves as a common interface for all programming languages. ML operators are deployed once and can be invoked from multiple languages without requiring separate implementations or translations, maintaining both versatility and performance.

Inventive Principle:

Principle #6Universality (Multi-functionality)

4Adaptability or versatility

If raw data is stored without pre-processing to improve data completeness, then analysis flexibility improves, but storage requirements and processing complexity increase

Engineering Contradiction:

Improveanalysis flexibilityVSAvoidstorage capacity

Core Design Contradiction:

Adaptability or versatilityVSQuantity of substance

Solution Approach 1:

The system stores raw data in compressed or encoded formats that preserve complete information while reducing storage footprint. Multiple levels of data representation (raw, indexed, aggregated) are nested within the same storage infrastructure, allowing efficient access at different granularities.

Inventive Principle:

Principle #7Nested doll (Nesting)

Data Source

PatentUS11748634B1Systems and methods for integration of machine learning components within a pipelined search query to generate a graphic visualization

Publication Date: 2023.09.05 CISCO TECHNOLOGY INC

AI summary

A computer-implemented method for integration of machine learning components within a pipelined search query to generate a visualization is described. Herein, an interface is provided for receipt of pipelined code into a web-based programming application. The pipelined code features a series of operators configured to perform one or more tasks based on collective operations by the series of operators, wherein a first operator of the series of operators is to receive input data from a selected data source and each remaining operator of the series of operators to receive input based on an output from a preceding operator of the remaining operators. The task(s) performed by the pipelined code generate results including visualizations. The visualization is rendered in a manner that allows the pipelined code to be scrolled to display the pipelined code or the visualization.