Active Pipeline Text File Updates Without Restart

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Current tools lack the capability to efficiently and intuitively search and analyze large sets of raw machine data for insights, as they often require pre-processing and discard significant amounts of data, limiting flexibility and insight derivation.

Innovation Solution

A data intake and query system that utilizes a late-binding schema and indexing techniques to store and process raw machine data, allowing for flexible schema development and field-searchable events, enabling users to extract insights through a user-friendly interface and pipelined search language.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If pre-processing and extracting specified data items from generated data, then retrieval and analysis efficiency is improved, but data flexibility and completeness deteriorate because only extracted data is available for later analysis

Engineering Contradiction:

Improveretrieval and analysis efficiencyVSAvoiddata flexibility

Core Design Contradiction:

ProductivityVSAdaptability or versatility

Solution Approach 1:

The system segments data processing into two paths: a pre-processing path that extracts and indexes specified data items for efficient retrieval, and a raw data storage path that preserves complete unprocessed data for flexible later analysis. This segmentation allows both efficiency and flexibility to coexist by serving different analysis needs through appropriate data paths.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The system changes the state of data from completely unprocessed raw form to partially processed extracted form, and maintains both states. This parameter change approach allows the system to optimize for different scenarios: extracted data for efficiency-critical operations, raw data for flexibility-critical operations.

Inventive Principle:

Principle #35Parameter changes

2Adaptability or versatility

If storing massive quantities of raw data for later retrieval, then data flexibility and analysis completeness are improved, but system complexity and resource requirements worsen

Engineering Contradiction:

Improvedata flexibilityVSAvoidsystem complexity

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The system segments data management into separate components: a raw data storage subsystem that maintains flexibility, and a pre-processed data indexing subsystem that manages complexity. By separating these functions, the system can handle massive raw data volumes without proportionally increasing overall system complexity.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The system introduces an intermediary indexing layer between raw data storage and analysis operations. This intermediary component abstracts the complexity of managing massive raw data volumes while preserving access flexibility, acting as a mediator that simplifies the interface between data storage and data analysis.

Inventive Principle:

Principle #24Intermediary (Mediator)

3Loss of information

If using traditional search tools that collect results over a network, then data analysis capability is improved, but ease of operation and user-friendliness deteriorate due to complex piecemeal analysis process

Engineering Contradiction:

Improvedata analysis capabilityVSAvoiduser-friendliness

Core Design Contradiction:

Loss of informationVSEase of operation

Solution Approach 1:

The system merges data collection, indexing, and search operations into a unified platform with a single user interface. Instead of requiring users to collect results from multiple network sources separately, the system combines these functions so users can perform comprehensive data analysis through a single intuitive interface, dramatically improving ease of operation.

Inventive Principle:

Principle #5Merging (Combining)

Solution Approach 2:

The system creates a universal data analysis platform that handles multiple data types and analysis operations through a single interface. This multi-functional approach eliminates the need for users to navigate complex piecemeal analysis processes across different tools, as one interface provides comprehensive data collection, search, and visualization capabilities.

Inventive Principle:

Principle #6Universality (Multi-functionality)

Data Source

PatentUS11687487B1Text files updates to an active processing pipeline

Publication Date: 2023.06.27 CISCO TECHNOLOGY INC

AI summary

Systems and methods are described for updating text files for a processing pipeline without restarting the processing pipeline. A processing pipeline may include a frontend thread and a backend thread. The frontend thread of the processing pipeline may generate transformed data using the text file. A backend thread of the processing pipeline may periodically determine whether an updated text file has been uploaded. The backend thread can determine that an updated text file has been uploaded and cause the frontend thread to pause generating transformed data. The backend thread can validate the updated text file by comparing the text file and the updated text file. Based on validating the updated text file, the backend thread can cause the frontend thread to resume transforming data using the updated text file.