Archived Data Search via Late Binding Schema

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Big data intake and search systems face challenges in managing storage costs as customers accumulate large amounts of data over time, leading to increased expenses, and deleting older data restricts historical analysis and search capabilities.

Innovation Solution

A search support system that autonomously archives data to external storage, allowing customers to retrieve and search archived data using a late binding schema, enabling distributed servers to communicate with external storage systems and process search queries efficiently.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Loss of information

If data is retained in the data intake and search system, then search and analysis capabilities are maintained, but storage costs increase

Engineering Contradiction:

Improvehistorical data availabilityVSAvoidstorage space consumption

Core Design Contradiction:

Loss of informationVSQuantity of substance

Solution Approach 1:

The patent segments data storage into two distinct locations: hot data remains in the data intake and search system for active searching and analysis, while cold data is moved to external archive storage. This segmentation allows the system to maintain search capabilities for recent data while reducing storage costs by archiving older data that is less frequently accessed.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent introduces an intermediary archiving system that acts as a bridge between the data intake and search system and external storage. This intermediary maintains indexed data that enables searching of archived data without requiring the full archived data to remain in the original system, thus reducing storage requirements while preserving search functionality.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Quantity of substance

If older data is deleted to reduce storage costs, then storage expenses decrease, but historical search and analysis capabilities are lost

Engineering Contradiction:

Improvestorage spaceVSAvoidhistorical data access

Core Design Contradiction:

Quantity of substanceVSLoss of information

Solution Approach 1:

The patent creates indexed copies of archived data that are stored separately from the full archived data. These indexes contain sufficient information to enable searching and analysis of the archived data without requiring the original data to be stored in the data intake and search system. This copying approach allows historical data access while minimizing storage requirements.

Inventive Principle:

Principle #26Copying

3Quantity of substance

If data is archived to external storage, then storage costs are reduced, but search query processing complexity increases

Engineering Contradiction:

Improvestorage costVSAvoidsearch query processing

Core Design Contradiction:

Quantity of substanceVSDevice complexity

Solution Approach 1:

The patent performs preliminary actions by creating and maintaining indexes of archived data before search queries are executed. These indexes are prepared in advance and stored in a format that facilitates efficient querying. When search queries are submitted, the system can quickly consult these pre-prepared indexes rather than processing raw archived data, thus reducing the complexity of query processing despite the archiving.

Inventive Principle:

Principle #10Preliminary action

Data Source

PatentUS10956362B1Searching archived data

Publication Date: 2021.03.23 CISCO TECHNOLOGY INC

AI summary

Raw data in distributed servers is divided into groups of data called buckets containing raw data that have timestamps that fall within a specific time range. When a bucket becomes inactive a server can archive the bucket to an external storage system. The external storage system containing archived data may be specified in a search query. Archived data from the external storage system is obtained, processed, and a search performed on the processed archived data using the search query.