Search Engine Index Offloading to External Object Store

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Conventional search engines face high hardware costs and technical complexity due to the need for internal persistent local disks for index replication and storage.

Innovation Solution

A search system that offloads index replication and storage to an external object store, separating ingest operations from search operations and using a controller to independently manage resources for the indexing and search tiers.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If conventional search engines replicate data between nodes and store data on local persistent disks for resilience and durability, then reliability is improved, but hardware costs increase and device complexity increases

Engineering Contradiction:

Improveresilience and durabilityVSAvoidsystem complexity

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The patent extracts the data persistence function from the search engine nodes themselves and places it in an external object store. The index structures are stored externally rather than replicated across multiple node disks, simplifying the search engine architecture while maintaining durability through the external storage service.

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

The external object store serves as a universal persistence layer for the distributed search system. Instead of each node requiring its own persistent storage infrastructure, a single external object store provides persistence services to all nodes, reducing overall system complexity and hardware requirements.

Inventive Principle:

Principle #6Universality (Multi-functionality)

2Reliability

If conventional search engines replicate data between nodes and store data on local persistent disks, then reliability is improved, but hardware costs increase

Engineering Contradiction:

Improveresilience and durabilityVSAvoidhardware costs

Core Design Contradiction:

ReliabilityVSQuantity of substance

Solution Approach 1:

The patent extracts the persistent storage requirement from individual node hardware and consolidates it into an external object store. This eliminates the need for multiple high-capacity local disks across search engine nodes, reducing hardware costs while maintaining data durability through the external storage service.

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

Instead of replicating data across multiple node disks, the system uses the external object store to maintain copies of index structures. The object store's internal replication mechanisms provide durability without requiring expensive hardware replication at each node.

Inventive Principle:

Principle #26Copying

Data Source

PatentUS20250117373A1Search and analytics engine with object store persistence

Publication Date: 2025.04.10 ELASTICSEARCH BV

AI summary

According to an aspect, a search system includes an indexing tier and a search tier. The indexing tier is configured to generate an index structure from a plurality of documents, store the index structure in a local storage device, and transmit the index structure to an external object store, the external object store configured to store the index structure. The search tier is configured to receive a search query from a client device and communicate with the external object store or the indexing tier to retrieve at least a portion of the index structure to search data based on the search query.