Distributed Index Sharding for Scalable Search Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Maintaining accurate and scalable search indices across large intranet systems is challenging due to the high effort and cost involved in compiling and updating master indices, with servers acting as a single point of failure.
Innovation Solution
Implementing a distributed index system where servers hosting shards are located near content items, and a database server manages searching features like user profiles and query history to determine relevant shards for search requests, reducing the need for a massive master index and allowing independent updates to each shard.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a master index referencing all content items across thousands of content servers is compiled and maintained, then accurate indexing of content items is achieved, but the effort, cost, and time required for compilation and updates increase significantly
Solution Approach 1:
The patent divides the monolithic master index into multiple distributed child indices (shards), where each shard maintains a subset of content item references. This segmentation allows independent compilation and updates of each shard without requiring global index synchronization, thereby maintaining indexing accuracy while dramatically improving maintenance efficiency and scalability.
Solution Approach 2:
The patent introduces a hierarchical dimension to the index structure by creating a master index that references child indices, which in turn reference content items. This two-level hierarchy transforms the flat index structure into a distributed tree structure, enabling parallel index maintenance across multiple shards while preserving accurate content item referencing through the hierarchical path.
2Loss of information
If a master index is updated immediately upon content item changes to ensure accuracy, then indexing freshness is maintained, but system latency and update overhead increase
Solution Approach 1:
By segmenting the master index into distributed child indices, the patent enables asynchronous and independent updates of individual shards. When content items change, only the affected child index shard needs to be updated, not the entire master index. This maintains index freshness for relevant content while minimizing update latency and avoiding system-wide synchronization overhead.
3Measurement precision
If a centralized master index server is used to manage all content references, then centralized control and accuracy are achieved, but the system becomes a single point of failure
Solution Approach 1:
The patent segments the centralized master index into multiple distributed child indices stored across different servers. This distribution eliminates the single point of failure by ensuring that if one index server fails, other shards remain accessible and can still provide search functionality. Index accuracy is maintained through the hierarchical structure where the master index references the distributed child indices.
Solution Approach 2:
The patent changes the architectural parameter from centralized to distributed index storage. By deploying child indices across multiple servers and locations, the system achieves both index accuracy (through the hierarchical reference structure) and improved reliability (through distribution and redundancy).
Data Source
Figure 1
Figure 2A
Figure 2B
AI summary
Computer systems, devices, and associated methods of providing distributed index searching are disclosed herein. In one embodiment, a method includes receiving, at a database server, search terms included in a search query for content from a member and in response to receiving the search terms, determining a subset of the distributed indices to be searched in response to the search query based on the received search terms and one or more records of searching features on the database server. The method also includes providing the determined list of distributed indices to be searched based on the search terms in the search query.