Federated Indexing via Hashed Primary Key Slices
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional data management systems face challenges in handling large volumes of data and search indices, as a single server may struggle to manage the data items and search indices efficiently, leading to the need for distributing these across multiple storage locations.
Innovation Solution
A method and system that utilize hash values from primary keys to index a chain of search indices across multiple storage locations, allowing for efficient distribution and management of data items, with each server responsible for specific primary keys and using a chain of search indices to locate data items.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data items and search index are stored in a single data repository, then data management is simple, but the system cannot handle large volumes of data efficiently
Solution Approach 1:
The search index is divided into multiple segments distributed across different servers in a federated system. Each server maintains a portion of the search index, allowing the system to handle larger volumes of data by parallelizing storage and query operations across multiple nodes rather than concentrating all data in a single repository.
2Productivity
If search index is distributed across multiple servers, then data handling capacity increases, but index management becomes complex
Solution Approach 1:
A coordinator server acts as an intermediary in the federated system, managing the distribution and coordination of search index segments across multiple servers. The coordinator handles queries by routing them to appropriate servers and aggregating results, thereby simplifying index management while maintaining distributed storage benefits.
3Speed
If a single server manages all search indices, then system architecture is simple, but search engine speed decreases due to bottlenecks
Solution Approach 1:
The search workload is segmented and distributed across multiple servers, each handling specific portions of the search index. This parallelization eliminates the bottleneck of a single server by allowing simultaneous query processing across multiple nodes, thereby increasing overall search engine speed.
Solution Approach 2:
The system transitions from a single-dimensional architecture (one server) to a multi-dimensional federated architecture where search operations can be performed in parallel across multiple servers. This dimensional expansion allows the system to process queries concurrently, significantly improving search speed.
4Quantity of substance
If search index size increases to handle more data, then data capacity increases, but single-server management becomes difficult
Solution Approach 1:
The large search index is segmented into smaller manageable portions distributed across multiple servers. Each server manages a subset of the total index, making the overall large index tractable through division. This segmentation allows the system to handle large quantities of data while maintaining ease of management at each individual server level.
Data Source
AI summary
A method and system stores and retrieves data items associated with a primary key, using search indices at multiple storage locations. A server receives a primary key, identifies one or more segments of the primary key, and hashes each segment with one or more hash functions to obtain a sequence of hash values. The hash values are used as keys to index a chain of search indices that are stored in multiple storage locations. One or more of the hash values in the sequence are used to form a host name, and the host name is mapped to an address of a server that stores a first search index in the chain. The last search index in the chain contains the data items associated with the primary key, or provides a reference to one or more locations at which the data items can be found.