Enterprise Search Index Partitioning for Memory and Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current search systems face limitations in memory usage and access overhead, particularly when indexing large document corpora, leading to inefficiencies in search operations and high latency in disk-based indexes, while live indexes are constrained by memory capacity and break down under heavy loads.
Innovation Solution
The method involves partitioning a document repository or index in multiple dimensions using metadata attribute values, such as calendar years and sender names, to create separate silos that can be indexed and searched independently, reducing the need for full-text indexing and optimizing access speeds.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If a live index is used to enable fast access to all information, then access speed is improved, but memory usage and overhead increase significantly
Solution Approach 1:
The patent divides the document repository into multiple partitions based on metadata attributes (e.g., calendar years, sender names). Each partition is indexed separately, allowing the system to load only relevant partitions into memory during search operations. This segmentation enables fast access to specific document subsets without requiring the entire index to reside in memory simultaneously.
Solution Approach 2:
The patent introduces a new dimension for index organization by partitioning documents according to metadata attributes rather than traditional full-text indexing alone. This multi-dimensional partitioning allows the system to access documents through attribute-based queries, reducing the amount of data that needs to be loaded into memory while maintaining fast access speeds for targeted searches.
2Quantity of substance
If a disk-based index is used to reduce memory overhead, then memory usage is reduced, but search latency increases
Solution Approach 1:
By partitioning the document repository into attribute-based segments, the system can load only the necessary partitions into memory based on query requirements. This selective loading reduces memory overhead while minimizing disk access operations, thereby reducing search latency compared to loading entire disk-based indexes.
Solution Approach 2:
The patent pre-partitions documents according to metadata attributes during indexing, organizing them into logical groups that can be quickly identified and loaded into memory when needed. This preliminary organization eliminates the need for full disk scans during search operations, reducing search latency while maintaining low memory overhead.
3Adaptability or versatility
If traditional inverted file indexing is used for large corpora, then comprehensive search coverage is achieved, but the index structure breaks down under heavy loads
Solution Approach 1:
The patent segments the large document corpus into multiple attribute-based partitions, preventing any single index structure from becoming overwhelmed. Each partition maintains a manageable size and structure, ensuring stability and reliability even when the overall corpus is very large. This segmentation allows the system to handle heavy loads by distributing the indexing burden across multiple smaller, stable partitions.
Solution Approach 2:
The patent moves beyond traditional single-dimension inverted file indexing by introducing attribute-based partitioning as an additional organizational dimension. This multi-dimensional approach distributes the indexing load across multiple attribute categories (e.g., time, sender, recipient), preventing any single index structure from breaking down under heavy loads while maintaining comprehensive search coverage.
Data Source
AI summary
A search system with a search engine applies a user search query to an index of the documents stored in a document repository for returning a result set of matching documents to a user. In order to more efficiently access, search and retrieve documents stored in document repositories, one of a document repository and an index thereof or both are partitioned in one or more dimensions, and a partition is configured in a specific dimension according to two or more document attribute values selected from one and the same attribute category. This shall enable a search application to access significantly less data in order to determine a search result and shall specifically improve the efficiency of enterprise search systems in a high degree.