De-duplication Indexes for Efficient Array Field Querying

The patent segments the index structure into two distinct components: a de-duplication index that stores unique array field values with their document identifiers, and a conventional index that maintains the traditional key-value structure. This segmentation allows each index type to serve its specific purpose optimally, resolving the contradiction between query performance and structural complexity by dividing the indexing function into specialized parts.

The patent merges the advantages of de-duplication indexing (reduced storage, improved query speed for array fields) with conventional indexing (flexibility, broad applicability) by creating a hybrid indexing system. The query processor can selectively use either index type based on the query characteristics, thereby achieving optimal query performance while maintaining indexing flexibility without being constrained by conventional index restrictions.

The patent creates a simplified copy of the array field data in the de-duplication index, storing only unique values with their associated document identifiers. This copying approach reduces the number of index entries while maintaining query capability, as the index contains a representative subset (unique values) that suffices for query operations. The system handles duplicate values by mapping them to the same unique entry, thereby achieving both entry reduction and duplicate value handling.

Instead of storing each array element individually as in conventional indexes, the patent inverts the approach by storing unique values and maintaining a mapping from value to document identifiers. This inversion allows the index to efficiently handle arrays with duplicate values by consolidating multiple occurrences into a single index entry, thereby reducing the number of entries while preserving the ability to retrieve all documents containing a given value.

The patent segments the query processing into independent filter evaluation stages, where each filter on array fields can be evaluated separately using the de-duplication index. This segmentation enables parallel processing of multiple filters, as each filter's evaluation can proceed independently without blocking others, thereby maintaining complex query capability while significantly improving execution efficiency through concurrent processing.

The patent performs preliminary organization of array field data in the de-duplication index during index creation and maintenance operations. By pre-organizing the data with unique values and their document identifier mappings, the system prepares the index structure in advance to support efficient parallel filter evaluation. This preliminary action eliminates the need for complex runtime processing of duplicate values, enabling multiple filters to be applied in parallel without efficiency loss.