Enterprise-Specific Sharding for Distributed Inverted Indexes

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Current file storage and search systems distribute files across shards uniformly, leading to inefficiencies when one shard is unavailable, causing bottlenecks and unnecessary resource consumption, as queries are routed to all shards regardless of relevance, and scaling increases query latency.

Innovation Solution

A deterministic sharding approach that routes queries to specific shards based on enterprise identifiers, grouping documents from the same enterprise together, using file keys composed of enterprise ID, folder ID, and file ID hashes to logically partition data across physical shards, allowing for targeted query routing and reducing query fanout.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Stability of the object's composition

If queries are routed to all shards uniformly, then shards grow uniformly, but query latency increases and resource consumption increases when one shard is unavailable

Engineering Contradiction:

Improveuniform shard growthVSAvoidquery latency

Core Design Contradiction:

Stability of the object's compositionVSLoss of time

Solution Approach 1:

The patent segments queries into enterprise-specific query sets and routes them to corresponding enterprise-specific shards. This segmentation allows queries to be processed only on relevant shards rather than all shards, reducing query latency while maintaining uniform shard growth through deterministic distribution of different enterprises across multiple shards.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent applies local quality by customizing query routing based on enterprise identifiers. Each enterprise's documents are concentrated on specific shards, allowing queries to be optimized for local (enterprise-specific) data locations rather than distributed uniformly across all shards, thereby reducing unnecessary query processing time.

Inventive Principle:

Principle #3Local quality

2Adaptability or versatility

If queries are routed to all shards, then query fanout is maximized, but resource wastage increases due to processing irrelevant data

Engineering Contradiction:

Improvequery coverageVSAvoidresource consumption

Core Design Contradiction:

Adaptability or versatilityVSLoss of energy

Solution Approach 1:

The patent extracts relevant enterprise identifiers from query parameters and uses them to filter and route queries to only the shards containing that enterprise's documents. This extraction and filtering mechanism eliminates unnecessary query processing on irrelevant shards, reducing resource consumption while maintaining comprehensive query coverage for each enterprise.

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

The patent performs preliminary action by pre-computing and storing enterprise-to-shard mappings during the indexing phase. This preliminary routing information is stored in the system metadata, allowing queries to be immediately routed to the correct shards without needing to scan all shards, thus reducing real-time resource consumption while maintaining full query coverage.

Inventive Principle:

Principle #10Preliminary action

3Reliability

If a single shard becomes unavailable, then the system can failover, but the whole cluster becomes unusable without backup

Engineering Contradiction:

Improvesystem availabilityVSAvoidcluster dependency

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The patent segments data by enterprise and distributes different enterprises across different shards. This segmentation ensures that the failure of one shard affects only the enterprises stored on that specific shard, not the entire cluster. Other shards continue to serve their respective enterprises, maintaining system availability without requiring complex backup mechanisms.

Inventive Principle:

Principle #1Segmentation

4Quantity of substance

If the system is scaled by adding a new machine, then storage capacity increases, but query fanout increases and query latency impacts worsen

Engineering Contradiction:

Improvestorage capacityVSAvoidquery latency

Core Design Contradiction:

Quantity of substanceVSLoss of time

Solution Approach 1:

The patent performs preliminary action by pre-determining which shards should handle which enterprises based on deterministic hashing of enterprise identifiers before queries are executed. When new machines are added to the cluster, the system can assign new enterprises to new shards without changing the routing logic for existing enterprises, maintaining fast query response times while increasing storage capacity.

Inventive Principle:

Principle #10Preliminary action

Data Source

PatentUS20230185765A1Systems and methods for sharding based on distributed inverted indexes

Publication Date: 2023.06.15 BOX INC

AI summary

According to one embodiment, distributing data across a plurality of storage shards can comprise generating a file key for each file of a plurality of files stored in a plurality of physical shards, each physical shard maintained by a node of a plurality of nodes in one or more clusters. The file key can comprise a hash of an enterprise identifier for an entity to which the creator of the file is a member, a hash of a folder identifier for a location in which the file is stored, and a hash of a file identifier uniquely identifying the file. The generated file keys can be sorted into an ordered list and the ordered list can be logically partitioning into a plurality of logical shards. Each logical shard of the plurality of logical shards can then be mapped to one of the plurality of physical shards.