Elastic Sharding for Multi-Tenant Cloud Data Indexing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In cloud environments, the increased quantity and size of data complicate the process of indexing and searching structured data, leading to input/output bottlenecks and inefficiencies in scalability.
Innovation Solution
An elastic scalable architecture is implemented for indexing and searching structured data, utilizing sharding across multiple nodes with dynamic elasticity to manage data distribution and capacity allocation, ensuring high availability, scalability, and flexibility.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is stored in dedicated databases with traditional indexing, then data can be efficiently stored and searched, but the system lacks scalability and creates input/output bottlenecks when data quantity and size increase in cloud environments
Solution Approach 1:
The patent divides the database into multiple shards distributed across different nodes in the cloud environment. Each shard contains a portion of the data and can be independently queried, indexed, and managed. This segmentation allows the system to handle large datasets by distributing the indexing and searching workload across multiple nodes, thereby maintaining efficiency while improving scalability.
Solution Approach 2:
The patent introduces a hierarchical sharding structure where data is first sharded across multiple nodes, and then further divided into segments within each node. This multi-dimensional approach to data organization enables the system to scale vertically by adding more nodes and horizontally by distributing data across multiple segments, resolving the contradiction between efficiency and scalability.
2Quantity of substance
If more data is stored in the cloud, then data capacity increases, but input/output bottlenecks occur and indexing becomes more complex
Solution Approach 1:
The patent segments data into multiple shards that are distributed across different nodes, which simplifies indexing by allowing each node to index only its local shard independently. This reduces the complexity of indexing large datasets compared to indexing the entire database on a single node.
Solution Approach 2:
The patent implements dynamic shard allocation where data can be redistributed across nodes based on changing workloads and data characteristics. This dynamic approach allows the system to optimize indexing complexity by adjusting the distribution of data and indexing operations in response to varying conditions.
3Reliability
If data is sharded across multiple nodes, then scalability and availability improve, but system complexity increases
Solution Approach 1:
The patent segments data into independent shards that can be distributed across multiple nodes, improving availability by ensuring that failures of individual nodes do not affect the entire system. Each shard can continue to operate independently, maintaining service continuity.
Solution Approach 2:
The patent implements replication of shards across multiple nodes, where copies of data are stored and indexed on different nodes. This copying mechanism improves availability by allowing read operations to continue even if some nodes are unavailable, while the replication itself is managed through automated mechanisms that reduce manual intervention.
Data Source
AI summary
In an example embodiment, primary data is received from a first tenant in a computer network. Then auxiliary data relating to the primary data is received from the first tenant. A first index is created for the primary data and the auxiliary data from the first tenant. Then the first index is stored as a first shardlet. The first shardlet is bundled with one or more other shardlets for the tenant in a shard group. The shard group is packed with one or more other shard groups in a first shard. Then the first shard is stored in a first instance of a distributed database, the distributed database comprising a plurality of instances, each instance operating on a different logical or physical device.