Microshard Mapping for Dynamic Data Sharding Overhead
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Data partitioning services like data sharding become inefficient over time due to the need for frequent updates in client devices when data allocation changes, leading to resource wastage from overprovisioning.
Innovation Solution
Implementing microshards within shards, with a microshard map to manage mappings and a migration controller for seamless data access, allowing continued access using the same microshard ID even after migration, reducing computing overhead and enabling on-demand provisioning.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data sharding is used to manage large amounts of user data, then data storage efficiency is improved, but computing resources are wasted due to overprovisioning of shards
Solution Approach 1:
The patent implements dynamic shard provisioning where shards are created and deactivated based on actual data storage needs. The system dynamically adjusts the number of active shards rather than maintaining a fixed overprovisioned set, allowing computing resources to be allocated efficiently as data grows or shrinks.
Solution Approach 2:
The system changes the parameter of shard count dynamically based on data volume and access patterns. By monitoring data growth and adjusting the number of active shards accordingly, the system avoids the energy waste of maintaining excessive shards while ensuring sufficient capacity for data storage.
2Quantity of substance
If new shards are added to accommodate growing data, then data storage capacity is improved, but client device updates are required consuming significant computing resources
Solution Approach 1:
The patent introduces a shard mapping service as an intermediary between client devices and the actual data shards. This service maintains and manages the mapping between logical shard identifiers and physical shard locations, absorbing the complexity of shard management away from client devices. When shards are added or relocated, only the mapping service needs to be updated, not all client devices.
Solution Approach 2:
The system extracts the shard management complexity from client devices and centralizes it in the shard mapping service. By separating the mapping function from data access, the patent eliminates the need for widespread client updates when shard configurations change.
3Ease of manufacture
If data is moved between shards to optimize storage, then data organization is improved, but the mapping formula at client devices must be updated increasing system complexity
Solution Approach 1:
The shard mapping service acts as an intermediary that handles all mapping logic between logical shard identifiers and physical shard locations. When data is moved between shards for optimization, the mapping service is updated to reflect the new locations, while client devices continue to use simple logical identifiers without needing to understand or update complex mapping formulas.
Data Source
AI summary
Data sharding partitions data associated with an application into multiple shards in which each shard stores a subset of the data. A shard is further divided into multiple microshards in which each microshard contains a subset of the shard data. A set of microshards are assigned to a specified shard. The assignment of microshards can change, e.g., when new shards are added and microshards are migrated from one shard to another shard. A microshard map, which indicates a mapping between the microshards and shards, is updated automatically when the assignments change. After identifying the shard to which a specified microshard is assigned using the microshard map, a data access request from a client device is automatically forwarded to the appropriate shard. Because data can be accessed efficiently regardless of whether the number of shards changes, the data sharding facilitates provisioning of the shards on a need basis.