Dynamic Index Shard Management for Streaming Data Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing indexing technologies for streaming data storage systems face challenges in scalability and efficiency, leading to decreased data access performance and increased computational overhead as data volumes grow, with fixed shard numbers limiting processing capabilities and query efficiency.
Innovation Solution
A method is introduced to dynamically manage the index by creating additional shards and layers based on load conditions, where an active layer handles new data and frozen layers store less active data, allowing for expansion and contraction of shards to maintain optimal performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If existing indexing technology is used for streaming data storage, then data can be stored and accessed, but as data volume grows the system cannot effectively manage the increasing data, causing decreased data access performance and reduced analysis efficiency
Solution Approach 1:
The patent divides the index into multiple shard groups, where each shard group contains multiple shards that store indexes of different data objects. This segmentation allows the system to distribute and manage large volumes of streaming data across multiple independent units, preventing any single shard from becoming a bottleneck and maintaining data access performance as data volume grows.
Solution Approach 2:
The patent introduces a layer dimension to the traditional single-layer index structure by creating frozen layers and active layers. This dimensional transformation allows the system to organize shard groups across multiple layers, enabling more efficient management of large-scale data by separating hot data (active layer) from cold data (frozen layers) and optimizing access patterns accordingly.
2Quantity of substance
If the number of shards is increased to handle more data, then data capacity increases, but system complexity and computational overhead increase
Solution Approach 1:
The patent implements dynamic shard group management where the system can automatically create new shard groups and promote frozen layers to active layers based on data volume and access patterns. This dynamic adjustment allows the system to scale capacity as needed while maintaining optimal complexity by only creating new structures when necessary, rather than pre-configuring for maximum capacity.
Solution Approach 2:
The patent changes the state parameters of existing shard groups and layers rather than always creating new structures. By promoting frozen layers to active layers and adjusting shard group configurations based on load conditions, the system can increase effective capacity while reusing existing computational resources, thereby avoiding linear increases in system complexity.
3Productivity
If more shards are created to improve processing capacity, then data handling capability increases, but maintenance and management difficulty increases
Solution Approach 1:
The patent implements self-service mechanisms where the system automatically monitors data volume and access patterns, then autonomously creates new shard groups, promotes frozen layers to active layers, and manages shard configurations without manual intervention. This automation significantly reduces maintenance difficulty despite increased processing capacity, as the system self-adjusts to optimal configurations based on actual workload conditions.
Solution Approach 2:
The patent incorporates feedback loops that continuously monitor system performance metrics such as data access frequency, shard load balance, and storage capacity utilization. Based on this feedback, the system dynamically adjusts the number and configuration of shard groups and layers, ensuring optimal processing capacity while automatically managing the complexity of maintenance through data-driven decisions rather than manual configuration.
Data Source
AI summary
An active layer of the index includes a first shard group, and shards in the first shard group are configured to store indexes of a part of data objects in a streaming storage system. In response to determining that the state of the first shard group meets a predetermined expansion condition, a second shard group is created in the index, and shards in the second shard group are configured to store indexes of data objects that will enter the storage system. The storage system is managed based on the shards in an active layer (where the second shard group is located) and frozen layers (where the second shard group) in the index. The number of shards in the storage system can be dynamically set to process streaming data at a relatively high speed, and it is suitable for processing streaming data that continuously enters the storage system.