Intra-shard Virtual Shards for Parallel Data Stream Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Data stream processing systems face lag issues when records are input at a faster rate than they can process, leading to systematic or transient delays due to uneven data distribution across shards, with some shards experiencing more complex processing than others.
Innovation Solution
Intra-shard parallelization using virtual shards, where a shard's processing load is distributed across multiple smaller virtual shards, allowing parallel processing and dynamic adjustment of the parallelization factor based on observed processing conditions to manage delays and resource usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a single shard processes records sequentially to maintain order, then processing reliability is improved, but processing speed deteriorates when input rate exceeds processing capacity
Solution Approach 1:
The patent divides a single shard into multiple virtual shards (e.g., shard_0, shard_1, shard_2) that can process records in parallel. Each virtual shard maintains its own processing queue and can handle records independently while preserving the original shard's ordering guarantees through virtualization layer coordination.
Solution Approach 2:
The patent implements a nested shard structure where virtual shards are contained within a parent shard. The virtualization layer encapsulates the parallel processing logic within the shard boundary, allowing multiple levels of processing hierarchy to coexist and maintain ordering relationships.
2Productivity
If the parallelization factor is increased to reduce processing delays, then productivity is improved, but device complexity increases
Solution Approach 1:
The patent implements dynamic parallelization where the number of virtual shards can be adjusted based on processing conditions. The system can scale the parallelization factor up when backlog increases and down when processing capacity is sufficient, allowing flexible adaptation to varying workloads without fixed complex configurations.
Solution Approach 2:
The patent changes the parallelization parameter dynamically by modifying the number of active virtual shards. This parameter adjustment allows the system to optimize between processing speed and resource usage, adapting to different processing conditions without requiring complex architectural changes.
3Productivity
If more virtual shards are created to distribute processing load, then processing speed is improved, but loss of energy increases due to additional resource consumption
Solution Approach 1:
The patent dynamically adjusts the number of active virtual shards based on processing backlog and workload conditions. When the system is under heavy load, more virtual shards are activated to increase throughput. When load decreases, virtual shards are deactivated or merged, reducing energy consumption and avoiding waste of computing resources.
Solution Approach 2:
The patent changes the active parallelization parameter by adjusting how many virtual shards are actively processing records. This dynamic parameter modification allows the system to match computational resource consumption with actual processing needs, optimizing the balance between throughput and energy efficiency.
Data Source
AI summary
A data stream may include a plurality of records that are ordered, and the plurality of records may be assigned to a processing shard. A first set of virtual shards may be formed, the first set of virtual shards having a first quantity of virtual shards that perform parallel processing operations on behalf of the processing shard. First records of the plurality of records may be processed using the first set of virtual shards. The first quantity of virtual shards may be modified, based at least in part on an observed record age, to a second quantity of virtual shards that perform parallel processing operations on behalf of the processing shard. A second set of virtual shards may be formed having the second quantity of virtual shards. Second records of the plurality of records may be processed using the second set of virtual shards.