Zone-Aware Stream Data Partitioning for Heterogeneous Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing streaming storage systems face challenges in managing and processing streaming data across multiple zones in heterogeneous infrastructures, leading to issues such as high latency, inconsistent write consistency, and complex data replication processes.
Innovation Solution
A zone-aware workload partitioning approach is introduced, which allows for dynamic resource allocation across zones, utilizes automatic integrated storage tiering, and implements read-only segment containers (SCs) for active-passive data replication, enabling efficient data management and processing across edge nodes and cloud servers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is replicated across multiple zones in heterogeneous infrastructures, then data availability and consistency are improved, but system complexity and inter-cluster traffic increase
Solution Approach 1:
The system segments data into segment containers (SCs) that can be independently managed and replicated across zones. Each SC is a self-contained unit that can be replicated to multiple zones, simplifying the replication process by treating data as discrete, manageable segments rather than continuous streams.
Solution Approach 2:
The patent introduces zone-awareness as an intermediary layer between data production and consumption. This zone-awareness mechanism mediates data replication by automatically routing segment containers to appropriate zones based on producer/consumer location, reducing the complexity of manual replication management while ensuring consistency.
2Reliability
If data is accessed from remote zones, then data availability is improved, but access latency increases
Solution Approach 1:
The system implements local quality by creating zone-specific segment containers that are locally cached in each zone. Consumers can access data from their local zone's SC rather than remotely accessing the primary zone, significantly reducing access latency while maintaining data availability across zones.
Solution Approach 2:
Segment containers are proactively replicated to multiple zones before they are needed. This preliminary replication action ensures that data is already present in local zones when consumers need it, eliminating the need for remote access and reducing latency.
3Adaptability or versatility
If traditional streaming storage systems are used in heterogeneous environments, then system compatibility is improved, but performance and efficiency deteriorate
Solution Approach 1:
The patent creates a universal segment container format that can operate across heterogeneous infrastructures (edge devices, fog nodes, cloud servers). This universal SC design maintains compatibility with traditional streaming storage systems while enabling optimized data processing through zone-awareness, achieving both adaptability and improved productivity.
Solution Approach 2:
The system dynamically adapts to heterogeneous environments by adjusting data placement and replication strategies based on zone characteristics and resource availability. This dynamic behavior allows the system to optimize performance for each specific infrastructure configuration while maintaining universal compatibility.
Data Source
AI summary
A first user-initiated data write request including stream data is received from an edge device executing in a first zone. The stream data is partitioned into at least one segment and an event included in the stream data is written into the at least one segment based on an event's routing key. A segment container of a segment store hosts the at least one segment. The stream data is transmitted to a tier-1 storage of the first zone in order to provide a low-latency access to the stream data for the first user. Thereafter, the stream data is moved to a tier-2 storage of a second zone for permanent storage of the stream data. When requested by a second user operating in the second zone, a read-only version of the segment container retrieves the stream data from the tier-2 storage.