Partition Mapping for Distributed Data Processing Integration
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current stateful streaming data processing engines for IoT4A face challenges in integrating multiple distributed data processing systems with different data partitioning and routing mechanisms, resource sharing policies, and lifecycles into a single process, leading to inconsistent performance and inefficient data access.
Innovation Solution
The solution involves dividing data into the same number of partitions using a shared hash function across systems and establishing a one-to-one mapping between partitions, allowing for dynamic mapping and consistent data processing between systems with and without states, while also managing resource sharing and lifecycles.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If multiple distributed data processing systems with different data partitioning and routing mechanisms are integrated, then data processing versatility is improved, but system complexity increases
Solution Approach 1:
The patent introduces a unified partition mapping mechanism as an intermediary layer between different distributed data processing systems. This mapping mechanism translates partition identifiers and routing information from one system to another, enabling integration without requiring the systems to share common partitioning mechanisms. The intermediary handles the complexity of translation and routing, allowing versatile data processing while managing system complexity through abstraction.
2Stability of the object's composition
If standalone servers are used for each distributed processing system, then system independence is improved, but data access performance deteriorates
Solution Approach 1:
The patent merges multiple standalone distributed data processing systems into a unified process that maintains their independence through virtualization. By combining system-level resources and sharing common infrastructure while preserving individual system boundaries through virtual partitions, the patent achieves both system independence and improved data access performance through shared memory and coordinated routing.
3Adaptability or versatility
If different routing mechanisms are used in different systems, then system flexibility is improved, but routing consistency deteriorates
Solution Approach 1:
The patent applies parameter transformation to convert routing parameters from one system's convention to another system's convention. By changing the representation of partition identifiers, routing keys, and data location information during the mapping process, the system maintains flexibility in supporting different routing mechanisms while ensuring consistent and reliable routing across integrated systems through standardized parameter translation.
Data Source
AI summary
A method is provided for consistent data processing by first and second distributed processing systems having different data partitioning and routing mechanisms such that the first system is without states and the second system is with states. The method includes dividing data in each system into a same number of partitions based on a same key and a same hash function. The method includes mapping partitions between the systems in a one-to-one mapping. The mapping step includes calculating a partition ID based on the hash function and a total number of partitions, and dynamically mapping a partition in the first system to a partition in the second system, responsive to the partition in the first system being unmapped to the partition in the second system.