Edge Data Store Partitioning for Scalable IoT Connectivity
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed computing systems face inefficiencies in storing and managing state data for network-enabled devices, particularly due to storage capacity limitations, security concerns, and the need for low-latency access across multiple servers, which affects performance and scalability.
Innovation Solution
A distributed data store system where each node has its own partition with followers for replication, allowing data ownership to move and be transferred securely across nodes for load balancing and fault resilience, with security checks and health evaluations to mitigate threats.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If state data is stored on the network enabled device itself, then storage capacity and security attributes are充分利用, but it becomes inefficient to transfer large volumes of data to server nodes on each connection and the device lacks sufficient storage capacity or security attributes
Solution Approach 1:
The system segments state data storage across multiple server nodes in a distributed manner. Each node stores portions of state data for different devices or device groups, eliminating the bottleneck of centralized storage and allowing parallel access during connections. This segmentation enables efficient data retrieval without requiring the entire state data to be transferred from a single location on each connection.
Solution Approach 2:
The patent introduces server nodes as intermediaries between network enabled devices and the centralized data store. These intermediary nodes cache and manage state data locally, reducing the frequency and volume of data transfers between devices and the central system. The intermediaries handle data retrieval and transfer operations, improving connection efficiency while the central system maintains overall data integrity.
2Reliability
If state data is stored in a dedicated storage subsystem, then data access is centralized, but the system performs poorly and does not scale sufficiently
Solution Approach 1:
The dedicated storage subsystem is segmented into multiple distributed storage nodes. Each node independently stores and manages portions of state data, enabling parallel data access operations. This segmentation eliminates the single-point-bottleneck performance limitation while maintaining data access reliability through the distributed architecture's redundancy and fault tolerance capabilities.
Solution Approach 2:
The system transitions from a single-dimensional centralized storage model to a multi-dimensional distributed storage architecture. Data is organized and accessed across multiple dimensions including spatial distribution across nodes, temporal caching strategies, and hierarchical storage levels. This dimensional transformation enables simultaneous access from multiple points, dramatically improving system performance while maintaining reliability.
3Adaptability or versatility
If all servers can access a given device's state data, then any server can serve the device regardless of connection reasons, but updates can collide and require significant time to reach consistency without segmentation and locks
Solution Approach 1:
State data is segmented into partitions, with each partition assigned to a specific owning server node. This segmentation allows multiple servers to access different partitions simultaneously without collision, maintaining server flexibility. When a server needs to access or update data, it operates on the specific partition it owns or has been granted access to, eliminating update collisions and reducing consistency time.
Solution Approach 2:
The system implements feedback mechanisms including version control, timestamps, and consistency protocols that track data updates across the distributed system. When multiple servers access or modify state data, the feedback system detects conflicts and coordinates resolution, ensuring eventual consistency without requiring lengthy synchronization periods. This feedback-driven approach maintains both server flexibility and data consistency.
Data Source
AI summary
A distributed computing system provides a distributed data store for network enabled devices at the edge. The distributed database is partitioned such that each node in the system has its own partition and some number of followers that replicate the data in the partition. The data in the partition is typically used in providing services to network enabled devices from the edge. The set of data for a particular network enabled device is owned by the node to which the network enabled device connects. Ownership of the data (and the data itself) may move around the distributed computing system to different nodes, e.g., for load balancing, fault-resilience, and/or due to device movement. Security/health checks are enforced at the edge as part of a process of transferring data ownership, thereby providing a mechanism to mitigate compromised or malfunctioning network enabled devices.