Distributed Data Management System Using Digital Hash Map
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data management systems in multicomputer environments face challenges in achieving both scalability and performance concurrently, often requiring transactional scope management and distributed locking, which lead to performance bottlenecks and limitations in concurrent data access.
Innovation Solution
A distributed data management system using a digital hash map data structure that enables concurrent, non-blocking access by storing equivalent representations of complex data types as compressed byte arrays, reducing resource storage footprint and speeding data transfer, and employing caching and synchronization to minimize node-to-node data transfers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If distributed DBMS solutions use transaction management and distributed locking to guarantee data correctness, then data consistency is improved, but concurrent data access performance deteriorates due to expensive transaction overhead and locking
Solution Approach 1:
The patent segments the data structure into a distributed hash map where each node maintains only a portion of the data (buckets), eliminating the need for global locking. Each node independently manages its local data segments, allowing concurrent access without transaction overhead while maintaining consistency through the hash map structure.
Solution Approach 2:
The patent uses a distributed copy of the hash map structure across multiple nodes, where each node holds a subset of data buckets. This allows multiple nodes to simultaneously access different data without blocking each other, replacing the single-source-of-truth model that requires locking.
2Adaptability or versatility
If prior SDDS solutions load balance data uniformly across multiple computer nodes, then scalability is improved, but access performance deteriorates because concurrent requests block access to data on a particular node
Solution Approach 1:
The patent implements dynamic scalability through the distributed hash map where nodes can dynamically join or leave the system without service interruption. The system automatically redistributes data buckets among nodes based on their availability and capacity, maintaining load balance while enabling ad-hoc node joining without blocking concurrent access.
Solution Approach 2:
The patent pre-configures the distributed hash map structure with multiple buckets that are distributed across nodes before data is inserted. This preliminary distribution ensures that data is already spread across multiple nodes, allowing concurrent requests to access different nodes simultaneously without blocking.
3Ease of operation
If SDDS solutions virtualize data access by resolving client requests from server nodes, then data accessibility is improved, but network bandwidth consumption increases due to repeated transfer of large objects
Solution Approach 1:
The patent stores data in compressed byte array format in the distributed hash map, creating efficient copies that can be quickly transferred and deserialized. This reduces the size of objects being transferred across the network compared to traditional object graph serialization, minimizing bandwidth consumption while maintaining accessibility.
Solution Approach 2:
The patent changes the data representation parameter from full object graphs to compressed byte arrays. This parameter change reduces the amount of data that needs to be transferred across the network while maintaining the ability to deserialize and access the original data structures, thereby reducing bandwidth consumption.
4Adaptability or versatility
If nodes join the distributed system in an ad-hoc fashion without service interruption, then system flexibility is improved, but system complexity increases due to dynamic reconfiguration requirements
Solution Approach 1:
The patent implements a dynamic distributed hash map where nodes can join or leave the system at any time without service interruption. The system automatically detects node availability, assigns data buckets to new nodes, and redistributes data as needed, handling ad-hoc joining through a standardized dynamic process.
Solution Approach 2:
The patent implements self-service through automatic data redistribution when nodes join or leave. The system automatically detects changes in node availability, determines which buckets need to be reassigned, and performs the data migration without manual intervention, reducing the operational complexity of ad-hoc joining.
Data Source
AI summary
A distributed data management system has multiple virtual machine nodes operating on multiple computers that are in communication with each other over a computer network. Each virtual machine node includes at least one data store or “bucket” for receiving data. A digital hash map data structure is stored in a computer readable medium of at least one of the multiple computers to configure the multiple virtual machine nodes and buckets to provide concurrent, non-blocking access to data in the buckets, the digital hash map data structure including a mapping between the virtual machine nodes and the buckets. The distributed data management system employing dynamic scalability in which one or more buckets from a virtual machine node reaching a memory capacity threshold are transferred to another virtual machine node that is below its memory capacity threshold.