Shared Memory Fabric for Distributed Data Partitioning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing distributed data systems face inefficiencies in processing large-scale iterative workloads due to immutable resilient distributed datasets (RDDs) that consume excessive memory and lead to inefficient cache eviction policies, resulting in increased memory consumption and processing delays.
Innovation Solution
Implementing an off-heap memory store with a shared memory fabric that allows direct memory access for attribute sharing between data partitions, enabling fast access and in-place updates of attributes using a high-speed interconnect, such as DRAM or non-volatile memory, to optimize memory usage and processing efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If immutable resilient distributed datasets (RDDs) are used to store data in distributed data systems, then data integrity and fault tolerance are improved, but memory consumption increases excessively and processing speed decreases
Solution Approach 1:
The patent segments the data storage system into multiple data partitions distributed across different nodes. Each partition can be independently managed and processed, allowing the system to maintain fault tolerance through distribution while reducing the memory burden on any single node. The segmentation enables parallel processing of different partitions simultaneously.
Solution Approach 2:
The patent extracts the immutability constraint from the data storage mechanism by allowing in-place updates of attributes within RDDs. This extraction of the immutability requirement enables the system to maintain data integrity and fault tolerance while significantly reducing memory consumption by updating data in place rather than creating new immutable instances.
2Stability of the object's composition
If immutable RDDs are used in iterative workloads, then data consistency is maintained, but processing delays increase due to inefficient cache eviction policies
Solution Approach 1:
The patent introduces dynamic update capabilities to traditionally immutable RDDs by enabling in-place attribute updates. This dynamic approach allows the system to maintain data consistency through controlled update mechanisms while significantly improving processing speed by avoiding the creation of new RDD instances for each update operation.
Solution Approach 2:
The patent enables continuous processing in iterative workloads by allowing updates to occur in-place within the same RDD structure. This continuity eliminates the need for repeated data serialization, deserialization, and redistribution that occurs with immutable RDDs, thereby maintaining data consistency while reducing processing delays.
3Loss of information
If multiple instances of cached RDDs are created to store updated data in each iteration, then data versioning is achieved, but memory consumption increases
Solution Approach 1:
The patent uses attribute tables as lightweight copies that reference the actual data partitions. Instead of creating full RDD instances for each version, the system creates compact attribute tables that store version information and metadata, significantly reducing memory consumption while maintaining the ability to track data versions across iterations.
Solution Approach 2:
The patent changes the parameter of data storage from full RDD instances to compact attribute representations. By storing only essential attribute information and version metadata rather than complete data copies, the system achieves data versioning capability while dramatically reducing memory consumption.
4Speed
If heap memory is used for RDD caching, then data access speed is improved, but cache eviction policies become inefficient and memory management becomes problematic
Solution Approach 1:
The patent extracts the caching mechanism from heap memory and implements it using attribute tables that reside in the distributed file system. This extraction allows the system to maintain fast data access through direct attribute table queries while eliminating the complex cache eviction policies and memory management overhead associated with heap-based RDD caching.
Data Source
AI summary
Examples herein involve processing data in a distributed data processing system using an off-heap memory store. An example involves allocating a shared memory region of a shared memory to store attributes corresponding to a first partition of a distributed data system, and updating, in the shared memory region, the attributes corresponding to updates to the local data from process iterations of the first partition, such that a second partition of the distributed data system has access to the updated attributes.