Micro-heap allocators for pointer-based data structures in key-value stores
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Key-value stores face challenges in supporting arbitrary pointer-based dynamic data structures due to high serialization and deserialization overhead, limiting their ability to handle complex data structures like trees and graphs without converting them into opaque formats.
Innovation Solution
Implementing micro-heaps on a per-data structure basis, each with its own allocator, allows for direct storage and replication of pointer-based data structures in key-value stores without serialization, using micro-heap allocators for memory management and relocation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If pointer-based dynamic data structures are stored in key-value stores, then the ability to handle complex data structures like trees and graphs is improved, but serialization and deserialization overhead increases
Solution Approach 1:
The patent divides the storage system into multiple micro-heaps, each managing a specific portion of the data structure. Each micro-heap allocator independently manages memory for its associated data structure, enabling parallel serialization and deserialization operations. This segmentation reduces the overhead by allowing concurrent processing of different data structure components rather than serializing the entire structure sequentially.
Solution Approach 2:
The patent introduces micro-heap allocators as intermediary components between the key-value store and the pointer-based data structures. These allocators act as mediators that manage memory allocation and deallocation, handle pointer validation, and coordinate serialization/deserialization operations. By inserting this intermediary layer, the system can efficiently manage complex data structures without directly burdening the key-value store with serialization overhead.
2Productivity
If micro-heaps are implemented on a per-data structure basis, then memory management efficiency is improved, but system complexity increases
Solution Approach 1:
The patent implements a universal micro-heap allocator design that can serve multiple data structures with different requirements. Each micro-heap allocator follows the same interface and management principles, allowing the system to handle various data structure types (trees, graphs, linked lists) uniformly. This universality reduces system complexity by providing a standardized approach rather than requiring custom memory management logic for each data structure type.
Solution Approach 2:
The patent employs a hierarchical structure where micro-heaps are nested within the broader key-value store system. Each data structure has its own micro-heap, which in turn contains multiple memory blocks and allocations. This nested organization allows efficient localized memory management at the micro-heap level while maintaining overall system coherence through the parent key-value store, thereby improving memory management efficiency without proportionally increasing system complexity.
3Loss of energy
If pointer-based data structures are stored without serialization, then storage overhead is reduced, but reliability and resiliency may be compromised
Solution Approach 1:
The patent performs preliminary validation and metadata recording during the memory allocation phase. Before storing pointer-based data structures without full serialization, the micro-heap allocators pre-validate pointer integrity, record allocation metadata, and establish consistency checks. This preliminary action ensures that when data is replicated or recovered, the pointer-based structures maintain their integrity and reliability without requiring complete serialization/deserialization cycles, thus reducing storage overhead while preserving resiliency.
Data Source
AI summary
A computer-implemented method includes receiving data structures in memory space and creating micro-heaps on a per-data structure basis. Each data structure is associated with a micro-heap allocator. The method also includes storing the data structures in a key-value store. Values of the key-value store are associated with the data structures. A computer program product includes one or more computer readable storage media, and program instructions collectively stored on the one or more computer readable storage media. The program instructions include program instructions to perform the foregoing method. A system includes a processor and logic integrated with the processor, executable by the processor, or integrated with and executable by the processor. The logic is configured to perform the foregoing method.