Memory Slice Data Structure for RDMA Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Document databases face challenges in efficiently storing and retrieving unstructured data due to the lack of predefined schema, which complicates data manipulation and querying, and existing solutions do not effectively support remote direct memory access (RDMA) for distributed data access.
Innovation Solution
A binary data format for documents is introduced, where data is stored in self-contained memory slices that can be persisted as checkpoints and updated with log entries, enabling efficient storage and retrieval across diverse memory and storage media, including RDMA for remote access.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If data is stored in separate tables with predefined schema in relational databases, then data manipulation and querying become complex, but data structure and relationships are well-defined
Solution Approach 1:
The patent segments data into self-contained memory slices that can be independently stored and retrieved. Each slice represents a complete data unit with its own structure, eliminating the need for complex table joins and relationships. This segmentation allows simple key-value access patterns while maintaining data integrity without requiring predefined schemas or complex relational structures.
2Ease of manufacture
If document databases store all information for an object in a single document without predefined structure, then mapping objects is simple, but efficient storage and retrieval of unstructured data becomes challenging
Solution Approach 1:
The patent changes the parameter of data organization from flat single-document storage to hierarchical memory slice structures. Each memory slice contains optimized internal structures including segment arrays and offset tables that enable efficient access patterns. This transformation maintains the simplicity of storing unstructured data while dramatically improving retrieval efficiency through structured access paths and reduced I/O operations.
Solution Approach 2:
The patent introduces a new dimensional organization of data by creating multi-level memory slice structures with segments and sub-segments. Instead of single-dimensional flat storage, data is organized across multiple dimensions including slice level, segment level, and offset level. This dimensional expansion enables efficient indexing and access patterns that work well with unstructured data while maintaining high retrieval performance.
3Speed
If memory slices contain absolute pointers to memory addresses for efficient access, then data retrieval is fast, but the data structure cannot be easily persisted to disk or accessed remotely
Solution Approach 1:
The patent introduces relative offsets as an intermediary between absolute memory addresses and data locations. Instead of storing absolute pointers that are valid only in specific memory regions, the system uses relative offsets from known base addresses. This intermediary allows the same data structure to be accessed from different base addresses, enabling both fast local access and portable persistence to disk or remote memory via RDMA without requiring address translation or restructuring.
Data Source
AI summary
A data storage and retrieval system for a computer memory including a memory slice formed of segments and adapted to contain one or more documents and a checkpoint adapted to persist the memory slice. The checkpoint includes a document vector containing a document pointer corresponding to a document. The document pointer including a segment identifier identifying a logical segment of the memory slice and an offset value defining a relative memory location of the first document within the identified segment. There are checkpoint memory blocks, each storing a copy of a corresponding segment of the memory slice. The segment identifier of the document pointer identifies a checkpoint memory block and the offset value of the document pointer defines a relative location of the document within the checkpoint memory block.