One-Sided Key-Value Store Lookup via Direct Client Memory Writes
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing partitioned key-value stores face challenges in efficient key look-ups, particularly for range-knowledgeable clients, due to the large amount of data that needs to be transferred and processed, which hinders performance in high-performance computing environments.
Innovation Solution
Implementing one-sided communications using MPI for secondary global key look-ups, allowing range-knowledgeable clients to access metadata and instruct servers to store values directly into client memory, thereby reducing the need for extensive data transfer and improving lookup efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If traditional two-sided MPI communications are used for key look-ups in partitioned key-value stores, then data transfer and processing can be performed, but synchronization overhead increases and lookup speed decreases
Solution Approach 1:
The patent inverts the traditional two-sided communication model by implementing one-sided communication where the client can directly access and store data in server memory without waiting for server confirmation. This allows the client to proceed with subsequent operations immediately, eliminating the synchronization wait time that normally occurs in two-sided communication protocols.
Solution Approach 2:
The patent introduces a memory address mapping mechanism as an intermediary that enables direct client-to-server memory access. By maintaining mappings between client memory addresses and server memory locations, the system allows data to be transferred and stored directly without requiring traditional request-response synchronization, thus reducing overhead while maintaining data integrity.
2Loss of information
If extensive data transfer is performed for secondary global key look-ups, then complete key-value pairs can be retrieved, but network bandwidth consumption increases and performance decreases
Solution Approach 1:
The patent extracts only the necessary data (the specific key-value pair corresponding to the requested secondary global key) from the distributed storage system, rather than transferring extensive amounts of data. By using the metadata to directly locate and retrieve only the required information through one-sided communication, the system minimizes network bandwidth consumption while ensuring complete key-value retrieval.
Solution Approach 2:
The patent performs preliminary actions by maintaining pre-computed metadata that maps secondary global keys to primary keys and their locations in the distributed system. This preliminary organization of data location information allows clients to directly access the required data without performing extensive data transfer or search operations, thereby reducing network bandwidth consumption while ensuring complete information retrieval.
Data Source
AI summary
Partitioned key-value stores with one-sided communications are provided for secondary global key lookup by range-knowledgeable clients. A secondary global lookups is performed in a partitioned key-value store by receiving a request from an application for a secondary global key; accessing metadata to identify a first server storing the secondary global key; instructing the identified first server to store a first value corresponding to the requested secondary global key into a memory address of the client; and returning program control to the application, wherein the first server identifies a primary key corresponding to the requested secondary global key and instructs a second server storing the corresponding primary key to store a second value corresponding to the primary key into the memory address of the client. The second server stores the second value corresponding to the primary key into the first memory address of the client.