Inter-Process ML Feature Serving from Mapped Memory for Low-Latency Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning models that rely on multiple data features aggregated across multiple dimensions in a sliding time window are resource-intensive and time-consuming, with inefficient access mechanisms like Unix sockets leading to high latency and resource contention.
Innovation Solution
A server system utilizing a memory-mapped file as an Inter-Process Communication mechanism, employing a wait-free synchronization technique and zero-copy deserialization to serve machine learning features, allowing concurrent access and minimal resource usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If Unix sockets are used for accessing ML features from file, then data can be served to inference models, but latency is high and resource contention occurs
Solution Approach 1:
The patent creates a memory-mapped copy of the ML features file in shared memory, allowing direct memory access instead of file I/O through Unix sockets. The feature manager service maps the features file into shared memory and serves features directly from this mapped memory to multiple inference model services simultaneously, eliminating socket communication overhead and reducing latency.
Solution Approach 2:
The patent introduces a feature manager service as an intermediary that manages shared memory allocation and feature serving. This mediator coordinates access between the ML features storage and multiple inference models, using wait-free synchronization techniques to allow concurrent access without blocking, thereby reducing resource contention while maintaining high-speed access.
2Productivity
If multiple inference models access ML features concurrently, then model predictions can be generated, but resource contention increases
Solution Approach 1:
The patent segments the feature data into distinct memory regions within shared memory, with each inference model service having dedicated access to specific feature sets. The feature manager service divides the ML features file into multiple segments in shared memory, allowing concurrent access by multiple inference models without contention, as each model reads from its own memory segment simultaneously.
Solution Approach 2:
The patent changes the access parameter from sequential file I/O through sockets to parallel direct memory access. By mapping the features file into shared memory and using memory-mapped access semantics, multiple inference models can simultaneously read feature data without the serialization overhead of Unix sockets, dramatically improving throughput while reducing CPU and memory resource contention.
3Reliability
If ML features are stored in periodically updated files, then feature data can be maintained, but access efficiency is reduced
Solution Approach 1:
The patent performs preliminary action by pre-loading and mapping the entire ML features file into shared memory before inference operations begin. The feature manager service maps the features file into shared memory at system initialization or when features are updated, preparing the data structure in advance. This eliminates the need for repeated file access operations during inference, maintaining data accuracy while simplifying access mechanisms to simple memory reads.
Data Source
AI summary
Inter-process serving of machine learning features from mapped memory for machine learning models is described. ML features are populated in a data structure that is serialized. State data is stored that indicates that reader process(es) are to read from a first memory mapped data file and not a second memory mapped data file. The serialized bytes are stored in the second memory mapped data file and the state data is updated to indicate that the reader process(es) are to read from the second memory mapped data file. A request is received and parsed to prepare keys from attributes of the request. Based on the state data, the serialized bytes are read from the second memory mapped data file that correspond to the keys. The serialized bytes are deserialized and copied to a data structure available to an inference algorithm.