Distributed Cache Data Placement for Machine Learning Models

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing distributed caching systems lack a mechanism to place or store data based on the probability of it being concurrently requested or used by machine learning models, leading to increased training and inference times due to accessing data from multiple cache nodes or servers.

Innovation Solution

A method that determines the probability of feature data being concurrently used by a machine learning model and stores it on the same cache node or server, using algorithms like bin-packing and set cover to optimize data placement, thereby reducing the number of nodes accessed.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If data is distributed across multiple cache nodes or servers, then data storage capacity and system reliability are improved, but data access time increases

Engineering Contradiction:

Improvesystem reliabilityVSAvoiddata access time

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The system performs preliminary actions by predicting which data subsets will be concurrently requested by machine learning models before the requests occur. Based on these predictions, data is pre-positioned on the same cache node or server, so when requests arrive, all required data is already available locally, eliminating access delays while maintaining distributed storage for reliability

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

A data placement manager acts as an intermediary between the distributed cache system and machine learning models. This intermediary analyzes request patterns, determines probability of concurrent access, and makes intelligent routing decisions about where to place data, thereby resolving the conflict between distributed storage benefits and access speed requirements

Inventive Principle:

Principle #24Intermediary (Mediator)

2Productivity

If data is accessed from multiple cache nodes or servers, then data availability is improved, but training and inference time increases

Engineering Contradiction:

Improvedata availabilityVSAvoidtraining and inference time

Core Design Contradiction:

ProductivityVSLoss of time

Solution Approach 1:

The system predicts which data subsets will be needed by machine learning models and pre-positions them on the same cache node or server before the models need them. This preliminary action ensures high data availability for training and inference while avoiding the time penalty of accessing multiple nodes, as all required data is co-located

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system dynamically changes the placement parameters of data based on predicted usage patterns. By adjusting where data is stored based on probability of concurrent access, the system optimizes both availability and access speed for machine learning workloads without sacrificing distributed storage benefits

Inventive Principle:

Principle #35Parameter changes

Data Source

PatentUS12124929B2System, method, and computer program product for distributed cache data placement

Publication Date: 2024.10.22 VISA INTERNATIONAL SERVICE ASSOCIATION

AI summary

A method may include obtaining feature data associated with a set of feature inputs of a machine learning model, determining a probability that a subset of the feature data is concurrently used as the set of feature inputs for the machine learning model, and storing the subset of the feature data on a same cache node or server of a plurality of cache nodes or servers based on the probability.