Shared Intermediate Caching for Multi-Target ML Inference

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing machine learning models require significant calculation resources and memory usage when handling multiple classification tasks due to the need for repeated inference processes and intermediate data caching for each piece of target information.

Innovation Solution

A technique that generates output information for multiple pieces of target information using a single inference process by caching and sharing intermediate data, reducing the need for repeated calculations and memory usage.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Measurement precision

If repeated inference processes are executed for each piece of target information, then output information can be generated accurately, but calculation resources and memory usage increase significantly

Engineering Contradiction:

Improveoutput information accuracyVSAvoidcalculation resource consumption

Core Design Contradiction:

Measurement precisionVSUse of energy by moving object

Solution Approach 1:

The patent performs preliminary calculation of intermediate data (key values and attention scores) for the query sequence before processing each target information piece. This pre-computed intermediate data is then reused across multiple inference processes, eliminating redundant calculations while maintaining output accuracy.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent creates a universal intermediate data structure (key value cache) from the query sequence that can be universally applied to multiple different target information pieces. This single intermediate data structure serves multiple inference functions, reducing overall calculation resource consumption.

Inventive Principle:

Principle #6Universality (Multi-functionality)

2Productivity

If intermediate data is cached for each inference process, then processing efficiency improves, but memory usage increases

Engineering Contradiction:

Improveprocessing efficiencyVSAvoidmemory usage

Core Design Contradiction:

ProductivityVSQuantity of substance

Solution Approach 1:

The patent extracts and caches only the essential intermediate data (key values and attention scores) from the full inference process, rather than caching all intermediate results. This selective extraction maintains processing efficiency while minimizing memory consumption.

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

The patent applies different caching strategies to different parts of the inference process. Specifically, it caches key values and attention scores which are most frequently reused, while not caching less critical intermediate data, optimizing the balance between processing efficiency and memory usage.

Inventive Principle:

Principle #3Local quality

Data Source

PatentUS20250342399A1Information processing system, information processing device, and information processing method

Publication Date: 2025.11.06 PREFERRED NETWORKS INC

AI summary

An information processing system includes at least one memory, and at least one processor. The at least one processor is configured to obtain information related to an output candidate and a plurality of pieces of target information, calculate first intermediate data by inputting the information related to the output candidate into a machine learning model, and generate output information for each of the plurality of pieces of the target information by executing a single inference process using the machine learning model for each of the plurality of pieces of the target information by using at least a portion of the first intermediate data.