Vision-Language Prompt Caching for Faster Consecutive Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Vision-language models (VLMs) face challenges in efficiently processing consecutive inputs due to repetitive operations on redundant information, leading to decreased inference speed and difficulty in implementation on resource-constrained devices.
Innovation Solution
A method and system that caches information associated with an input prompt during the generation of a first inference result, allowing subsequent inference results to be generated using this cached information, thereby reducing redundant operations and enhancing inference speed.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If the VLM processes each consecutive input independently without caching, then the inference results are accurate and complete, but the inference speed decreases due to repetitive operations on redundant information
Solution Approach 1:
The patent applies preliminary action by caching the input prompt and attention information from the first input before processing subsequent inputs. This pre-computed information is stored and reused during processing of consecutive inputs, eliminating the need to re-process redundant information and significantly improving inference speed while maintaining accuracy
2Reliability
If the VLM performs full inference operations on each consecutive input, then the inference accuracy is maintained, but the computational resources required increase making deployment on edge devices difficult
Solution Approach 1:
The patent extracts and caches only the essential information (input prompt and attention information) from the full inference process. By separating and reusing these critical components across consecutive inputs, the system maintains inference accuracy while dramatically reducing the computational resources needed for each subsequent input, enabling deployment on resource-constrained edge devices
3Productivity
If the VLM processes consecutive inputs by re-computing all operations, then the results are consistent and reliable, but unnecessary repetitive operations reduce processing efficiency
Solution Approach 1:
The patent creates copies of the cached input prompt and attention information from the first input and uses these copies for processing subsequent inputs. This copying approach allows the system to avoid re-computing identical operations while maintaining result consistency, thereby reducing computational energy consumption and improving processing efficiency
Data Source
AI summary
Provided is an inference method using a vision-language model (VLM). The VLM is pretrained to sequentially generate inference results for consecutive inputs according to an input prompt, and the inference method includes caching information associated with the input prompt acquired during an operation for generating a first inference result for a first input among the consecutive inputs to the VLM, maintaining the cached information after the first inference result is generated; and generating a second inference result for a second input following the first input among the consecutive inputs to the VLM, based on the cached information.