Dynamic Round Robin Scheduler for LLM FSDP Memory Utilization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large language models (LLMs) for AI engines consume vast amounts of processing time, necessitating methods to reduce processing time and improve processor memory utilization.
Innovation Solution
A dynamic round robin scheduler optimizer is employed to improve processor memory utilization by receiving tokens related to user queries, placing relevant tokens in a ready queue and less relevant tokens in a waiting queue, and using a fully shared data parallel engine to retrieve data segments into cache memory proximate to each other.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data segments are retrieved and stored in cache memory without optimized placement, then memory access speed is maintained at baseline levels, but processor memory utilization efficiency deteriorates due to scattered data placement
Solution Approach 1:
The system performs preliminary actions by retrieving and placing data segments into cache memory before they are actually needed for processing. The round robin scheduler optimizer proactively manages data segment placement in advance, ensuring that frequently accessed data is already in cache when needed, thereby reducing processing time and improving overall productivity.
Solution Approach 2:
The system implements dynamic data segment placement in cache memory based on real-time processing needs. The round robin scheduler optimizer dynamically adjusts which data segments are placed in cache and their positions, rather than using static placement rules. This dynamic approach optimizes memory utilization efficiency and reduces access time for relevant data segments.
2Reliability
If all tokens are placed in the ready queue for processing, then processing completeness is maintained, but memory utilization efficiency deteriorates due to lack of prioritization
Solution Approach 1:
The system segments tokens into different priority categories: hot tokens (most relevant) and cold tokens (less relevant). This segmentation allows the round robin scheduler optimizer to manage memory resources efficiently by placing hot tokens in the ready queue for immediate processing while placing cold tokens in the waiting queue, thereby improving memory utilization efficiency without compromising processing completeness.
Solution Approach 2:
The system applies local quality differentiation by treating different tokens with different priority levels based on their relevance to the query. Hot tokens receive preferential treatment with immediate cache placement and processing, while cold tokens are handled with lower priority. This local quality approach ensures that critical data is processed efficiently while maintaining overall processing completeness.
3Quantity of substance
If data segments are placed far apart in cache memory, then cache memory capacity is preserved, but access efficiency deteriorates due to increased access time
Solution Approach 1:
The system merges related data segments by placing them in close proximity within cache memory. The round robin scheduler optimizer identifies data segments that are likely to be accessed together (related to the same query or processing task) and positions them adjacently in cache memory. This merging strategy improves data access speed by reducing the time required to fetch multiple related segments, while the system dynamically manages cache capacity through selective placement.
Data Source
AI summary
A method for using a round robin scheduler optimizer to improve processor memory utilization in an AI engine includes receiving, from a retrieval engine, a plurality of tokens identified as being related to key words from a user's query submitted to an artificial intelligence engine. The tokens are indexed to data segments stored by a large language model. The method includes placing, using a round robin scheduler optimizer, first tokens identified as most relevant to the user's query in a ready queue and placing, using the round robin scheduler optimizer, second tokens identified as less relevant to the user's query in a waiting queue, and using a fully shared data parallel engine to retrieve first data segments indexed to the first tokens into cache memory proximate to each other and to retrieve second data segments indexed to the second tokens into the cache memory proximate to the first data segments.