Logical Tree Structure for Foundation Model Inferencing Server Selection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional AI models are trained on specifically tailored data, whereas foundation models are trained on broad, unlabeled data for various tasks, leading to high inferencing costs due to their large size, resulting in memory pressure issues during model inferencing.
Innovation Solution
A logical tree structure is used to identify an inferencing server that satisfies predetermined prerequisites, such as having a larger portion of the target model pre-loaded, to alleviate memory pressure by loading only the necessary components, exploiting commonality between downstream task models.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If foundation models are used for multiple downstream tasks, then versatility is improved, but memory pressure increases due to large model size
Solution Approach 1:
The patent segments the foundation model into multiple downstream task models, each representing a specific task or function. These segmented models are organized in a logical tree structure where common components are shared at higher levels and task-specific components are at lower levels. This segmentation allows the system to load only the necessary segments for a given task, reducing memory pressure while maintaining versatility across multiple tasks.
Solution Approach 2:
The patent implements a nested structure where downstream task models are organized in a hierarchical tree with shared common components at parent nodes and task-specific components at child nodes. This nesting allows smaller task-specific models to be contained within the larger foundation model structure, sharing common layers and parameters. When a specific task is executed, only the nested sub-tree corresponding to that task needs to be loaded, significantly reducing memory requirements while preserving the ability to perform multiple tasks.
2Reliability
If full foundation model is loaded for each inferencing request, then reliability is improved, but loss of time increases due to repeated loading
Solution Approach 1:
The patent performs preliminary actions by pre-processing and organizing the foundation model into a logical tree structure of downstream task models before inferencing requests arrive. Common components are identified and placed at shared nodes in advance. When inferencing requests arrive, the system can quickly identify and load only the pre-organized necessary components rather than loading the entire model each time, reducing inferencing time while maintaining reliability through the pre-validated model structure.
Solution Approach 2:
The patent applies local quality by allowing different parts of the model to be loaded selectively based on the specific task requirements. Instead of uniformly loading the entire foundation model, the system loads only the local sub-tree components necessary for the current task. This localized loading approach maintains the reliability of the full model structure while significantly reducing the time and memory required for each individual inferencing request.
3Manufacturing precision
If conventional AI models are trained on specifically tailored data, then manufacturing precision is improved, but adaptability deteriorates
Solution Approach 1:
The patent creates a universal foundation model that can perform multiple downstream tasks by organizing it as a logical tree of task-specific models. The root and intermediate nodes represent common components trained on broad, general data that apply to multiple tasks. Leaf nodes represent task-specific components. This universal structure allows a single model to adapt to various tasks by selectively activating appropriate sub-trees, combining the precision of task-specific training with the adaptability of a universal framework.
Data Source
AI summary
A computer-implemented method, according to one embodiment, includes determining a plurality of downstream task models of a foundation model, and arranging the downstream task models into a logical tree structure. Each node of the logical tree structure represents a sequence of layers of an associated one of the downstream task models. In response to a determination that a request for inferencing on a target model has resulted in a cache miss occurring, the logical tree structure is used to identify an inferencing server that satisfies at least a first predetermined prerequisite for fulfilling the inferencing request. The method further includes causing the identified inferencing server to fulfill the inferencing request. A computer program product, according to one embodiment, includes a computer readable storage medium having program instructions embodied therewith. The program instructions are readable and/or executable by a computer to cause the computer to perform the foregoing method.