Method for implementing inference acceleration for machine-learning and large-language models based on encapsulation engine
By combining kvcache based on the encapsulation engine and the Triton inference server, the problem of high computational load in the inference process of large language models is solved, achieving efficient inference acceleration, meeting the needs of real-time applications, reducing hardware resource consumption, supporting online model updates and version management, and improving user experience and system performance.
Patent Information
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- WHALE CLOUD TECH CO LTD
- Filing Date
- 2025-06-23
- Publication Date
- 2026-06-04
AI Technical Summary
The reasoning process of large language models is computationally intensive, resulting in a high dependence on hardware resources. This makes it difficult to meet the speed and accuracy requirements in real-time application scenarios. Existing reasoning acceleration methods, such as model quantization, distillation, and pruning, have limitations and cannot meet the requirements of real-time performance and efficiency.
We adopt an engine-based approach, utilizing the kvcache caching mechanism and the Triton inference server. Through cache lookup and dynamic model management, we optimize the use of computing resources, realize online model updates and version management, and improve inference efficiency by combining dynamic batch processing and parallel execution.
Significantly accelerates inference response time, reduces hardware resource requirements, maintains high accuracy, supports rapidly changing application environment iterations, reduces operating costs, and improves user experience and system performance.
Smart Images
Figure CN2025102726_04062026_PF_FP_ABST
Abstract
Description
Methods for accelerating inference in machine learning and large language models based on encapsulation engines Technical Field
[0001] This invention relates to the fields of artificial intelligence and machine learning technology, and specifically to a method for accelerating inference of machine learning and large language models based on an encapsulation engine. Background Technology
[0002] With the rapid development of artificial intelligence technology, especially the increasingly widespread application of large language models (LLMs) across various industries, sectors such as finance, healthcare, education, and government have ushered in intelligent transformation. In the financial sector, large models can accurately assess credit risk and quickly analyze insurance terms, thereby optimizing customer service. In the healthcare industry, they are used for case analysis and assisted diagnosis, significantly improving doctors' efficiency and diagnostic accuracy. In education, large models can provide students with personalized learning guidance, recommend suitable learning resources, and promote the development of personalized teaching. In government services, large models improve the efficiency of information retrieval and business processing, enhancing user experience. Furthermore, large models also play a crucial role in technological scenarios such as intelligent customer service, computer vision, speech recognition, and recommendation systems.
[0003] However, the inference process of large models often requires massive computational resources, typically involving hundreds of millions of parameters and complex network structures. This makes the computational load of the inference process enormous, with a very high dependence on hardware resources such as CPUs and GPUs. Especially in application scenarios such as real-time data analysis and online question answering, inference speed often becomes a critical requirement. These scenarios need to complete complex inference calculations within a limited time, but traditional inference methods often struggle to balance speed and accuracy, leading to increased inference latency, impacting user experience, and limiting the widespread use of large models in practical applications.
[0004] Currently, while the technical community has proposed inference acceleration solutions such as model quantization, distillation, and pruning, each has its limitations. Model quantization reduces model size by converting floating-point parameters to low-bit integers, but may result in some loss of precision. Model distillation reduces computational requirements by training small models to mimic the output of large models, but it typically struggles to match the performance of large models. Model pruning improves computational efficiency by removing redundant neurons or connections, but this process often involves performance degradation and requires additional fine-tuning. While these existing techniques alleviate the resource pressure on inference to some extent, they are still insufficient to address the real-time and efficiency requirements of large models. Therefore, a novel inference acceleration method is urgently needed to achieve more efficient computation and meet the ever-growing application demands. Summary of the Invention
[0005] To overcome the shortcomings of existing technologies, this invention proposes a method for accelerating inference of machine learning and large language models based on an encapsulation engine. This method can support online updates and version management of models, facilitating iteration and optimization in rapidly changing application environments.
[0006] To achieve the above objectives, this invention provides a method for accelerating inference in machine learning and large language models based on an encapsulation engine, comprising the following steps:
[0007] Step 1: Implement the performInferenceWithCacheCheck method in the inference engine, which accepts the boolean parameter enable_cache to control whether to enable the kvcache acceleration function, ensuring that the system can flexibly adapt to different inference needs.
[0008] Step 2: Based on the input enable_cache value, execute the conditional judgment logic to determine whether to perform a cache query; if true, perform a kvcache query to improve inference efficiency.
[0009] Step 3: Create a Triton model repository in the specified file path, ensuring that the file structure is clear and meets the requirements, and create an independent folder for each model for easy access and management later.
[0010] Step 4: In the config.pbtxt file, precisely define the model version information, input and output tensor names, data types, and dimensions to ensure data compatibility and stability during the inference process.
[0011] Step 5: Start the Triton Inference Server using the command-line tool and specify the model repository path so that the server can load and prepare to process subsequent inference requests.
[0012] Furthermore, the specific method for determining user configuration in step 2 is as follows:
[0013] Step 2.1: In the performInferenceWithCacheCheck method, the boolean value of the enable_cache parameter is determined by a conditional statement.
[0014] Step 2.2: If enable_cache is true, call the kvcache query method to check if the corresponding inference result is already in the cache.
[0015] Step 2.3: If the cache is hit, return the result in the cache directly; if the cache is not hit, record the current input parameters and continue with the subsequent inference calculation.
[0016] Furthermore, the specific process of configuring and creating the Triton model repository in step 3 is as follows:
[0017] Step 3.1: Specify the path to the model repository in the file system, such as / models / , and ensure that the path exists and is writable.
[0018] Step 3.2: Create a separate folder for each model to be deployed under this path, named / models / model_name / , for easy management later.
[0019] Step 3.3: In each model folder, prepare the model file and its configuration file config.pbtxt, ensuring that the file format conforms to the Triton specification.
[0020] Furthermore, the detailed configuration of the model file in step 4 is as follows:
[0021] Step 4.1: In the config.pbtxt file, define the basic information of the model, including the platform (e.g., TensorRT), max_batch_size, and the names and data types of the input and output.
[0022] Step 4.2: Specify the data type (e.g., FP32, INT64), shape (e.g., [1,3,224,224]), and other relevant parameters for each input and output tensor to ensure that the model can correctly parse the data.
[0023] Step 4.3: Ensure that the model version information is clear, and set a version number for each model as needed so that a specific version can be specified when calling it later.
[0024] Furthermore, the specific steps for starting the Triton inference server in step 5 are as follows:
[0025] Step 5.1: Use the command-line tool to execute the Triton Inference Server startup command, as shown in the example below:
[0026] Step 5.2: During startup, monitor terminal output to confirm that all models are loaded correctly without error messages, ensuring the server is ready.
[0027] Step 5.3: Verify that the inference server can receive and respond to inference requests by accessing the specified API interface to ensure that the system is running normally.
[0028] Furthermore, the specific method for implementing kvcache queries in step 2 is as follows:
[0029] Step 6.1: Store the input parameters and corresponding output results for each inference in kvcache to form a key-value pair mapping.
[0030] Step 6.2: When making a new inference request, first construct a query key based on the input parameters and check if the key exists in kvcache.
[0031] Step 6.3: If the query result exists, return the corresponding output result directly; if the query result does not exist, record the request and prepare to perform inference calculation.
[0032] Furthermore, in step 3, the dynamic update function of the Triton model is configured as follows:
[0033] Step 7.1: In the Triton model repository, create a monitoring script to periodically check for new model versions and updates.
[0034] Step 7.2: When a new model version is discovered, automatically update the config.pbtxt file and replace the old version in the model repository.
[0035] Step 7.3: After the update, restart the Triton inference server to load the new version of the model and ensure that the new model can be used for inference immediately.
[0036] Furthermore, the specific steps for achieving data compatibility and stability during the inference process in step 4 are as follows:
[0037] Step 8.1: When an inference request arrives, record the source IP address and request time for traffic analysis.
[0038] Step 8.2: Based on real-time traffic conditions, distribute inference requests to different backend instances to ensure load balancing for each instance.
[0039] Step 8.3: When an instance is overloaded, dynamically adjust the request routing strategy to prioritize distributing requests to instances with lower current load.
[0040] Furthermore, the processing of subsequent inference requests in step 5 is as follows:
[0041] Step 9.1: Upon receiving an inference request, immediately return a confirmation of receipt to inform the client that inference is in progress.
[0042] Step 9.2: Place the inference task into the task queue and use the thread pool to execute the inference calculation asynchronously to improve the system's responsiveness.
[0043] Step 9.3: After the reasoning is complete, send the result to the client through the callback mechanism, or make it available for the client to query through the specified interface.
[0044] Furthermore, step 5 also includes implementing persistent storage of the inference results, as detailed below:
[0045] Step 10.1: After the inference is completed, record the input parameters and output results of each inference in the database to form a complete inference log.
[0046] Step 10.2: Assign a unique identifier to each record for subsequent querying and analysis.
[0047] Step 10.3: Regularly clean up expired inference records to maintain the efficient operation of the database and ensure the effective use of storage space.
[0048] Compared with the prior art, the beneficial effects of the present invention are:
[0049] 1. This invention provides a method for accelerating inference of machine learning and large language models based on an encapsulation engine. By using the efficient kvcache caching mechanism, data access latency is reduced, thereby significantly accelerating the inference response time and meeting the needs of real-time applications. This method optimizes the use of computing resources during the inference process, reduces the demand on hardware such as CPU and GPU, and enables efficient operation even in resource-constrained environments.
[0050] 2. This invention provides a method for accelerating inference of machine learning and large language models based on an encapsulation engine. Combined with the optimization features of Triton, it can improve the inference efficiency of large models while maintaining high accuracy, ensuring a smooth user experience. The flexibility of Triton makes this method compatible with multiple deep learning frameworks, facilitating model integration and deployment.
[0051] 3. This invention provides a method for accelerating inference of machine learning and large language models based on an encapsulation engine. By accelerating the inference process, it reduces the long-term consumption of computing resources, thereby reducing operating costs, especially in large-scale deployments. The fast inference response time improves the interactivity of the application and user satisfaction, especially in fields with high real-time requirements such as finance, healthcare, and education.
[0052] 4. This invention provides a method for accelerating inference of machine learning and large language models based on an encapsulation engine. This method can support online updates and version management of models, facilitating iteration and optimization in rapidly changing application environments. Attached Figure Description
[0053] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.
[0054] Figure 1 is a schematic diagram of the steps of the present invention.
[0055] Figure 2 shows the comparison of KV-Cache inference acceleration.
[0056] Figure 3 shows the comparison of KV-Cache inference acceleration time.
[0057] Figure 4 shows the comparison of time spent per token for KV-Cache inference acceleration.
[0058] Figure 5 is a comparison data chart of the verification of the present invention. Detailed Implementation
[0059] The technical solution of the present invention will be more clearly and completely explained below with reference to the accompanying drawings and through the description of preferred embodiments of the present invention.
[0060] Terminology Explanation:
[0061] kvcache: A caching mechanism used to store input parameters and their corresponding output results during inference, thereby accelerating subsequent inference requests. By looking up the cache, redundant calculations can be avoided, thus improving efficiency.
[0062] Triton: Triton Inference Server is an open-source inference server designed to provide efficient inference services for machine learning models. It supports various deep learning frameworks (such as TensorFlow, PyTorch, etc.) and provides features such as dynamic batching and model parallelism to optimize inference performance.
[0063] `performInferenceWithCacheCheck:` is a method name used for cache checking during inference. If caching is enabled, this method queries kvcache to check if a result is already available to return.
[0064] config.pbtxt: This is the standard format for Triton model configuration files, containing basic information about the model, such as input and output names, data types, dimensions, etc. It helps the Triton server understand how to process a specific model.
[0065] max_batch_size: The maximum batch size specified in the Triton configuration, which refers to the maximum number of input samples that can be processed simultaneously in a single inference request.
[0066] FP32: Represents a 32-bit floating-point number, a format used to represent numerical values, often used for the input and output of deep learning models.
[0067] INT64: Represents a 64-bit integer, which is another format used to represent numerical values and is typically used to store a wide range of integer values.
[0068] Dynamic updates: This refers to the ability of the Triton model repository to automatically check and update model versions. This ensures that the latest model version is being used.
[0069] Traffic analysis: The process of monitoring and analyzing request traffic within a system in order to optimize system performance and user experience.
[0070] Load balancing: A technique used to distribute requests across multiple backend instances to ensure that the workload of each instance is relatively even, thereby improving the overall performance and responsiveness of the system.
[0071] Task queue: A data structure used to store pending inference requests, enabling asynchronous inference computations and thus improving system responsiveness.
[0072] Callback mechanism: A programming pattern that returns the result to the client in a specific way (such as notification or API call) after inference is completed.
[0073] As shown in Figure 1, the present invention specifically comprises:
[0074] Step 1: Implement the performInferenceWithCacheCheck method in the inference engine, which accepts the boolean parameter enable_cache to control whether to enable the kvcache acceleration function;
[0075] Step 2: Based on the input enable_cache value, execute the conditional judgment logic to determine whether to perform a cache lookup; if true, then perform a kvcache lookup.
[0076] Step 3: Create a Triton model repository in the specified file path, and create a separate folder for each model;
[0077] Step 4: In the config.pbtxt file, define the model's version information, input and output tensor names, data types, and dimensions;
[0078] Step 5: Start the Triton Inference Server using the command-line tool and specify the model repository path.
[0079] The concept of this invention is as follows:
[0080] In the process of business production, we will accelerate the inference of large models. The goal is to improve inference efficiency, significantly reduce the computational complexity and resource consumption of the inference process, and increase the inference speed while ensuring the inference accuracy of large models. This is achieved by accelerating the inference through kvcache caching and encapsulating it with the Triton inference framework.
[0081] KVCache caching enabled
[0082] The system supports configuring the impact of enabling / disabling KV-Cache on inference efficiency. Model inference can be accelerated by enabling KV-Cache. The inference engine can dynamically modify the KV-Cache option after receiving parameters, enabling and disabling KV-Cache via `use_cache`. When `use_cache` is true, the model initializes `past_key_values` to store the key and value vectors during inference, and maintains it at each inference step. If `use_cache` is false, `past_key_values` is ineffective. (See Figures 2-4).
[0083] Triton framework encapsulation
[0084] Extending the existing inference engine to support the Triton framework for inference acceleration, the dynamic batching and concurrent execution features of the Triton framework can be used to accelerate inference when performing large model inference, thereby improving the performance of large model inference.
[0085] Inference Startup Custom Enablement
[0086] When encapsulating the capability state of the model, the page can dynamically enable kvcache to accelerate inference.
[0087] The technical implementation process is described as follows:
[0088] (I) Enable custom kvcache caching acceleration
[0089] The specific implementation is as follows:
[0090] The performInferenceWithCacheCheck method has been added to the inference engine to support dynamically setting whether kvcache is enabled. After the check is made, the original inference logic is called to continue execution.
[0091] (II) Accelerating Triton Framework Encapsulation
[0092] 1. Triton creates a model repository
[0093] 2. Model file configuration
[0094] 3. Start the Triton inference server
[0095] 4. The existing inference engine supports the Triton framework, thereby improving inference acceleration efficiency.
[0096] As a specific implementation method, the system architecture is designed as follows:
[0097] Overall Architecture: The system consists of a client, a KVCACHE layer, and a Triton Inference Server. The client is responsible for sending inference requests and receiving results, KVCACHE is used to cache data, and Triton handles model inference.
[0098] Module division:
[0099] Client module: Responsible for data collection, request sending and result display, supporting multiple data formats (such as images and text).
[0100] The KVCACHE module is responsible for caching input data and intermediate results, providing a fast access interface, and using key-value pairs for storage to ensure efficient lookup and updates.
[0101] Inference engine module: Triton Inference Server, which supports parallel inference of multiple models and has automatic scaling capabilities.
[0102] Data flow management:
[0103] Input data processing:
[0104] The client sends inference requests via REST API or gRPC. The data undergoes preliminary verification and preprocessing before being sent, including format conversion and standardization.
[0105] After preprocessing, the data is transformed into a format suitable for the model input (e.g., scaling the image to a specified size).
[0106] Caching strategy:
[0107] Caching Structure: KVCACHE uses a distributed caching structure, storing the hash value of the input data as the key and the intermediate inference results as the value.
[0108] Caching strategy: The LRU (Least Recently Used) strategy is used to manage the cache in order to improve the hit rate and save storage space.
[0109] Model loading and inference:
[0110] Model Management:
[0111] Triton preloads multiple versions of the model to accommodate different inference needs (such as different precision or optimized versions).
[0112] Specify the model's input / output format, batch size, and other inference parameters through the configuration file.
[0113] Reasoning process:
[0114] Upon receiving an inference request, the system first queries KVCACHE to determine if cached data exists.
[0115] If the cache is hit, return the cached result directly to reduce inference latency.
[0116] If the cache misses, the system calls Triton to perform inference, and after the inference is completed, the result is stored in KVCACHE for later use.
[0117] Result caching and update mechanism:
[0118] Result caching: After inference is completed, the results and their input data are stored in KVCACHE, and a reasonable time-to-live (TTL) is set to ensure the timeliness of the data.
[0119] Dynamic update mechanism:
[0120] When new input data is detected, the system automatically updates the relevant cache in KVCACHE to ensure that the latest data is available.
[0121] It supports event-based update strategies, such as prioritizing updating the corresponding cache when similar data is requested multiple times.
[0122] Performance monitoring and tuning:
[0123] Monitoring module: Implements real-time performance monitoring and collects metrics such as inference time, cache hit rate, and system load.
[0124] By displaying real-time performance data through a visual dashboard, developers can identify performance bottlenecks in a timely manner.
[0125] Tuning strategy:
[0126] Based on monitoring data, the model loading strategy and cache configuration are automatically adjusted to optimize inference performance.
[0127] Regularly analyze system operation, generate reports, and propose improvement suggestions, such as increasing cache size or optimizing model configuration.
[0128] Security and fault tolerance mechanisms:
[0129] Data security: During data transmission, HTTPS protocol is used to encrypt communication and ensure data security.
[0130] Fault tolerance mechanism: A retry mechanism is set up so that when an inference request fails, the system will automatically retry or roll back to the previous stable state to ensure the high availability of the system.
[0131] Through the detailed implementation methods described above, the system can effectively utilize the KVCACHE and Triton engines to accelerate large model inference and improve overall system performance.
[0132] As a specific implementation example, a technology company developed a deep learning-based image classification application with the goal of processing a large number of image requests in real time. The company used the following method:
[0133] 1. System Architecture Design
[0134] Overall architecture:
[0135] The system consists of a user client, a KVCACHE layer, a Triton Inference Server, and a data storage layer.
[0136] The user client interacts with the system via API. KVCACHE is used to store historical inference results, and Triton is responsible for model inference.
[0137] Component details:
[0138] User client: Built with the React framework, it supports image uploading and displaying inference results.
[0139] KVCACHE: Implemented using Redis, it sets a reasonable expiration policy to reduce memory usage.
[0140] Triton Inference Server: Deployed on NVIDIA GPU servers, it supports multiple models and is invoked via a RESTful API.
[0141] 2. Data Flow Management Process
[0142] User input:
[0143] When a user uploads an image on the client, the system first verifies its format and size (JPEG / PNG is supported, maximum size 5MB).
[0144] After successful upload, the client will convert the image into a suitable Tensor format (224x224 RGB).
[0145] Hash generation and cache lookup:
[0146] After processing, the image generates an MD5 hash value as a unique identifier.
[0147] The client queries KVCACHE to see if the hash value already exists.
[0148] Cache processing logic:
[0149] Cache hit:
[0150] If KVCACHE returns a result, the client directly displays the classification result and records the number of hits.
[0151] Cache missed:
[0152] If no result is found, the client sends the image data to Triton for inference.
[0153] 3. Model Loading and Inference Mechanism
[0154] Model configuration and loading:
[0155] The Triton configuration file defines the model input and output specifications, uses a ResNet model optimized with TensorRT, and supports batch inference (batch size is set to 4).
[0156] The model preloading mechanism ensures a fast response during inference.
[0157] Reasoning process:
[0158] The client sends image data to Triton via an HTTP POST request.
[0159] After processing the request, Triton performs inference and returns the results, including the category label and confidence level.
[0160] The system records the inference time and compares it with the set target time (e.g., 200 milliseconds).
[0161] 4. Result caching and update strategy
[0162] Result storage:
[0163] After Triton inference is completed, the classification results (labels, confidence scores, image hashes) are stored in KVCACHE with a TTL of 10 minutes, supporting fast repeated requests.
[0164] Dynamic updates and cleanup:
[0165] The LRU (Least Recently Used) algorithm is used for cache eviction to ensure that frequently used results are retained in the cache.
[0166] Regularly check the cache status, calculate the hit rate, and optimize the TTL settings.
[0167] 5. Performance monitoring and optimization
[0168] Monitoring tools:
[0169] Use Prometheus to collect performance metrics, including inference latency, cache hit rate, and request volume.
[0170] Configure Grafana to display dashboards and monitor system health status in real time.
[0171] Performance optimization:
[0172] Analysis of monitoring data revealed that the main bottleneck to inference latency lies in model loading.
[0173] Measures were taken, such as increasing the number of Triton instances and utilizing NVIDIA GPUs for parallel processing.
[0174] 6. Safety and Fault Tolerance Design
[0175] Data transmission security:
[0176] All API interfaces are encrypted using HTTPS to ensure user data security.
[0177] Fault tolerance:
[0178] The system is designed with a retry mechanism. If Triton inference fails, the system will retry after 1 second, with a maximum number of retries set to 3.
[0179] The logging system records all failed requests and retries, facilitating later analysis and repair.
[0180] 7. Results and Evaluation
[0181] Through the above design and implementation, the system achieved the following results:
[0182] Improved inference speed: Average inference time reduced from 350 milliseconds to 150 milliseconds.
[0183] Cache hit rate: up to 90%, greatly reducing the computational burden of repeated inference.
[0184] Improved user experience: Basic real-time response has been achieved, and user satisfaction surveys show that 94% of users said the system responded quickly.
[0185] This case study demonstrates the comprehensive implementation path and effectiveness of an inference acceleration system based on KVCACHE and Triton, providing a valuable reference for subsequent related projects.
[0186] This paper proposes a design for accelerating large-model inference based on kvcache and the Triton encapsulation engine. Technically, it employs kvcache caching and the Triton encapsulation engine, which is superior to traditional methods that directly implement an inference acceleration engine. In terms of business applications, it can better and more flexibly support enterprise data intelligence, especially in intelligent question-answering inference on large models, helping enterprises reduce costs, increase efficiency, and enhance their competitiveness. The relevant verification results are shown in Figure 5.
[0187] This method has several beneficial effects, as detailed below:
[0188] First, by leveraging kvcache's efficient caching mechanism, this method significantly improves inference speed. When processing large-scale data, kvcache effectively reduces data access latency and optimizes memory and storage usage. This results in a substantial reduction in inference response time, meeting the demands of real-time applications such as online question answering and real-time data analysis.
[0189] Secondly, this approach effectively optimizes the use of computing resources. Large models typically have very high demands on hardware resources such as CPUs and GPUs, while kvcache reduces the consumption of computing resources by caching intermediate results and duplicate data. This is especially important in resource-constrained environments, enabling low-power devices to run complex models and expanding the boundaries of applications.
[0190] Furthermore, by incorporating various optimization techniques from the Triton engine, this method not only maintains high model accuracy but also improves inference efficiency. Triton supports strategies such as dynamic batch processing and model parallelism, enabling multiple requests to be processed simultaneously, thereby further enhancing throughput. This ensures stable service quality under high concurrency in large-scale application scenarios such as intelligent customer service and financial risk control.
[0191] In terms of user experience, rapid reasoning responses significantly enhance interactivity and user satisfaction. Users can experience instant feedback when using financial analysis tools, medical diagnostic systems, or personalized learning platforms, increasing the system's usability and appeal.
[0192] By reducing computing costs, businesses can achieve significant economic benefits from large-scale deployments. Reduced demand for high-performance computing hardware lowers overall operating costs, allowing businesses greater flexibility in their technology investments.
[0193] This method also boasts excellent scalability, supporting online model updates and version management, facilitating rapid response to changing market demands. In fast-paced application environments, developers can easily deploy new model versions while maintaining high service availability and stability.
[0194] Finally, Triton's flexibility and compatibility allow the method to support multiple deep learning frameworks, facilitating model integration and deployment. This enables different teams and projects to quickly adapt to and meet their respective technical needs, driving innovation and development across the industry.
[0195] In summary, this method demonstrates significant advantages in improving inference speed, optimizing resource utilization, enhancing user experience, reducing costs, and increasing scalability, providing strong support for the practical application of large-scale models.
[0196] The above-described specific embodiments are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Various modifications, substitutions, and improvements made by those skilled in the art to the technical solutions of the present invention based on the provided textual description and drawings, without departing from the design concept and spirit of the present invention, should all fall within the scope of protection of the present invention. The scope of protection of the present invention is determined by the claims.
Claims
1. A method for accelerating inference in machine learning and large language models based on an encapsulation engine, characterized in that: Includes the following steps: Step 1: Implement the performInferenceWithCacheCheck method in the inference engine, which accepts the boolean parameter enable_cache to control whether to enable the kvcache acceleration function; Step 2: Based on the passed-in enable_cache value, execute the conditional judgment logic to determine whether to perform a cache lookup; If true, then perform a kvcache lookup; Step 3: Create a Triton model repository in the specified file path, and create a separate folder for each model; Step 4: In the config.pbtxt file, define the model's version information, input and output tensor names, data types, and dimensions; Step 5: Start the Triton Inference Server using the command-line tool and specify the model repository path.
2. The method for accelerating inference of machine learning and large language models based on an encapsulation engine according to claim 1, characterized in that, The specific method for determining user configuration in step 2 is as follows: Step 2.1: In the performInferenceWithCacheCheck method, determine the boolean value of the enable_cache parameter using a conditional statement; Step 2.2: If enable_cache is true, call the kvcache query method to check if the corresponding inference result is already in the cache; Step 2.3: If the cache is hit, return the result in the cache directly; if the cache is not hit, record the current input parameters and continue with the subsequent inference calculation.
3. The method for accelerating inference of machine learning and large language models based on an encapsulation engine according to claim 1, characterized in that, The specific process of configuring and creating the Triton model repository in step 3 is as follows: Step 3.1: Specify the path to the model repository in the file system; Step 3.2: Create a separate folder for each model to be deployed under the path in Step 3.1, with the naming format / models / model_name / ; Step 3.3: In each model folder, prepare the model file and its configuration file config.pbtxt, ensuring that the file format conforms to the Triton specification.
4. The method for accelerating inference of machine learning and large language models based on an encapsulation engine according to claim 1, characterized in that, The detailed configuration method for the model file in step 4 is as follows: Step 4.1: In the config.pbtxt file, define the basic information of the model, including the platform, max_batch_size, input and output names and data types; Step 4.2: Specify the data type, shape, and related parameters for each input and output tensor; Step 4.3: Ensure that the model version information is clear and set a version number for each model as needed.
5. The method for accelerating inference of machine learning and large language models based on an encapsulation engine according to claim 1, characterized in that, The specific steps to start the Triton inference server in step 5 are as follows: Step 5.1: Use the command-line tool to execute the startup command for the Triton Inference Server; Step 5.2: During the startup process, monitor the terminal output to confirm that all models are loaded correctly and there are no error messages; Step 5.3: Verify whether the inference server can receive and respond to inference requests by accessing the specified API interface.
6. The method for accelerating inference of machine learning and large language models based on an encapsulation engine according to claim 1, characterized in that, The specific method for implementing kvcache lookup in step 2 is as follows: Step 6.1: Store the input parameters and corresponding output results for each inference in kvcache, forming a key-value pair mapping; Step 6.2: When making a new inference request, first construct a query key based on the input parameters and check if the query key exists in kvcache; Step 6.3: If the query result exists, return the corresponding output result directly; If the query result is not found, the request is recorded, and the inference calculation is prepared.
7. The method for accelerating inference of machine learning and large language models based on an encapsulation engine according to claim 1, characterized in that, In step 3, the dynamic update function of the Triton model is configured as follows: Step 7.1: In the Triton model repository, create a monitoring script to periodically check for new model versions and updates; Step 7.2: When a new model version is detected, automatically update the config.pbtxt file and replace the old version in the model repository; Step 7.3: After the update, restart the Triton inference server to load the new version of the model.
8. The method for accelerating inference of machine learning and large language models based on an encapsulation engine according to claim 1, characterized in that, Step 4, which implements data compatibility and stability during the inference process, is detailed as follows: Step 8.1: When an inference request arrives, record the source IP address and request time for traffic analysis. Step 8.2: Based on real-time traffic conditions, distribute inference requests to different backend instances to ensure load balancing for each instance; Step 8.3: When an instance is overloaded, dynamically adjust the request routing strategy to prioritize distributing requests to instances with lower current load.
9. The method for accelerating inference of machine learning and large language models based on an encapsulation engine according to claim 1, characterized in that, Step 5 involves processing subsequent inference requests as follows: Step 9.1: Upon receiving an inference request, immediately return a confirmation of receipt to inform the client that inference is in progress; Step 9.2: Place the inference task into the task queue and use the thread pool to execute the inference calculation asynchronously to improve the system's responsiveness; Step 9.3: After the reasoning is complete, send the result to the client through the callback mechanism, or make it available for the client to query through the specified interface.
10. The method for accelerating inference of machine learning and large language models based on an encapsulation engine according to claim 1, characterized in that, Step 5 also includes implementing persistent storage of the inference results, as detailed below: Step 10.1: After the inference is completed, record the input parameters and output results of each inference in the database to form a complete inference log; Step 10.2: Assign a unique identifier to each record for subsequent querying and analysis; Step 10.3: Regularly clean up expired inference records to maintain the efficient operation of the database and ensure the utilization of storage space.