The embodiment of the invention provides a
large model service traffic scheduling method and
system in a K8S environment, a storage medium and a
computer program. According to the
large model service traffic scheduling method in the K8S environment, firstly, a model reasoning request is submitted through a
unified interface gateway, then target model information and
authentication information of the request are extracted, and deployment instances of corresponding model services are automatically identified and matched by analyzing model names in the request. A
priority queue algorithm based on the number of tasks is introduced to obtain a service instance with the minimum number of reasoning tasks in current execution in real time to serve as a target instance, the model reasoning request is preferentially allocated to the target instance to achieve
concurrency control, and a heap
sorting algorithm in a memory is adopted to dynamically maintain the current connection state of back-end model service. And identifying the GPU type of the node and intelligently distributing flow according to a preset flow distribution strategy. The stability, the
resource utilization rate and the cost management capability of the
large model service are remarkably improved.