Patents
Literature
Patsnap Eureka AI that helps you search prior art, draft patents, and assess FTO risks, powered by patent and scientific literature data.

470 results about "Request queue" patented technology

Dynamic resource scheduling method and device for multi-GPU cluster reasoning service

The invention provides a dynamic resource scheduling method and device for multi-GPU cluster reasoning service, and belongs to the field of computing resource scheduling. Comprising the steps of obtaining load levels, request queue lengths and resource utilization rates of a multi-GPU cluster system at a current moment and a historical moment, and determining a resource allocation strategy of a GPU cluster by utilizing a reinforcement learning model based on the load levels, the request queue lengths and the resource utilization rates of the multi-GPU cluster system at the current moment and the historical moment; predicting a future load state by using a time sequence prediction model based on the load level, the request queue length and the resource utilization rate of the current moment and the historical moment, and adjusting a resource allocation strategy based on a prediction result; and after the resource allocation strategy is adjusted, updating the reinforcement learning model based on an online learning model according to a feedback result of actual operation of the GPU cluster. According to the method, the scheduling strategy is optimized through reinforcement learning, and the capacity expansion and shrinkage scheme is dynamically adjusted according to the system actual load, the resource utilization rate and the like.
Owner:BEIJING ACAD OF ARTIFICIAL INTELLLIGENCE

Data full-flash storage optimization method and system based on cloud computing

The invention provides a data all-flash storage optimization method and system based on cloud computing, and the method comprises the steps: receiving a storage operation request stream of a user terminal through collecting a multi-dimensional performance data set of each node of an all-flash storage cluster in real time, wherein the multi-dimensional performance data set comprises a storage space fragmentation rate, an input / output request queue depth and a network link bandwidth utilization rate; analyzing a to-be-stored data block identifier set and an operation mode label, traversing a historical access record library based on to-be-stored data block identifiers, extracting historical access time sequence characteristics and physical storage position proximity of associated data blocks, and combining the multi-dimensional performance data and the historical access time sequence characteristics to obtain a data block identifier set; according to the method, a dynamic access popularity prediction model in a preset time period is constructed, and finally, a cross-node fragmentation storage topological graph and a cache preloading strategy matrix are generated according to the proximity of the dynamic access popularity prediction model and a physical storage position, so that the optimization of all-flash storage is realized, and the data storage performance in a cloud computing environment is improved.
Owner:YUGE ELECTRONIC TECH CO LTD

Three-dimensional data visualization rendering method and visualization rendering engine system

The invention relates to the technical field of visual rendering, and provides a three-dimensional data visual rendering method and a visual rendering engine system.The method comprises the steps that space segmentation processing is conducted on three-dimensional model data to be rendered through the LOD technology to generate block slice data with different geometric detail levels, organizing the block slice data through a tree structure with a spatial relationship, and outputting the tree structure as an index file; the method comprises the following steps: adaptively screening block and slice data of different levels of detail based on a user vision field range, storing the block and slice data into a rendering request queue, querying block and slice data to be rendered from the rendering request queue according to index codes of the block and slice data, and updating a layer browser database; according to the method, to-be-rendered block slice data are downloaded and analyzed in parallel in batches in a separated collaborative operation mode, and a rendering task is executed according to a data analysis result, so that the rendering scheduling frame rate is remarkably improved, and rapid visualization of urban mass three-dimensional slice data at a webpage end is realized.
Owner:INTERSTELLAR SPACE (TIANJIN) TECH DEV CO LTD

Large language model reasoning service optimization method, system, equipment and medium

The invention discloses a big language model reasoning service optimization method, system and device and a medium. The method comprises the steps of obtaining a current reasoning request queue; performing dynamic sorting on each reasoning request in the current reasoning request queue; and performing dynamic batch processing on each reasoning request in the reasoning request queue by using an SLO-BS algorithm to obtain a current batch of reasoning requests, and inputting the current batch of reasoning requests into a large language model, the method, the system, the equipment and the medium can solve the problem of video memory fragmentation occurring during multi-task concurrent reasoning, and reduce the SLO violation rate at the same time.
Owner:SHENZHEN INST OF ADVANCED TECH CHINESE ACAD OF SCI

Parallel rollout verification processing for deploying updated software

In some implementations, a rollout component may deploy, in a pilot mode, an updated version of a software application in a runtime environment executing a legacy version of the software application. The rollout component may process requests in a request queue using the legacy version to generate processing results that are dispatched to a client interface and may process, using the updated version, copies of the requests processed by the legacy version. The rollout component may indicate whether the updated version passes one or more tests based on a comparison between the validation data generated by the updated version and processing results generated by the legacy version. The rollout component may switch the updated version from the pilot mode to a live mode based on the updated version passing the one or more tests.
Owner:CAPITAL ONE SERVICES LLC

Read-write request processing method, protocol conversion module, electronic equipment and storage medium

The embodiment of the invention provides a read-write request processing method, a protocol conversion module, electronic equipment and a storage medium, and relates to the technical field of chips. The method comprises the steps that a protocol conversion module is arranged in a processor cluster, an original read-write request is converted into a first read-write request conforming to a CHI protocol through a first conversion module in the protocol conversion module, and a corresponding transmission channel is distributed to the first read-write request; and the request queue module orderly sends the received first read-write request, and the second conversion module recombines the first read-write request information to a second read-write request conforming to the CHI protocol, and allocates a corresponding channel for the second read-write request. According to the embodiment of the invention, the internal data channel (AXI or similar AXI) is directly converted into the CHI interface in the processor cluster, the internal circuit resources of the processor cluster are reused, the circuit area is saved, and the access path time delay is reduced.
Owner:ARM TECH CHINA CO LTD

Data read-write method, system and device, computing equipment and storage medium

The embodiment of the invention provides a data read-write method, system and device, computing equipment and a storage medium, and the data read-write method comprises the following steps: receiving a data read-write request sent by a user mode through an asynchronous I / O layer, and storing the data read-write request into a request queue; under the condition that the asynchronous I / O layer determines that the data read-write request is of a straight-through type, the data read-write request is sent to a driving layer; and analyzing the data read-write request through the driving layer, and sending request information of the data read-write request to target processing end equipment for read-write processing. The asynchronous I / O layer receives the data read-write request sent by the user mode and sends the data read-write request to the driving layer after the data read-write request is determined to be of the straight-through type, context switching and data copying expenses are reduced, the processing efficiency of I / O operation is improved, the request is analyzed in the driving layer and sent to the processing end equipment for read-write processing, scheduling of I / O tasks is optimized, and the user experience is improved. The disk addressing time and I / O delay are reduced, and the overall performance and the system stability are improved.
Owner:ALIBABA CLOUD COMPUTING CO LTD

Model reasoning request management method, device and system

The invention discloses a request management method, device and system for model reasoning. The method comprises the following steps: obtaining an observation index of a model reasoning service; index prediction is carried out according to the observation indexes, prediction indexes are obtained, and the prediction indexes comprise the target memory utilization rate of the central processing unit and the target video memory utilization rate of the graphics processing unit; determining a scheduling strategy according to the prediction index, determining a scheduling suggestion according to the scheduling strategy in combination with the request queue, and adding the scheduling suggestion into a suggestion buffer queue; when the model reasoning service executes request scheduling, the scheduling suggestion is read from the suggestion buffer queue, a target scheduling decision is determined according to the scheduling suggestion in combination with a scheduling decision of the model reasoning service, and the model reasoning service is controlled to schedule corresponding request execution model reasoning according to the target scheduling decision; the problem of unreasonable request scheduling in the model reasoning process is solved, and efficient utilization and reasonable allocation of resources are realized.
Owner:TRANSWARP TECHNOLOGY (SHANGHAI) CO LTD

Heterogeneous network intelligent slice resource scheduling method and device, electronic equipment and medium

The invention relates to the technical field of network resource scheduling, and provides a heterogeneous network intelligent slice resource scheduling method and device, electronic equipment and a medium. The method comprises the following steps: performing three-dimensional joint modeling on heterogeneous network physical resources to obtain a dynamic resource state tensor, and performing a dynamic resource allocation decision by combining a deep reinforcement learning model with a service demand matrix to obtain an initial slice resource allocation matrix; and performing mixed integer linear programming through a preemption decision model in combination with the URLLC service request queue to obtain a resource allocation sequence, and performing occupancy rate updating on the dynamic resource state tensor according to the resource allocation sequence to obtain ground network resource data, and carrying out resource fusion in combination with the satellite beam capacity data to obtain a space-ground integrated resource pool tensor, thereby carrying out traffic prediction in combination with historical traffic data through a long-short memory network to obtain a bandwidth pre-adjustment strategy. Through organic integration of technologies such as mixed integer linear programming and traffic prediction, the overall network resource utilization rate is improved.
Owner:SHENZHEN AUGOO COMM EQUIP CO LTD

Large model batch reasoning and data flow optimization system oriented to MOE architecture

The invention relates to the technical field of project management, in particular to a large-model batch reasoning and data flow optimization system oriented to an MOE architecture. According to the method, a collaborative architecture of the request access module, the environment sensing module, the expert routing engine, the resource scheduling module and the dynamic optimization control module is set, the text length and the subject type are extracted by using the request access module, a basis is provided for accurate routing, and the GPU video memory, the I / O bandwidth and the request queue depth are acquired in real time through the environment sensing module, so that the real-time routing is realized. The system load is comprehensively monitored, meanwhile, an expert sub-network is activated through an expert routing engine according to request features, invalid calculation is avoided, weight loading and resource allocation are managed through a resource scheduling module, the I / O bottleneck is reduced, and finally an optimization strategy is intelligently triggered through a dynamic optimization control module based on routing conflict factors. The problems of large reasoning delay fluctuation and unbalanced resource utilization rate mentioned in the background technology are solved, and stable low-delay response and resource collaborative optimization in a high-concurrency scene is realized.
Owner:VIRTAI TECH BEIJING CO LTD

Video stream concurrent access front-end dynamic scheduling control system and method based on digital twinborn scene

The invention discloses a video stream concurrent access front-end dynamic scheduling control system and method based on a digital twinborn scene, and relates to the technical field of computer software and network communication, the system comprises three core modules: a self-adaptive request queue module manages UE rendering resources based on a session token pool, distributes tokens through preemptive scheduling, and sends the tokens to a server; in combination with a GPU frame rate feedback dynamic adjustment strategy, queuing through a Promise asynchronous queue when the request fails; the front-end fusing and backoff retry module monitors the connection failure rate, when the connection failure rate reaches a threshold value, fusing is triggered, retry is delayed by adopting an exponential backoff strategy, and a semi-open state tends to recover; and the local cache collaboration module intercepts the request through Service Worker, caches the latest video clip, pushes the local content when the request is delayed, and ensures the freshness through ETag verification. The method and the device are used for efficiently managing loading and interaction of multiple users on real-time rendering pictures at a browser end.
Owner:浪潮智慧城市科技有限公司

Dynamic graphics processor sharing method and device and storage medium

The invention relates to the technical field of graphics processors, and provides a dynamic graphics processor sharing method, which comprises the following steps of: receiving submission of a plurality of deep learning tasks; generating a dynamic priority queue containing high-priority tasks and low-priority tasks; transmitting a resource state to the low-priority task through a broadcast mechanism, and triggering resource adjustment according to a comparison result of the real-time thread block usage amount of the high-priority task and a preset threshold value; according to the difference value between the real-time thread block usage amount of the high-priority task and a preset threshold value, the number of executable thread blocks of the low-priority task is dynamically adjusted; checking whether the number of thread blocks required by the kernel is smaller than or equal to the number of executable thread blocks of the current low-priority task, if yes, starting execution, and if not, reserving the kernel in a kernel request queue to wait for next cycle scheduling; and periodically updating the dynamic priority queue until all deep learning tasks are completed. According to the technical scheme, the overall efficiency of the system can be improved, and resource waste is reduced.
Owner:HARBIN INSTITUTE OF TECHNOLOGY (SHENZHEN) (INSTITUTE OF SCIENCE AND TECHNOLOGY INNOVATION HARBIN INSTITUTE OF TECHNOLOGY SHENZHEN)

Chip verification method and device, equipment, medium and program product

The invention provides a chip verification method and device, equipment, a medium and a program product, and the method comprises the steps: receiving a first operation instruction of a to-be-verified module and a second operation instruction of a reference model in parallel, the reference model being a function model used for simulating the expected behavior of the to-be-verified module; searching a first instruction record matched with the first operation instruction from a request queue of the reference model, and searching a second instruction record matched with the second operation instruction from a request queue of the to-be-verified module; and determining a verification result of the to-be-verified module according to a comparison result of the first operation instruction and the first instruction record and / or a comparison result of the second operation instruction and the second instruction record. According to the invention, a structured and reusable universal chip verification method is realized, so that a tedious customized scoreboard design is replaced, and the efficiency and accuracy of verification work are greatly improved.
Owner:SHANGHAI BIREN TECH CO LTD

Request method and device for cache resources, equipment and medium

The invention discloses a cache resource request method and device, equipment and a medium, and relates to the technical field of computers, and the method comprises the steps that a consumer determines a target cache resource needing to be requested; when the consumer determines that the priority request cache partition corresponding to the target cache resource is a private cache partition, comparing whether the first current available size of the private cache partition is greater than or equal to the size of the request cache resource; if yes, applying for a first target cache resource fragment of the target cache resource type from a first provider of the private cache partition; if not, judging whether a request record of the shared cache partition exists or not; if not, comparing whether a second current available space corresponding to the consumer in the shared cache partition is greater than or equal to the size of the requested cache resource; if yes, applying for a second target cache resource fragment of the target cache resource type from a second provider of the shared cache partition; and if not, generating a request record, and writing the shared cache resource request instruction into a shared cache resource request queue.
Owner:SHANDONG YUNHAI GUOCHUANG CLOUD COMPUTING EQUIP IND INNOVATION CENT CO LTD

Method and system for storing multiple concurrent large files

The invention relates to the technical field of file storage, and discloses a method and a system for storing multiple concurrent large files. According to the method and the system, the format and the size of a file are verified, a unique file name is generated and used for identifying metadata of a storage file and outputting the file, each small block of the file is divided into a fixed size and is divided into small blocks, a unique identifier is generated for each small block, a database is connected, and the metadata of the file is stored in the database. Then traversing each small block, selecting a storage node for each small block, uploading the small blocks by using multiple threads, setting a request queue, processing file uploading and downloading requests in sequence, avoiding concurrent write-in conflicts of the same file by using a lock mechanism for file uploading, and reading and controlling a read request through a read-write lock; and acquiring the information of the storage file from the database, acquiring each small block by utilizing a concurrent downloading mode, recombining the small blocks into an original file in sequence after downloading is completed, and performing integrity verification after file recombining.
Owner:SHENZHEN QUANXING TECH CO LTD

Data backup method and device

The invention relates to a data backup method and device, and belongs to the technical field of computer storage. Comprising the steps of installing a kernel mode drive program, starting a user mode backup application program, obtaining logical volume information of a target server, sending the logical volume information to the kernel mode drive program, registering a block device consistent with the logical volume information in a kernel of the target server, distributing a block device request queue in the kernel of the target server, the method comprises the steps of receiving a logical volume request queue in a target server, binding the logical volume request queue in the target server to a block device request queue, receiving logical volume bitmap information, determining change data according to the logical volume bitmap information, transferring the change data, and receiving a backup data reading request through a block device; and judging whether the data reading request relates to changed data, if so, returning the transferred data to the user state backup application program, otherwise, returning the disk data to the user state backup application program, and storing the obtained returned data to the storage medium.
Owner:CHINA TELECOM DIGITAL INTELLIGENCE TECH CO LTD

Serverless request scheduling device and system based on time delay perception

The invention relates to a Serverless request scheduling device and system based on time delay awareness, and the device comprises the steps: a first scheduling module determines a determined user request at the head of a current request queue as a target user request, and carries out the scheduling of the target user request under the condition that a target pod which meets an execution condition and executes the target user request does not exist; sending a pod creation request to a second scheduling module; the second scheduling module selects a target node from a plurality of nodes and newly builds a pod in the target node under the condition of determining that the newly built pod meets a node building condition based on the pod building request, and sends the information of the newly built pod to the first scheduling module; and the first scheduling module manages the newly-built pod according to the received information of the newly-built pod, determines the newly-built pod as a target pod, and sends the newly-built pod to the target pod for execution according to a target user request. Node resources can be utilized more effectively, and the SLO violation proportion is reduced.
Owner:HUAWEI TECH CO LTD

Separate command address (SCA) based memory controller

The subject application relates to separate command address (SCA) protocol-based memory controllers. For instance, a method may include receiving, by a separate command address (SCA)-based memory controller, a plurality of commands, scheduling, by the SCA-based memory controller, the plurality of commands using two or more data path scheduler (DPS) request queues, converting the scheduled plurality of commands into SCA-based commands, selecting, from the scheduled SCA-based commands, one or more scheduled SCA-based commands for a sequence execution, and executing in sequence the selected one or more SCA-based commands using one or more logical units (LUNs) of a channel in a memory component.
Owner:MICRON TECHNOLOGY INC

Managing changes to a computing environment

There is provided a computer-implemented method for handling updates to a computing environment, the method comprising: receiving, from a user, a change request comprising a text description justifying a change in hardware and / or software within the computing environment; applying natural language processing to the text description to extract one or more text features from the text description; performing a validation check of the received change request by applying a trained classification model to the extracted one or more text features, the classification model having been trained on a dataset of historical change requests; and if the received change request fails the validation check, preventing submission of the change request into a queue of change requests and / or implementation of the change associated with the change request.
Owner:BARCLAYS EXECUTION SERVICES LTD

Large model adaptive batch reasoning system and method based on time delay measurement

The invention discloses a large-model adaptive batch reasoning system based on time delay measurement, and the system comprises a request receiving and queue management module which is used for monitoring and receiving an LLM reasoning request sent by an external user or an application; the real-time delay sensing and batch monitoring module is used for measuring and analyzing performance indexes directly related to the current batch processing strategy in real time in the LLM reasoning process; the Token-by-Token batch processing and reasoning execution module is used for selecting a corresponding number of requests from the request queue according to the size of the current batch determined by the scheduler, organizing the requests into an effective calculation batch and submitting the effective calculation batch to the LLM reasoning core at the bottom layer to execute one or more decoding steps; and the double-stage batch adjustment decision module is used for obtaining the normalized time delay ratio. The invention also discloses a large-model adaptive batch reasoning method based on time delay measurement. According to the invention, substantive improvement and intelligent management of the overall performance of the LLM reasoning service are realized.
Owner:NANJING UNIV

Deep learning inference engine tensor optimization method and system for GPU acceleration

The invention relates to the technical field of data processing, and provides a deep learning inference engine tensor optimization method and system for GPU acceleration. Performing asymmetric quantization on the first weight tensor of the original model to generate a second weight tensor, and updating the original model based on the second weight tensor to obtain an updated model; blocking the data tensor according to a preset dimension to generate input sub-blocks, and loading the input sub-blocks and a convolution kernel in the update model to a shared memory; calling the input sub-blocks from the shared memory, and processing the input sub-blocks of different batches in parallel based on a single kernel function formed by convolution kernels to generate a processing result; and based on the GPU utilization rate and the request queue length corresponding to the to-be-processed data, performing dynamic output adjustment to output a processing result. The performance and efficiency of a deep learning inference engine are improved, memory occupation and calculation delay are reduced, and powerful support is provided for deep learning application of GPU acceleration.
Owner:SHENZHEN HUANDONG INTELLIGENT TECH CO LTD

Method and system for dynamically scheduling micro-service gateway based on coroutine

The invention discloses a coroutine-based micro-service gateway dynamic scheduling method, which comprises the following steps that: 1, a coroutine pool is created to process a network request, and a gateway receives the network request and adds the network request into a request queue; 2, identifying a request type, distributing a preset static priority for the network request, and adjusting the dynamic priority in combination with features on the basis of the static priority and the configured initial dynamic priority; and 3, collecting data in coroutine operation in real time, and reducing coroutine processing delay by calculating an overtime threshold, adopting an overtime differentiation strategy, a low-priority discarding strategy and updating a data state according to a data operation condition. The invention further discloses a dynamic scheduling micro-service gateway system based on the coroutine. The method can significantly improve the gateway performance, reduce the system delay, intelligently manage the request queue under a high load condition, ensure the normal operation of key services, and can be widely applied to the field of computer data.
Owner:CHANGJIANG SECURITIES

Multi-thread parallel computing efficiency optimization system

The invention relates to the technical field of GPU parallel processing, in particular to a multi-thread parallel computing efficiency optimization system, which comprises a GPU, a processor and a memory in which a computer program is stored, and when the computer program is executed by the processor, the following steps are realized: a plurality of instruction streams corresponding to a parallel computing module at a preset time point, traversing each instruction stream, acquiring first I instructions in the corresponding instruction stream to obtain a plurality of instructions, generating a plurality of requests to be arbitrated according to the instructions meeting preset conditions, traversing all the requests to be arbitrated, and adding the requests to be arbitrated into request queues of corresponding access objects according to the access objects corresponding to the requests to be arbitrated, according to the multi-thread parallel computing method, the request queues corresponding to the access objects are obtained, each request queue is arbitrated, and each arbitration result is sent to the corresponding access object to be processed, so that instructions of different access objects can be executed in parallel, and the multi-thread parallel computing efficiency is improved.
Owner:METAX INTEGRATED CIRCUITS (SHANGHAI) CO LTD

Dynamic management of memory read requests

This application is directed to dynamic management of memory read request in a memory system of an electronic device. The electronic device identifies a queue of memory access requests to access the memory system. The queue of memory access requests including at least one host read request and a current system read request. The electronic device monitors a workload condition of the memory system based on the queue of memory access requests, and generates at least a first system read request and a second system read request from the current system read request based on the workload condition of the memory system. The queue of memory access requests is updated by inserting the at least one host read request after the first system read request and before the second system read request.
Owner:SK HYNIX NAND PRODUCT SOLUTIONS CORP

Managing instances of serverless functions in a cloud computing system

Computer-implemented methods for managing instances of serverless functions in a cloud computing system are provided. Aspects include obtaining a service level objective for a serverless function, obtaining a command queue length for a graphical processing unit disposed on each of a plurality of compute nodes in the cloud computing system, and obtaining a request queue length of the serverless function. Aspects also include calculating a number of instances of the serverless function to deploy in the cloud computing system, wherein the number of instances is determined based on the service level objective and the request queue length of the serverless function, identifying compute nodes from the plurality of compute nodes to deploy each of the number of instances of the serverless function, and creating an instance of the serverless function on each of the identified compute nodes.
Owner:INTERNATIONAL BUSINESS MACHINE CORPORATION

Data playback method, data processor, network interface card, device and storage medium

The invention relates to a data playback method, a data processor, a network interface card, equipment and a storage medium. The method comprises the steps of obtaining a to-be-replayed request queue from a persistent memory under the condition that a storage stack is restarted; the to-be-replayed request queue is a queue obtained by mapping a transmission buffer area in a persistent memory, and the transmission buffer area is used for receiving a request from a host side; obtaining a to-be-replayed request from the persistent memory according to the to-be-replayed request queue; and resending the to-be-replayed request to a target acceleration component pointing to the storage device to indicate the storage device to store target data corresponding to the to-be-replayed request in the persistent memory. According to the method, the request queue, the request and the data corresponding to the request are stored in the persistent memory in advance, if the storage stack is restarted, the to-be-replayed request is recovered from the persistent memory, DAM and encryption and decryption technologies are not needed, the interruption duration of a data storage path can be shortened, and the data storage efficiency of a host side can be improved.
Owner:NANJING JAGUAR MICROSYSTEMS CO LTD +1

Method and system for realizing RDMA network card request queue

The invention relates to the technical field of computer networks, in particular to an implementation method and system for an RDMA network card request queue, and the method comprises the steps: checking whether a residual space exists in an RDMA Cache module or not when a write request is received; if yes, writing the writing information of the writing request into an RDMA Cache module, updating a free bit, and recording a queue pair number and an unfinished pointer; if not, writing the writing information of the writing request into a host memory module; when a read request is received, checking whether the queue pair number and the unfinished pointer carried by the read request are matched with historical records in the RDMA Cache module or not; if yes, reading from the RDMA Cache module; and if not, reading from the host memory module. According to the invention, multiple QPs in the RDMA network card share the internal resources of the Cache, the resource consumption is reduced, and the utilization rate of the cache is improved.
Owner:WUXI STARS MICRO SYSTEM TECHNOLOGIES CO LTD

Load balancing scheduling method and device for model reasoning and electronic equipment

The invention discloses a load balancing scheduling method and device for model reasoning and electronic equipment. The method comprises the following steps: receiving a current request of a user; estimating the processing time of the current request according to the parameters of the historical request, the processing time of the historical request and the parameters of the current request; judging whether idle service instances exist or not according to the service instance state list; wherein the service instance state list comprises service instance numbers and states and priorities thereof; if no idle service instance exists, the current request and the corresponding parameters are placed in a request queue; and if the idle service instance exists, distributing the request selected from the request queue according to the first index or the current request to the service instance selected according to the second index, and updating the state of the service instance in the service instance state list. According to the method, load balancing is carried out on the request by recording and managing the processing state of the back-end service instance, and the method is more suitable for model reasoning service.
Owner:BEIJING ACAD OF ARTIFICIAL INTELLLIGENCE