2 results about "Real time routing" patented technology

This invention belongs to the field of distributed large language model inference scheduling technology, and relates to a multi-model dynamic scheduling method for high-concurrency dialogue requests. The method includes: collecting the network card hard interrupt timestamp sequence at the streaming response bus to establish a streaming output time interval vector; extracting the differential fluctuation envelope of the streaming output time interval vector and comparing it with a reference transmission delay line to separate the channel congestion index; calculating the real-time routing suppression coefficient based on the queue backlog length and the channel congestion index and reconstructing the cost-aware routing matrix; compressing the sampling window when concurrent data traffic is overloaded, and sending a redirection signal to shift the concurrent data stream when the real-time routing suppression coefficient exceeds the limit. This invention eliminates the need for active probing, decouples the calculation of congestion and network jitter in situ, effectively eliminates long-tail response latency, and improves throughput.

The application discloses a kind of real-time routing switching method and device based on multi-rule decision engine.The method is calculated by the real-time feedback data of the combined node collected, and the investment return rate of each node is obtained;Through routing decision engine, according to the investment return rate, the pre-defined multi-dimensional performance evaluation and decision rule is executed, and the performance of the combined node is evaluated and compared, and the routing switching decision is generated;During marketing activity, according to the routing switching decision, the routing switching between suppliers or strategies is executed;The routing switching decision and the corresponding execution effect are recorded to the strategy effect knowledge base.The application can achieve the technical effect of automatically switching data from inefficient suppliers or strategies to efficient suppliers or strategies by monitoring ROI data in real time.