94 results about "Distributed reasoning" patented technology

The invention relates to the technical field of artificial intelligence, in particular to a multi-mode agent RAG-ReAct double-engine cooperative training method. The method comprises the following steps: acquiring multi-modal data, and converting the multi-modal data into a high-dimensional vector; constructing a knowledge graph based on the high-dimensional vector; encoding the semantic relationship of the knowledge graph into a model fine adjustment gradient direction by using a dynamic distillation technology; designing a distributed architecture based on the high-dimensional vector; performing hybrid retrieval based on a distributed architecture to obtain hybrid retrieval data; executing distributed reasoning based on a preset recursive reflection mechanism and the mixed retrieval data to generate distributed reasoning data; performing data parallel detection according to the distributed reasoning data to obtain data parallel parameters; and performing node resource scheduling based on the data parallel parameters so as to obtain node load balancing data. Based on the artificial intelligence technology, the reasoning accuracy and the resource utilization rate of the multi-modal agent in a complex task environment are effectively improved.

The invention relates to an industrial Internet of Things cloud edge collaboration method based on distributed reasoning, and the method comprises the steps: segmenting a deep learning reasoning model into a plurality of sub-models according to a reasoning task, and packaging each sub-model into a lightweight WASM model through format conversion, operator detection and structured pruning in combination with containerized deployment and WasmEdge operation, the memory and CPU occupation of the edge node is obviously reduced, the loading and reasoning time delay is shortened, and the network and computing resource utilization rate is improved; and then, distributing each Docker container mirror image to each edge node according to a resource distribution strategy to perform distributed reasoning, thereby realizing multi-node load balancing and reliable scheduling, and improving the throughput and the overall reasoning rate of the industrial Internet of Things.

The invention relates to the field of artificial intelligence, and provides a distributed reasoning method, device and equipment based on a hybrid expert architecture, and a medium, and the distributed reasoning method based on the hybrid expert architecture applied to an edge computing power gateway comprises the steps: determining a plurality of target intelligent vehicles participating in a reasoning task if the reasoning task is received; obtaining an expert reasoning model, and deploying the expert reasoning model to each target intelligent vehicle; wherein the expert reasoning model comprises a plurality of expert reasoning modules, and at least one expert reasoning module is deployed in each target intelligent vehicle; sending the reasoning task to each target intelligent vehicle, so that each target intelligent vehicle executes the reasoning task based on the expert reasoning module to obtain a reasoning result; and aggregating the reasoning results of the target intelligent vehicles to obtain a target reasoning result. Through the technical scheme provided by the invention, the computing power resource of the idle intelligent vehicle is fully utilized.

The invention provides a distributed reasoning task allocation method for an edge computing large model, which belongs to the technical field of mobile edge computing and comprises the following steps of: considering a large language model distributed deployment scene, constructing a workflow for cooperatively finishing a reasoning task by a plurality of edge servers, and adopting accurate secondary reconstruction by considering the generation type characteristic of the reasoning task; the uncertainty of reasoning task resource occupancy is solved, mathematical model expression is constructed for the edge reasoning task allocation problem in the scene, the aim is to maximize the service provider income, the constructed mathematical model is reconstructed into a combination selection problem, an approximate solution with theoretical guarantee is given by adopting a dual method, and the problem of edge reasoning task allocation is solved. And the reasoning task distribution decision of the service provider is optimized, so that the income of the service provider is maximized.

The invention relates to the technical field of large models, in particular to a large model distributed reasoning acceleration method based on multi-modal feature fusion and dynamic weight optimization, and the method comprises the following steps: S1, carrying out real-time semantic analysis through a reasoning context analysis module embedded in a load balancer to obtain semantic features; s2, calculating a cache adaptation degree according to the semantic features; s3, querying a global cache directory service to obtain a matched node list, and making a decision by using a multi-fusion decision algorithm in combination with various parameters of cache adaptation condition, node load condition and network quality; and S4, deploying a global cache directory service in the load balancing layer according to the decision, maintaining a local cache pool at each computing node, and synchronizing the cache metadata to the global cache directory service in real time. The method solves the problems that when a conventional load balancing strategy is adopted for multi-node deployment, waste of computing resources is easily caused, the overall energy consumption of the system is increased, and the request processing speed is reduced.

The invention discloses a neural network pipeline distributed reasoning acceleration method for a multi-NPU heterogeneous platform, and the method comprises the steps: dividing a neural network model into a plurality of sub-models according to the theoretical calculation time of each operator in the neural network model needing distributed reasoning; generating a first splitting strategy and a second splitting strategy according to the plurality of sub-models; establishing an NPU execution time delay model, and calculating the load when the first splitting strategy and the second splitting strategy are executed on the NPU; according to the load, selecting a better one from the first splitting strategy and the second splitting strategy as a current splitting strategy, and performing fine adjustment on the current splitting strategy through iteration; and respectively deploying each sub-model in the obtained optimal splitting strategy to each NPU, and scheduling by the main control CPU to carry out pipeline distributed reasoning of the input image. According to the invention, a pipeline parallel architecture is formed among multiple NPUs, so that each NPU performs reasoning on different input images in different time units, and parallel computing of multiple NPU cores is realized.

An inference method and apparatus for a large language model, which method and apparatus are used for improving the inference calculation efficiency of a large language model. The method comprises: receiving an inference task, which carries a morpheme number, wherein the morpheme number is used for identifying a morpheme to be subjected to inference calculation, the morpheme comprising one or more sub-words determined on the basis of input text of a large language model (201); on the basis of the morpheme number corresponding to the inference task, querying a global index tree to determine a target inference node, wherein the global index tree comprises sub-trees corresponding to a plurality of inference nodes in a distributed inference cluster, and a sub-tree corresponding to the target inference node is a sub-tree in the global index tree, the matching degree of similarity of which sub-tree with the morpheme number is greater than a threshold, the matching degree of similarity indicating the number of pieces of reusable key-value cache data during inference calculation (202); and on the basis of the target inference node, executing the inference task (203).

The invention relates to the technical field of micro-services, and discloses an Istio-based inference service global activation connection number limiting method, which comprises the following steps: step 1, when a client request is received, extracting label information in a request header, and carrying out label analysis; 2, after label analysis is completed, threshold value query is conducted, and interface response content is obtained; 3, acquiring real-time load data of the target service, and calculating a real-time dynamic threshold value; 4, performing three-level quota verification from the tenant level, the service level and the instance level; and 5, after three-level quota verification, connection processing is carried out, a connection processing result is reported, and closed-loop feedback is carried out. According to the scheme, a global and intelligent connection number limiting mechanism is constructed in a micro-service architecture, the problems of service overload, resource competition and system stability caused by sudden increase of the connection number in distributed reasoning service are solved, and dynamic sensing, intelligent decision making and accurate control of the reasoning service connection number are achieved.

The invention discloses an unmanned aerial vehicle cluster control method and system based on distributed reasoning, and the method comprises the steps: generating a task calculation instruction, and transmitting an own resource condition reporting instruction to all slave unmanned aerial vehicles; determining the number of slave unmanned aerial vehicles needing to cooperate according to the calculated amount of the generated task, the priority and the time delay requirement; evaluating the calculation capability of each slave unmanned aerial vehicle according to the received resource information of the slave unmanned aerial vehicles, and determining the cooperative slave unmanned aerial vehicles; based on a load balancing target, determining a task splitting result and a data splitting result model splitting result, splitting the task into a plurality of sub-tasks according to the task splitting result, and distributing the sub-tasks to corresponding slave unmanned aerial vehicles needing collaboration; receiving an execution result of executing the allocated sub-task by the cooperative slave unmanned aerial vehicle; and receiving a final execution result of the task, and issuing a corresponding execution instruction to the unmanned aerial vehicles in the unmanned aerial vehicle cluster according to the final execution result. According to the invention, adaptability and flexibility are improved.

The invention discloses a hierarchical self-adaptive full block pre-filling scheduling method and system for large language model reasoning, and the method comprises the steps: carrying out the hierarchical portrait analysis of a to-be-served model, and dividing the to-be-served model into partitions with different calculation characteristics according to the calculation intensity and memory access characteristics of each layer; then, making a layering and partitioning strategy based on a partitioning result, allocating a larger partitioning size to a calculation-intensive partition, allocating a smaller partitioning size to a memory bandwidth-intensive partition, and generating a layering and partitioning mapping table; and finally, when the online scheduling is executed, querying the mapping table according to the request processing progress to determine the block target size, and jointly forming a batch processing unit by the decoding task and the pre-filled block with the heterogeneous size under the constraint of the iteration time budget to be executed. According to the method, accurate matching of calculation and bandwidth resources is achieved, the system throughput can be effectively improved, tail delay and fluctuation thereof can be remarkably reduced, bubbles under pipeline parallelism are reduced, and the method is suitable for various attention mechanisms and distributed reasoning architectures.

The embodiment of the invention discloses an inference method and device for a large language model, which are used for improving the inference calculation efficiency of the large language model. The method comprises the steps that an inference task is received, the inference task carries morpheme numbers, the morpheme numbers are used for identifying morphemes to be subjected to inference calculation, and each morpheme comprises one or more sub-words determined based on an input text of a large language model; a global index tree is inquired based on morpheme numbers corresponding to the reasoning tasks, target reasoning nodes are determined, the global index tree comprises sub-trees corresponding to multiple reasoning nodes in a distributed reasoning cluster, and the sub-trees corresponding to the target reasoning nodes are sub-trees with the similarity matching degree with the morpheme numbers larger than a threshold value in the global index tree; the similarity matching degree indicates the number of reusable key value cache data in reasoning calculation. And executing the reasoning task based on the target reasoning node.

The embodiment of the invention provides a large model distributed reasoning acceleration method and device based on speculation sampling, and relates to the field of artificial intelligence. And inputting the terminal prefix information into a draft small model for content generation to obtain a plurality of candidate tokens and corresponding small model probability distributions, and transmitting the parameter sequence, each candidate token and a small model probability value corresponding to the candidate token to a base station, so that the base station generates a large model probability distribution of each candidate token by using a target large model, and the candidate tokens are divided into accepted tokens and / or rejected tokens. And if the terminal receives the large model probability distribution corresponding to the first rejection token, resampling the first rejection token to obtain a resampling token, writing the resampling token into the parameter sequence, updating terminal prefix information according to the resampling token, and performing iterative execution until a reasoning result is obtained. The rejection / receiving process of the candidate tokens is executed on the base station, and the resampling process after rejection is executed on the terminal equipment, so that the cooperative reasoning efficiency is improved.

The embodiment of the invention provides a multi-modal model distributed reasoning optimization method for a dynamic intelligent internet of things. The method comprises the following steps: constructing a multi-modal model distributed reasoning optimization system for the dynamic intelligent internet of things; the system comprises a plurality of Internet of Things devices for generating and uploading DNN reasoning tasks and a plurality of servers for receiving and processing the DNN reasoning tasks, determining a DNN reasoning task data parallel scheduling strategy for the system; determining system reasoning time and system energy consumption by using a DNN reasoning task data parallel scheduling strategy; according to the system reasoning time and the system energy consumption, establishing a minimum long-term time delay under the constraint of long-term energy consumption, and taking the minimum long-term time delay as an optimization target to establish a long-term distributed reasoning stability optimization model; and solving the model by adopting a heuristic algorithm to realize distributed reasoning optimization of the multi-modal model. In this way, distributed reasoning tasks of the multi-modal model can be effectively managed, and reasoning time and energy consumption can be optimized.

The invention provides a cloud edge collaborative diffusion model reasoning method and system based on a block chain and reinforcement learning, and the method comprises the steps: obtaining a current text prompt word submitted by a user in a block chain network composed of a cloud server and an edge server, and calling a semantic matching model through an intelligent contract, and judging whether a historical intermediate result can be reused or not; obtaining a server environment state, generating a collaborative reasoning strategy by using a pre-trained multi-agent attention actor-commentator model, and determining cloud edge denoising step number distribution; if the image cannot be reused, the cloud server executes partial denoising to generate an intermediate result and stores the intermediate result to the block chain, and the edge server continues to complete residual denoising to generate a final image; and if the edge server can be reused, the edge server directly completes denoising based on the historical intermediate result. According to the method, historical intermediate results can be effectively reused, dynamic intelligent task allocation is realized, the diffusion model reasoning efficiency and the image generation quality are improved, and the credibility and the traceability of a distributed reasoning process are guaranteed.

The invention belongs to the technical field of intelligent document processing, and particularly relates to a large model information extraction and structure restoration system for a long text document. The system comprises an extraction point defining and text preprocessing module which is used for carrying out deep preprocessing and analysis on an input unstructured document, extracting text and coordinate information and identifying and protecting a special structure; meanwhile, an intelligent dynamic blocking strategy based on semantic boundaries is adopted, a dynamic overlapping mechanism is combined, and an input document is divided into text blocks keeping semantic integrity; the high-concurrency processing and scheduling control module is used for processing the text blocks in batches, calling a large language model to carry out distributed reasoning and outputting a dispersion result; and the extraction reasoning and result fusion module is used for performing semantic deduplication, entity alignment and confidence fusion on the dispersion result returned by the large language model to generate globally consistent structured output. The method has the advantages of being high in precision, high in speed, controllable in cost and high in robustness.

Apparatus and methods to perform vertical federated learning (VFL) in a network, e.g. 5G New Radio network. At step 13, a VFL server, such as a Network Data Analytics Function (NWDAF) hosting VFL functionality, receives a message from a service consumer such as a consumer network function (NF). The message may be a subscription to analytics or a request for analytics. At step 14, a distributed VFL inference process is triggered by the VFL server invoking a Machine Learning (ML) model inference request. The request is transmitted to a participant, i.e. a VFL client entity which may also be a NWDAF. At step 15, based on the request, the VFL client entity obtains intermediate inference results by performing local inference computation using a local ML model. At step 16, if the VFL client entity is the last participant, the intermediate inference results are transmitted to the VFL server. At step 17, the VFL server performs further inference computation on the received intermediate inference results and derives the requested analytics which may be delivered to the NF consumer at step 18. The VFL server may discover or select the VFL client entities to participate in the VFL procedure.

The invention discloses a distributed large model reasoning optimization method and device based on dynamic micro-batch scheduling, and the method comprises the steps: (1) initializing a system, and building a large model distributed reasoning assembly line; (2) evaluating the computing power and the network state of each node and summarizing the computing power and the network state to a head node; (3) determining the number of Micro-Batches and the scheduling quota of each Micro-Batch according to the request distribution condition, the computing power of each node and the network state; and (4) sequentially scheduling the Micro-Batch by adopting a Control Batch strategy and a Chunked Prefill strategy, and starting to execute the Micro-Batch. According to the method, by dynamically adjusting the number of Micro-Batch, the serious assembly line cavitation problem in a distributed large model reasoning system is effectively solved, the GPU utilization rate and the system throughput are remarkably improved, and meanwhile key indexes such as TTFT (first token time delay) and TPOT (inter-token time delay) in the large model reasoning field are also improved. The method has good adaptability, can adaptively adjust the dynamic scheduling strategy under different hardware equipment, network conditions and request loads, is suitable for different distributed large model deployment scenes, and has wide application value.

The invention relates to the technical field of road detection and maintenance, in particular to an expressway disease detection system and method based on integrated learning, and the system comprises a data collection module, a model training module, a model deployment module, a real-time detection module, a result integration module and a user feedback module. Based on a deep learning detection model and a model integration method, the strong learning ability of the deep learning model is specifically utilized, and original data and enhanced data of multiple scenes and multiple time periods are used for training the model, so that the model has relatively high generalization performance and can keep stable detection performance during actual detection, and during actual deployment calculation, the detection efficiency is greatly improved. A plurality of reasoning models are deployed at the cloud simultaneously, multi-model parallel reasoning is realized by distributing reasoning tasks to different models, and finally multi-model reasoning results are summarized and integrated, so that the model reasoning efficiency is improved, the vulnerability of a single deep learning model is effectively slowed down, and the detection effect is more stable.

The invention discloses a large model reasoning method and device based on cross-domain edge device collaboration, and the method comprises the steps: dynamically sensing the available resource condition of a graphics processor in a plurality of working nodes based on a distributed hash table, and distributing a large model block to the working nodes according to the available resource condition; the available resource condition represents an unallocated working node set; receiving a user reasoning request, wherein the reasoning request at least comprises one or more reasoning tasks; based on the large model blocks allocated by the working nodes, allocating the reasoning tasks to one or more management nodes; an area where each management node is located comprises a working node set; the management node distributes the reasoning task to the working node set of the region for reasoning; the working node set comprises a plurality of working nodes. By means of the method and device, the efficiency of the distributed reasoning task can be improved, and flexible allocation of resources and high reliability of the system can be achieved.

What is disclosed is a system to implement distributed reasoning for autonomous operation in a team of members, wherein the team of members comprise a first member, wherein the first member comprises a first team autonomous subsystem and first member autonomous subsystem coupled to each other, the first member executes a first hierarchical plan comprising a team plan and a first member plan, wherein the team plan is related to one or more team goals, and the first member plan is related to one or more self-goals for the first member, further wherein the execution comprises the first team autonomous system executing the team plan, and the first member autonomous system executing the first member plan.

The invention discloses a cross-GPU video memory dynamic scheduling system and method based on equipment demand perception, belongs to the field of artificial intelligence of computer science, and is suitable for a distributed reasoning scene of a large language model. The method comprises the following steps: aiming at the problems of unbalanced utilization of video memory resources and cross-device communication bottleneck in a PD separation architecture, a system constructs a video memory exchange area in a pre-filling instance, when the video memory occupation of a decoding instance is critical, part of KV cache is migrated to the exchange area and returns when the resources are allowed, so that the idle video memory resources of the pre-filling instance are fully utilized, and the resource utilization rate of the decoding instance is improved. And the video memory pressure in the decoding stage is relieved. The system realizes the conversion from the low-efficiency KV cache migration process transferred by the original CPU to the GPU direct access mode through a high-speed communication path established by direct connection between the GPUs. A layered asynchronous transmission strategy is adopted in the migration process, communication overhead is hidden in a calculation assembly line, migration delay is reduced, and cross-GPU video memory sharing multiplexing and efficient scheduling are achieved.

Approaches presented herein provide for the management of resources to be used to process a request, such as may involve orchestration of nodes for an inference request. Upon receiving an inference request, an orchestrator can determine a sequence of context nodes and inference nodes to be used to process the inference request, based in part upon a determined class of inferencing to be performed. The orchestrator can append metadata to the inference request that identifies the sequence, and can transmit the appended request to one or more first nodes in the sequence. If the nodes have a network programmable device, or similar capability, the request can be forwarded to the nodes in sequence without having to go back to the orchestrator between nodes.

The invention discloses a set communication unloading method, system and device and a medium, and is applied to the technical field of computers, and the method comprises the steps: in a model deployment stage, a distributed reasoning controller generates a communication primitive blueprint based on a computational graph description file of a tensor parallel reasoning model and issues the communication primitive blueprint to a DPU; the DPU establishes a hardware-level communication context semantic environment based on the communication primitive blueprint; in the model reasoning stage, the GPU / NPU sends a trigger signal to the DPU when calculating to a communication boundary; and the DPU executes a DMA data pulling assembly line and an RDMA data sending assembly line in parallel based on a hardware-level communication context semantic environment, performs aggregation calculation on all tensor fragment data to be synchronized, and writes an aggregation calculation result back to the GPU / NPU. A hardware-level communication context semantic environment is established in advance to a model deployment stage, and double assembly lines are executed in the DPU in parallel, so that end-to-end communication delay is remarkably reduced, and zero participation of a host CPU is realized.

The invention discloses distributed reasoning techniques, and particularly discloses devices, systems, and techniques for executing neural networks. In at least one embodiment, a processor causes information to be distributed to a processing core. In at least one embodiment, the processor is configured to distribute inference of two or more contiguous portions of information only between two or more corresponding processing cores.

A distributed inference communication method of a super-large-scale graph neural network comprises the following steps: dividing a global topology to be processed into a plurality of local sub-domains matched with the number of GPUs, and determining a local master node and a boundary master node allocated to each GPU; calculating coloring priorities of nodes, and performing parallel graph coloring based on the coloring priorities to construct a global independent set sequence; mapping the global independent set sequence to the local sub-domains of the GPUs to generate a local task slice sequence, which comprises boundary node slices and internal node slices, evaluating communication load and calculation load of the slices, and taking the internal node slices as calculation fillers to be executed in the communication waiting period of the boundary node slices; and each GPU executes an inference task by using a multi-stream asynchronous pipeline constructed locally according to the local task slice sequence. The method can improve the throughput of distributed inference.

The invention provides a distributed cluster construction method, a distributed reasoning method and a resource scheduler, which can be applied to the technical field of computers. The distributed cluster construction method comprises: in response to a submission operation for a system arrangement file, binding a virtual node created based on the system arrangement file to a physical node equipped with a hardware accelerator, the system arrangement file indicating an expected construction state for a distributed cluster; calling a plurality of physical nodes, pulling a target reasoning mirror image from a mirror image warehouse, and starting respective container instances of the plurality of physical nodes by using the target reasoning mirror image, the target reasoning mirror image being obtained by packaging a development kit of a hardware accelerator of the physical node and a distributed reasoning framework; calling an accelerator plug-in of the physical node, and allocating resources of a hardware accelerator on the physical node to the container instance; and constructing a distributed cluster according to the plurality of container instances.

The embodiment of the invention provides a task reasoning method and device and electronic equipment. The method is applied to a distributed reasoning architecture, the distributed reasoning architecture comprises a first node, a second node and a shared memory pool, and the method comprises the steps that the first node obtains at least one first to-be-reasoned task; the first node retrieves the first to-be-reasoned task to obtain at least one first retrieval result related to the first to-be-reasoned task; the first node stores the at least one first retrieval result in a shared memory pool; the second node reads at least one first retrieval result from the shared memory pool; and the second node takes the at least one first to-be-reasoned task and the at least one first retrieval result corresponding to the first to-be-reasoned task as input of a model, and executes the first to-be-reasoned task by utilizing the model. By adopting the mode, the delay problem caused by cross-node transmission can be avoided, so that the overall efficiency of the reasoning task is improved.

The invention provides a model distributed reasoning test system based on multiple machines and multiple cards. The system comprises a model reasoning test middle table, a server matching module, a test environment construction module, a reasoning test module, an index evaluation module and a test efficiency evaluation module. Obtaining a target multi-card server; constructing a multi-machine multi-card environment; starting a distributed reasoning test in a multi-machine multi-card environment to obtain model reasoning indexes of each model test strategy in different reasoning test stages; performing evaluation according to the model reasoning indexes of each model test strategy in different reasoning test stages to obtain a calculation speed-up ratio curve of each model test strategy in different reasoning test stages; and performing test efficiency evaluation according to the calculation speed-up ratio curve of each model test strategy in different reasoning test stages to obtain an optimal test strategy. According to the invention, the test efficiency of the model in the distributed environment is improved.

The application discloses a VLLM-based heterogeneous cluster distributed reasoning cooperation method and system, and the method comprises the following steps: S1, generating a heterogeneous device performance image signal through a performance probe arranged in each computing unit in the cluster; S2, performing non-uniform task division according to the heterogeneous device performance image signal and combining a calculation graph structure of a to-be-reasoned model to form an optimal task scheduling strategy signal for a heterogeneous environment; S3, generating a task allocation and loading instruction signal based on the optimal task scheduling strategy signal; S4, coordinating each heterogeneous computing unit to perform reasoning calculation in a distributed reasoning execution process to generate a distributed cooperative reasoning signal; and S5, integrating and post-processing the distributed cooperative reasoning signal to output a final reasoning result. The VLLM-based heterogeneous cluster distributed reasoning cooperation method and system can solve the problems of low resource utilization and poor reasoning efficiency when a large model is run on a heterogeneous cluster.

The disclosure pertains to the field of data analysis and services, specifically to an online inference method, service system, and device based on an artificial intelligence geospatial data cube. The method involves constructing a cube organizational model based on a spatiotemporal grid, where the cube organizes and manages geospatial data and GeoAI models in a unified manner. It performs task-oriented model matching through a combination of explicit and implicit matching, where explicit matching converts user inference requests into multidimensional query conditions to retrieve candidate models, and implicit matching determines the optimal model by computing the feature similarity between inference data tiles and candidate models. Based on a distributed inference framework, an efficient inference workflow is executed in parallel across multiple computing nodes. The disclosure aims to address the limitations of existing technologies in multi-scenarios or large-scale spatial scenarios, such as low inference accuracy and slow inference speed.