68 results about "Computer cluster" patented technology

An end-cloud cooperative data mining method, comprising: determining a target text and a task configuration file according to a business requirement by a cloud end; encoding the target text by a text encoder to obtain a text feature; placing the text feature in the task configuration file and issuing it to a vehicle end together; encoding image data by a first picture encoder to obtain an image feature by the vehicle end; calculating a value of a similarity of the text feature and the image feature; determining a target picture according to the value of the similarity and the task configuration file; and uploading the target picture to the cloud end; wherein the first picture encoder is obtained by compressing and optimizing a second picture encoder; and the text encoder and the second picture encoder are two modules of a picture-text multimodal large model. The application is applied to an automatic driving shadow mode, can complete data mining of any interesting target text by using one large model, and does not need to design and develop detection rules for each type of interesting target. The picture encoder of the vehicle end can be continuously updated and optimized, and the model iteration efficiency is improved.

The invention discloses a computer cluster security protection method and system. The method comprises the following steps: collecting multi-dimensional operation data in real time through an agent program deployed at each node; feature parameters are extracted from the data, and a safety evaluation model is adopted to calculate a safety score and map the safety score into a discrete safety level; gradient protection strategies including basic monitoring, enhanced protection and strict isolation strategies are automatically matched and loaded according to the security level; and monitoring a strategy execution effect, and performing closed-loop optimization on the model parameters based on feedback data. The system correspondingly comprises a data acquisition module, a safety evaluation module, a strategy management module, a strategy execution module and a feedback optimization module. According to the method, the problems of static strategy stiffness, inaccurate risk quantification, single response means and lack of self-learning are solved, dynamic self-adaptive protection is realized, the evaluation accuracy is improved, false alarms and missing alarms are reduced, and the cluster environment is ensured to be safe and reliable.

The invention discloses a cluster computing power configuration management and automatic deployment system, which belongs to the technical field of computer cluster management, and comprises a resource state tracking module used for monitoring resource state information of each computing node in a cluster in real time and establishing a resource pool, an exclusive resource pool and a shared resource pool according to the resource state information, the resource pool comprises a first idle node; the resource allocation arbitration module is used for generating a second idle node and a third idle node according to the first idle node; the task resource matching module is used for screening a matched third idle node from the shared resource pool; and the resource recovery coordination module is used for recovering the low-utilization-rate computing node and the first idle node, the second idle node and the third idle node which have completed the computing task into a resource pool. The multi-level resource pool is established through the resource state tracking module, the problem of rigidity of resource allocation of a traditional system is effectively solved, and the computing resource allocation flexibility and utilization rate are remarkably improved.

A method, computer system, and a computer program for managing computer jobs in a queue is provided. This comprises extracting metadata from a new job received for processing and upon determining when a similar enriched metadata exists in a database. A job score and storage footprint may then be determined for the new job from the extracted metadata. It is then determined whether the new job can be grouped for processing with any other jobs already placed on a queue. The new job is then added to the queue based on the new job's score and footprint, and whether it can be grouped with other jobs. The queue is then updated and sent to a scheduler for further processing.

The embodiment of the invention discloses a computer cluster parallel processing method for a reverse time migration program, which comprises the following steps that: a main node carries out parallel processing according to a list file containing to-be-processed shot information in a global disk, a flag file in a first global file directory and a flag file in a second global file directory; determining a specified gun and distributing a gun data processing task corresponding to the specified gun to the computing node; the first global file directory is used for storing mark files of shots which are subjected to reverse time migration processing on each computing node, and the second global file directory is used for storing mark files of shots which are subjected to reverse time migration processing on each computing node; and the computing node responds to the received shot data processing task, calls a single-shot reverse time migration program to execute reverse time migration processing for the corresponding specified shot, and updates the flag file in the first global file directory and / or the second global file directory according to the processing state. In this way, the processing efficiency of the multi-shot reverse time migration task can be improved.

The invention relates to the technical field of computer cluster communication, in particular to a set communication method suitable for a core particle equipment cluster with unified bus interconnection, which comprises the following steps of: acquiring hierarchical topology and link information of the core particle equipment cluster, and constructing a global topology information base according to the acquired hierarchical topology and link information; optimizing the core size level primitive, the node level primitive and the super node level primitive; a link monitoring unit is deployed to monitor a link state index in real time, an alarm is triggered when a link fault is detected, an optimal backup link is screened based on a global topology information base, a communication path is updated, and a dynamic resource scheduling algorithm is adopted to allocate computing power and communication resources; and dynamically adjusting a communication path and bandwidth allocation according to a set communication task requirement and a link dynamic parameter. The method is suitable for a three-level link architecture of a core particle equipment cluster, can improve the communication efficiency and the bandwidth utilization rate, enhances the cluster reliability and the resource utilization rate, and effectively meets the requirements of high computing power and high communication intensity of large model training.

Provided is a processing system that executes a process flow in which computation results are aggregated after data is divided and processed. This processing system comprises: a computer cluster 10 that is composed of a plurality of computation units which each process divided data; a DPU cluster 20 that is composed of at least one DPU which aggregates computation results; and a cross-connect switch 30 that connects devices. A setting device enumerates process flows which can be established by the processing system while changing the number l of interfaces of a computation unit to be used and the number m of DPUs to be used, combines the process flows such that data communication time in the processing system is minimized, and sets the cross-connect switch 30 such that a combination of the process flows can be executed.

In some embodiments, a computer cluster system comprises a plurality of nodes and a software package comprising a user interface and a kernel for interpreting program code instructions. In certain embodiments, a cluster node module is configured to communicate with the kernel and other cluster node modules. The cluster node module can accept instructions from the user interface and can interpret at least some of the instructions such that several cluster node modules in communication with one another and with a kernel can act as a computer cluster.

The application provides a resource fragment consolidation method, a controller, a control node and a computer cluster, and relates to the technical field of data processing. The method simulates resource scheduling according to a first resource capacity and a first task currently being executed to obtain a plurality of candidate resource scheduling paths; determines a target resource scheduling path from the plurality of candidate resource scheduling paths; and schedules the first task based on the target resource scheduling path to perform resource fragment consolidation. After each candidate resource scheduling path is used to perform resource fragment consolidation, the residual resource capacity of at least one node can reach the first resource capacity. That is, the method can actively eliminate resource fragments in the computer cluster by scheduling the first task to perform resource fragment consolidation. Compared with the method of slowing down the generation of new resource fragments from the source, the scheduling reliability of a task with high resource demand can be effectively improved.

The application relates to the technical field of data processing, and discloses a computer cluster data processing method and device, the method comprises the following steps: acquiring cluster data, wherein the cluster data is any one of continuous storage data and dispersed storage data; performing chain layer storage and analysis on a computer cluster according to the cluster data to obtain a cluster data block; wherein the computer cluster refers to a computer executable program running on a cluster storage structure; performing vectorization coding according to the cluster data block to obtain a data block vector; performing mapping according to the data block vector to obtain mapping data which can be operated on a computer; and inputting the mapping data into a data calculation formula to perform calculation processing, so as to obtain a processed data linked list. The method can solve the problem of low data processing efficiency in the prior art due to the limited capacity of structured modeling for mass data processing.

The application discloses a dual-computer high-availability implementation method and device, equipment and a medium, relates to the technical field of computers, and is applied to a Keepalived service tool of any service node in a dual-computer cluster, and comprises the following steps: acquiring a current node state of a local service node; determining a current script corresponding to the current node state from a plurality of preset scripts; wherein the plurality of preset scripts comprise a first preset script corresponding to a node state being a master node state, a second preset script corresponding to the node state being a backup node state, and a third preset script corresponding to the node state being a Keepalived service hanging state; and executing the current script. When the node state is the master node state, the backup node state and the Keepalived service hanging state respectively, the corresponding current script is automatically executed, so that the operation and maintenance of the local service node is implemented, and the high availability and stability of the dual-computer cluster can be ensured.

A device-cloud collaborative data mining method includes: A cloud determines a target text and a task configuration file based on a service requirement; encodes the target text by using a text encoder to obtain a text feature; and places the text feature in the task configuration file and delivers the task configuration file to a vehicle. The vehicle encodes image data by using a first picture encoder to obtain an image feature; calculates a value of a similarity between the text feature and the image feature; determines a target picture based on the value of the similarity and the task configuration file; and uploads the target picture to the cloud. The first picture encoder is obtained by compressing and tuning a second picture encoder, and the text encoder and the second picture encoder are two modules of a large image-text multimodal model. This application is applied to an autonomous driving shadow mode, and data mining of any target text of interest can be completed by using a large model, without designing and developing a detection rule for each type of target of interest. In this way, a picture encoder on the vehicle may be continuously updated and optimized, to improve model iteration efficiency.

A plurality of different virtualization system images are configured for deployment to a plurality of nodes in heterogeneous environments. Individual ones of the virtualization system images are configured such that once deployed, the nodes form a computer cluster having a storage pool that is shared across the nodes. When configuring the virtualization system images, information that describes the heterogeneous computing environments is accessed, and constraints pertaining the heterogeneous computing environments are reconciled in advance of configuring the different virtualization system images. A common subnet across the heterogeneous environments is established. The plurality of different virtualization system images are configured to access the common subnet once deployed. The common subnet serves as a storage I / O communication path over which a cluster-wide storage pool is implemented. The virtualization system images are configured to correspond to address portions of a contiguous address space that is used to access data in the storage pool.

The embodiment of the invention provides a distributed query method and related equipment. The distributed query method comprises the following steps: in response to an SQL task submitted by a submitting end, splitting the SQL task to obtain a plurality of SQL sub-tasks; executing each SQL sub-task in a distributed manner through each cluster node to obtain a corresponding sub-task data result, the cluster nodes being nodes of the computer cluster; and aggregating the subtask results to obtain a final query result. According to the technical scheme, the SQL query task submitted by the submitting end is intelligently split, the complex SQL query task is decomposed into a plurality of subtasks capable of being processed in parallel, and the query speed is remarkably increased through the parallel computing capacity of a computer cluster. When the subtasks are executed, the split SQL subtasks are distributed to a plurality of computing nodes to be executed, parallel execution mechanisms of some cluster frames can process a plurality of tasks at the same time, and therefore the overall processing time is shortened.

The invention discloses a code optimization method and device, and relates to the technical field of computers. The code optimization method comprises the steps that a computing device obtains a code comprising at least one first loop statement, then determines a data division mode according to a dependency relationship among data indicated by the first loop statement, and converts the first loop statement into a second loop statement according to the division mode to obtain an optimized code. Wherein the division mode comprises one or a combination of the following modes: slice data division used for indicating to divide the data in a single dimension, and block data division used for indicating to divide the data in any multiple dimensions. The second loop statement can be processed in parallel by at least two processes on a computer cluster, and different processes are used for respectively processing slice data or block data obtained after data division, so that the data division efficiency is improved, and the development efficiency of parallel codes is improved.

In some embodiments, a computer cluster system comprises a plurality of nodes and a software package comprising a user interface and a kernel for interpreting program code instructions. In certain embodiments, a cluster node module is configured to communicate with the kernel and other cluster node modules. The cluster node module can accept instructions from the user interface and can interpret at least some of the instructions such that several cluster node modules in communication with one another and with a kernel can act as a computer cluster.

The application provides a data processing method and device, equipment, a storage medium and a program product. The method comprises: obtaining a first logical view of an input first tensor, wherein a first logical dimension arrangement of the first logical view comprises a batch dimension, a sequence dimension and a hidden state dimension in sequence; performing first rearrangement of the first logical view based on a second logical dimension arrangement to obtain a second logical view of the first tensor, wherein the second logical dimension arrangement comprises the sequence dimension and a merged dimension in sequence, and the merged dimension is a product of the batch dimension and the hidden state dimension; and performing set communication on the first tensor in the sequence dimension based on the second logical view. The application can reduce the memory usage of data processing in a computer cluster, improve the speed of data processing, and improve the performance of the computer cluster.

The invention discloses a multi-modal high-speed data storage method for a potassium collecting machine, which relates to the technical field of data storage, and comprises the following steps: selecting key value storage according to the storage demand of multi-modal data of the potassium collecting machine, and defining the structure of a key matched with the type of the multi-modal data, each multi-modal data item needing to be stored has a unique corresponding key; uniformly distributing the multi-modal data items on different nodes in a computer cluster matched with the potassium collecting machine by using a consistent Hash algorithm so as to realize load balancing; after the multi-modal data item is allocated to the target node, writing an operation log record of the multi-modal data item into a disk; writing the multi-modal data item into a memory cache; and finally, flashing the multi-modal data in the memory cache into a disk according to a preset period. The number of the virtual nodes can be intelligently adjusted according to the change of the system, so that better load balancing and system performance are realized, and the flexibility, the stability and the reliability of the system are improved.

A process receives a batch of one or more first job requests to be executed by a high-performance computing (HPC) cluster. This batch of first job requests is received by a container orchestration platform. The batch of one or more first job requests is translated into one or more second job requests. The second job requests are interpretable by a scheduler corresponding to the HPC cluster. The second job requests are then sent to the scheduler.

The application relates to a cluster type safety computer platform, which is centrally deployed, can realize centralized maintenance and is favorable for improving system availability. The safety computer platform is deployed on a track side and does not need to bear the severe working environment of a vehicle-mounted device, is high in reliability and can adopt high-performance hardware, thus being favorable for realizing more intelligent functions. The safety computer units in the safety computer platform communicate based on a wired network, and compared with the wireless communication between a vehicle-mounted safety computer platform and a track-side safety computer platform in a traditional train control system, the wired communication is better in reliability. A master and standby database cluster is introduced, the statelessness of safety applications can be realized, data synchronization during master and standby switching of the safety applications is avoided, the switching speed is accelerated, and the consistency of data during the master and standby switching process is ensured. The large-scale safety computer cluster platform is realized, and the problems existing in real-time performance and determination of virtualization on the safety computer platform do not need to be faced.

A system for configuring an adaptive computer cluster is disclosed. The system includes a cluster configuration server communicatively coupled to a cluster hosting environment through a network, the cluster configuration server having a processor and a memory. The memory includes a plurality of inert containers and a configuration tool configured to receive at least one procedure having a trigger event and at least one task and further configured to instruct the cluster hosting environment to instantiate the adaptive computer cluster based upon the at least one procedure and using the plurality of inert containers. The instantiation instructions include instructions to instantiate, within the cluster hosting environment, an API gateway container, a storage container, a distributed computing master node container, at least one solution-specific container, and an orchestrator container. The instructions from the configuration tool further include instructions to communicatively couple all of the containers to the orchestrator container.

The embodiment of the application provides a database operation method, system and computer device, the method is applied to the cloud platform, and comprises the following steps: a coordination node receives a data operation request sent by a user end through a target session; the coordination node determines whether a session temporary table created for a global temporary table and belonging to the target session exists at present, wherein metadata corresponding to the session temporary table is stored in a disk of a data node; in the case that it is determined that the session temporary table does not exist, the coordination node creates the session temporary table in a target data node in a plurality of data nodes, and sends the data operation request to the target data node; in the case that it is determined that the session temporary table exists, the coordination node sends the data operation request to the target data node, and the data operation request is used to instruct the target data node to perform data operation on the session temporary table according to the target data node. By adopting the database operation method provided by the application, the stability of the distributed database can be improved, and resource waste can be avoided.

A distribution device stores, in a storage unit, remaining capacity information of a protected area received from each of a plurality of host computers forming a computer cluster. Each time a processing request is received from one of the plurality of terminals, requesting execution of processing using an encrypted protected area on the memory of the host computer, the distribution device refers to the remaining capacity information of each of the plurality of host computers stored in the storage unit in accordance with the processing request to select a destination to which the processing request is to be distributed.

A system and method of processing datasets that are compliant to a layer 2 (L2) protocol of an open systems interconnection (OSI) model in a cloud computing environment on behalf of an edge computing environment. The method includes receiving, from the edge computing device, a set of L2 frames. The method includes detecting one or more errors in the set of L2 frames based on an L2 protocol of the OSI model. The method includes correcting the one or more errors in the set of L2 frames to generate a corrected set of L2 frames. The method includes generating, by a processing device of a cloud computing system, a processed dataset based on the corrected set of L2 frames.

The application discloses a kind of based on loop current method's power electronic system cluster simulation implementation method, it is related to power electronic technical field.The application includes: based on loop current method's power electronic model cluster distribution initialization method and cluster computing scheduling method, wherein model distribution initialization method will large-scale power electronic model be distributed to each computing node according to the local circuit after decoupling of loop current method and complete the initialization of local circuit model simulation on each node, wherein cluster computing scheduling method realizes the multi-level parallel computing scheduling and synchronous process of main node to computing node in cluster.The application is suitable for large-scale power electronic simulation based on supercomputer or high-performance computer cluster, fully utilizes the advantage of large memory capacity of computer cluster to expand the scale of power electronic model simulation, fully utilizes the advantage of many processor core quantity of computer cluster to improve the efficiency of power electronic model simulation.

The invention discloses a large-scale high-throughput task query request processing method and system, and relates to the field of computer cluster scheduling. The large-scale high-throughput task query request processing system comprises a system architecture design module, a core processing mechanism module, a key configuration parameter setting module and a service operation process module. The large-scale high-throughput task query request processing system has the following technical advantages that the scheduling performance is improved through an architecture decoupling and resource isolation mechanism; the performance interference of lock competition, thread switching and frequent I / O operation in a traditional scheme on a scheduler is eliminated through architecture decoupling, the actual measurement scheduling delay is reduced by 40%-60%, the process creation overhead is reduced through a main control process merging request mechanism and maxrequest sonfork parameter regulation and control, the system throughput is improved, the resource utilization rate is optimized, and the system performance is improved. The dual memory cleaning strategy periodically cleans and keeps the timeliness to balance the query real-time performance and memory resource occupation, and the completed task is prevented from residing for a long time.

The invention discloses an optimization and principle system of an ODS management platform, and belongs to the technical field of bank data management. The system comprises a database server, a computer cluster, a data stream processing center, a data access service module, a data processing assembly and an infrastructure assembly, and an ODS serves as a database system center point and is connected with application servers through ODSH online data service. According to the system, a perfect ETL control flow mechanism is established and comprises three key steps of extraction, conversion and loading, and unified management of metadata and configuration information is achieved through SDM, FDM and ADM management libraries of an ODS processing module. Event driving is adopted as a job scheduling core, and various event types such as data file arrival, data stream arrival, database table readiness and service request arrival are supported. According to the method, the system operation quantity can be reduced by more than 40%, the data landing frequency is remarkably reduced, the bank information security requirement is completely met, and the operation efficiency and the management level of the ODS system are effectively improved.