58 results about "High performance cluster" patented technology

The invention discloses a convolution neural network parallel processing method based on a large-scale high-performance cluster. The method comprises the steps that (1) a plurality of copies are constructed for a network model to be trained, model parameters of all the copies are identical, the number of the copies is identical with the number of nodes of the high-performance cluster, each node is provided with one model copy, one node is selected to serve as a main node, and the main node is responsible for broadcasting and collecting the model parameters; (2) a training set is divided into a plurality of subsets, the training subsets are issued to the rest of sub nodes except the main mode each time to conduct parameter gradient calculation together, gradient values are accumulated, the accumulated value is used for updating the model parameters of the main node, and the updated model parameters are broadcast to all the sub nodes until model training is ended. The convolution neural network parallel processing method has the advantages of being capable of achieving parallelization, improving the efficiency of model training, shortening the training time and the like.

The invention relates to a load balance and node state monitoring method in high performance computing. The method is suitable for reducing resources which occupy a clustering system, increasing the utilization rate of the resources, effectively improving the performance of server clustering and providing high-quality service for a user. The method concretely comprises the steps that firstly, according to the operation state and load and performance parameters of server nodes, computing is performed to obtain load weights of all the nodes, and an alternative node set for next task distribution is selected; afterwards, according to load difference values and distribution probabilities, the probabilities of all node distribution tasks in the alternative node set are computed, and new requests are distributed to the selected nodes in a random probability distribution mode to solve the problem of uneven load distribution; finally, based on a load correction formula, node loads of the distribution tasks are corrected to improve the load balance effect and improve the reliability and stability of the cluster system.

The invention discloses a method for building high-performance computing application service based on virtualization technology, wherein the method divides the whole system into four layers: a physical resource virtualization layer, a system resource management layer, an application service deployment layer and a user interaction layer. The physical resource virtualization layer virtualizes the physical host resource by a virtual machine monitor; the system resource management layer is divided into a monitoring module, a control module and a transmission module, manages the system resource by the virtual machine manager interface downwards and provides the system base service upwards; the application service layer receives the user interaction layer requirement upwards, distributes the system resource and deploys the application service; and the user interaction layer provides a convenient usage interface module to the user. The system provides an operation method for building high-performance computing application service, so that the system can manage the virtual host effectively by the prior virtualization technology and capability-ranking strategy dynamically according to user requirement, build high-performance cluster scientific computing application service according to user requirements through fabrication and rapid distribution of virtual devices, provide the usage interface, simplify building steps of the high-performance cluster scientific computing application service and improve the resource utilization rate of the whole system.

A high-performance cluster computing system based on an x86 framework PC platform comprises a first access switch and a second access switch, as well as two servers based on the x86 framework PC platform and a computer cabinet for placing the switches and the servers, wherein each server is equipped with a special business data port and a document data port, which are respectively connected with different switches for transmission of both the business data and the document data; and the business data ports and the document data ports are configured in different network segments for expanding I/O (Input/Output) bandwidth.

The invention provides a method and a device for data collection, which relate to the field of mass data processing and solve the problem of data lag caused by database storage. The method comprises the steps of: collecting data from nodes in cluster; and establishing aggregation in cache, using the collected data as the object of the aggregation and storing the data in the cache. The technical proposal provided by the invention is suitable for high performance cluster.

The invention discloses a large-data clustering algorithm based on a cloud computing platform. Primitive data is pre-processed; data are divided into M sub-data and distributed to M Map functions and local clustering is carried out on the sub-data; clusters with the same key are combined; if the number R of practical clusters is smaller than the number k of clusters, the number c of representative points and a constriction factor a are regulated and clustering is carried out again until termination conditions are achieved. if a new data set is generated, local clustering is carried out according to judgment conditions that the number K of new data source centers is larger than the obtained number K of clusters before updating or the number of new data source points is larger than the number of data source points before updating. According to the large-data clustering algorithm based on the cloud computing platform, the parallel computing capacity of a high-performance clustering system of cloud computing is used for solving the problem that mass data need to be processed in clustering, and therefore a relation of the data can be rapidly and efficiently dug up.

The invention discloses a parallel indexing technology for vector QR trees. Efficient indexing for massive spatial data is particularly important in a novel parallel environment, and the spatial data indexing efficiency is critical in measuring the integral performance of a spatial database. The main node access bottleneck cannot be broken through by an existing parallel indexing technology, and load balance among processes is difficult to realize. In order to solve the problems, the parallel indexing technology for collaboration among vector Q trees and vector R trees includes 1), dividing adjacent spatial data sets into different processes by a multi-channel method and realizing load balance of tasks among the processes; 2), constructing the QR trees on the basis of central points of minimum bounding rectangles of spatial objects, and optimizing indexing paths; 3), performing master-slave storage for the QR trees and optimizing index reading; and 4), collaboratively retrieving the QR trees and breaking the root node access bottleneck. The parallel indexing technology for the QR trees comprises spatial object set division, QR tree construction, master-slave storage and collaborative retrieval for the QR trees. The parallel indexing technology has the advantage that an efficient vector spatial retrieval method can be provided for developing and serving single-unit and multi-core or integrated-core massive data spatial indexing software in a high-performance cluster environment via parallel indexing for the QR trees.

The invention discloses a method and device for realizing a high-performance cluster system for scheduling mirror images. The method comprises the following steps of: downloading a mirror image and storing the downloaded mirror image into a shared path of a management node; obtaining an operation task, and scheduling a computation node according to the operation task so as to generate computation task scheduling; when the computation task scheduling is received, loading the mirror image to the local by the computation node through the shared path; and enabling a host network and a host process to communicate with containers of other nodes through a container. According to the method and device for realizing the high-performance cluster system for scheduling mirror images, the management node provides a mirror image gateway function, the monitored idle computation node loads the mirror image to start the container, and a sub-process is started according to an MPI parent process so as to complete MPI parallel computation processing.

The invention relates to a task scheduling method in the field of high-performance clusters. The task scheduling method comprises the steps: after analysis of each working node of a Hadoop in a job submitting stage, acquiring an actual stage weight according with a task, after processing of a geometric mean method, establishing a reference standard of the stage weight for left tasks of the job; in a task feedback stage, adopting the reference standard for the left tasks of the job, and estimating the task residual execution time by combining with progresses of sub-stages; in a job feedback stage, solving a geometric mean of stage weights of all tasks by using a staging manner, and establishing a job name-stage weight mapping record to be used as a reference of executing subsequent jobs on the node. According to the task scheduling method, self-learning and information feedback can be respectively carried out on the task of each job in a multi-job parallel executing environment to obtain more accurate stage weight estimation, the accuracy of pre-estimating the task residual execution time is improved, the hit rate of selecting dated tasks is increased, and the optimal utilization of cluster resources is promoted.

The invention relates to a high-performance cluster management method based on Slurm, which comprises the following steps of: randomly selecting one machine as a control node and other machines as computing nodes; obtaining host names or IP information of all computing nodes in the cluster, and copying a cluster installation package and an installation script to each computing node; logging in each computing node on the control node through an SSH (Secure Sockets Hierarchy) service, and finishing the construction and deployment of a cluster environment on the node through the installation script; deploying a control receiving process at the control node for monitoring computing resources and receiving information sent by the computing node; wherein a daemon process exists on each computingnode to control the computing nodes in the cluster, regularly acquiring node states and information on the nodes, and sending the node states and the information to the control node through an SSH service; carrying out collaborative management on the computing nodes and the redundant backup nodes; and based on a Slurm job management mechanism and a node state monitoring process, deploying, monitoring and allocating jobs in the queue according to the current condition of cluster system resources.

A method, apparatus and program product simulate a high performance computing (HPC) application environment by creating a cluster of virtual nodes in one or more operating system instances executing on one or more physical computing node, thereby enabling a plurality of parallel tasks from an HPC application to be executed on the cluster of virtual nodes.

The invention discloses a method for realizing system energy saving, and belongs to servers and the technical field of storage and high-performance cluster energy saving. The method structurally comprises a power supply unit and a server mainboard system unit, wherein the power supply unit is connected with the server mainboard system unit and supplies electric power to the server mainboard system unit, and comprises an AC/DC (Alternating Current/Direct Current) conversion module, a DC/DC conversion module, a sampling quantization circuit and a PMBUS (Power Supply Management Bus) module, and the server mainboard system unit is composed of a DC/DC conversion module, a mainboard hardware monitoring chip, a CPU, a north bridge chip, a south bridge chip and other control chips; the DC/DC conversion module of the power supply unit is connected to the DC/DC conversion module of the server mainboard system unit, and the PMBUS module of the power supply unit is connected to the DC/DC conversion module and the mainboard hardware monitoring chip of the server mainboard system unit. Compared with the prior art, the method can reduce the total energy consumption of a device to a certain extent, and has strong practicability.

The invention discloses a vector target set balanced partitioning method aiming at spatial measure and direction relation concurrent computation. The vector target measure (vector target length and area computation) and direction relation (direction relation among vector targets) concurrent computation belongs to a compute-intensive algorithm, i.e., all points of geometric objects in vector target sets participate in the computation during a target measure and direction relation computation process. Therefore, the load balance of tasks in all processes needs give consideration to the number of vertexes of the vector target sets. Therefore, the balanced partitioning method giving consideration to the number of vertexes of the vector target sets is adopted aiming at the vector target measure and direction relation to evenly partition the vector target sets to all the processes, so that the load of the tasks of the vector target sets in all the processes is balanced. By utilizing the vector target set balanced partitioning method aiming at the spatial measure and direction relation concurrent computation disclosed by the invention, the high balance of inter-process computation loads can be realized, and further, the algorithm efficiency is improved. A high-efficiency data partitioning method is provided to the development and services of spatial measure and direction relation software for mass data in a single-machine multi-core or many-core high-performance cluster environment.

The invention discloses a cluster system. The cluster system comprises a service node, cluster nodes and a shared storage node; the service node is used for deploying PXE service; the cluster nodes are used for setting PXE starting as a first starting item, an operation system is obtained through PXE starting, the operation system is made to operate in an internal storage, and an output file operated by the cluster nodes is saved in the shared storage node by mounting a specified directory in the shared storage node; the shared storage node is used for storing output files operated by all the cluster nodes; the cluster system is based on a high-performance cluster framework of an LXD virtualization technique customized system and has the advantages of being easy and convenient to deploy, flexible in management, capable of reducing the cost, high in data security and the like. The invention further discloses a deployment method of the cluster system, and the deployment method has the above mentioned advantages.

A programmable media includes a processing unit capable of independent operation in a machine that is capable of executing 10¹⁸ floating point operations per second. The processing unit is in communication with a memory element and an interconnect that couples computing nodes. The programmable media includes a logical unit configured to execute arithmetic functions, comparative functions, and/or logical functions. The processing unit is configured to detect computing component failures, memory element failures and/or interconnect failures by executing programming threads that generate one or more chaotic map trajectories. The central processing unit or graphical processing unit is configured to detect a computing component failure, memory element failure and/or an interconnect failure through an automated comparison of signal trajectories generated by the chaotic maps.

The invention provides a method for characterizing application software operation characteristics of various industries in the field of high performance computing. The method comprehensively examines the load pressure of application programs in five links of inputting, storage, processing, transmission and outputting, and sequentially divides the applications into four large types of compute-intensive type, memory constraint type, I/O intensive type and network intensive type. Through the quantitative representation of the four aspects, the resource demands of the applications on the aspects of CPU occupation, memory capacity, memory throughput, inputting/outputting and network data exchange are fully represented to maximally reflect the operation characteristics of the application software. The method provided by the invention is simple, practical, reliable and effective, and can directly reflect the quantity demand of one application software on the high-performance hardware resource, so that the applications can be operated on a proper high-performance platform in order to maximally exert the performances of the application software. According to the characteristics, the performance bottleneck of the application software can be improved and broken through in purpose, and the application expandability is enhanced.

The invention discloses a high-performance cluster monitoring method, an apparatus, a device and a storage medium. The method comprises the following steps: acquiring a node to be monitored in the high-performance cluster; collecting performance data of the node to be monitored when the node to be monitored runs target application software; generating an application running characteristic file ofthe high-performance cluster according to the performance data; an offline analysis being performed on the application running characteristic file. By collecting the performance data of each node in real time, the running status of application software in high performance cluster can be monitored in real time, and the performance data can be saved as application running characteristic file while monitoring the performance status of the whole cluster, which can be used for offline performance analysis.

The invention discloses a vector target set balance partition method aiming at topological relation parallel computation. Vector target topological relation parallel computation belongs to the non-computational intensive algorithm, and in parallel computation, main computation resource consumption is to judge whether minimum bounding rectangles among vector targets are intersected or not, but topological computation only occupies a small part of computation resources. Therefore, vector target partition emphasizes to consider balance of the vector target quantity in each progress rather than to give consideration to geometrical complexity of the vector target. Aiming at vector target topological relation parallel computation, data are assigned to each progress by means of efficient balance partition, so that the vector target quantity among the progresses is balanced, namely, task loads are balanced. By the method, the computation loads among the progresses are highly balanced, and accordingly efficiency of the algorithm is improved, and the efficient vector data partition method is provided for development and service of topological relation software for mass data of single-computer multi-core and single-computer many-core high-performance cluster environments.

The invention provides a method for dispatching a HPC (high performance computing) cluster work and a device for dispatching the HPC cluster work. The method comprises the following steps: setting a grouping rule, setting a grouping sequencing rule, and setting conditions for judging the grouping; according to the set grouping rule, dividing all works in the HPC cluster into multiple groups; according to the set grouping sequencing rule, sequencing multiple groups; according to multiple groups after sequencing, orderly polling every group, and taking out one work from groups satisfied with the setting condition to execute; after all groups are polled, repeatedly executing the step until all groups cannot meet the setting conditions; when remaining calculation resources in the HPC cluster are restored until the group can meet the setting condition; sequencing according to the group and taking out one work from the groups satisfied with the setting conditions to execute until all works are finished. The method and the device for dispatching the HPC cluster work can realize the reasonable distribution of the HPC cluster calculation resource.

The invention discloses a load balancing method and a device of a database system based on a high-performance cluster distribution. The method comprises: sorting priority of database operations in a database system; allocating corresponding operation time for database operations of different priorities; creating a table file to generate a task queue and an expired data processing task queue; in the data acquisition cycle, executing the data storage operation, that is, splitting a plurality of sub-table files according to the number of data nodes and the size of the table files, and storing thesub-table files on a plurality of data nodes; executing the table file generation operation in the table file generation cycle; at that expire data processing cycle, performing the expire data processing operation. The invention adjusts the distribution of the table on the database node by taking the data table as an operation unit, monitors the load distribution of the data, and properly makes load balancing adjustment, thereby effectively balancing the load of the database and preventing excessive load of the database.

The invention discloses a method for generating a smooth entity by contour lines. The method comprises the steps of: (1) inspecting and correcting a contour line to be used; (2) transforming the contour line into a Delaunay triangulation network by a Delaunay triangulation algorithm; (3) inquiring a obtuse triangle in the Delaunay triangulation network, and performing grid refinement on the obtusetriangle to convert the obtuse triangle into a Grid matrix grid, then generating a smooth entity. The step (3) specifically comprises: traversing all triangles and establishing indexing geographicalnumbers; when a triangle is an obtuse triangle and an obtuse angle is greater than 120 degree, performing grid refinement on the obtuse triangle and circumference according to the indexing geographical numbers and a row and column matrix; and deleting the obtuse triangle so as to obtain the smooth entity. A solid surface generated in the invention is a locally smooth curved surface, can provide obvious surface features and greatly improves the resource utilization rate and the result convergence rate of the high performance cluster computing of the smooth entity.

The invention provides a virtual cluster processing method and system based on cloud computing, and is applied to the field of cloud computing. The method comprises the following steps: on the basis of physical clusters, creating an empty virtual cluster; moving a new virtual machine or a virtual machine which does not belong to any virtual cluster in the physical clusters into the empty virtual cluster, and building a new virtual cluster; carrying out control and management on the new virtual cluster. Through the implementation of the technical scheme of the invention, a virtual cluster system can build a high performance cluster system platform in a cloud computing environment more quickly and easily, and can more flexibly create virtual computing resources, thereby being very convenient for a user to use.

The present application discloses a method and a system for fault diagnosis of a high-performance cluster. The diagnosis method includes: collecting basic information of each node in the high-performance cluster in a master node of the high-performance cluster; converting the basic information into HTML document format; according to the obtained loading command, loading the basic information in the form of HTML document locally and displayed visually in the form of web interface. According to the contents displayed in the web interface, fault location and fault handling are carried out. The diagnostic system includes: an information collection module, a format conversion module, a display module and a fault processing module. The application adopts script to collect, transform and visuallydisplay the basic information of each node in the high-performance cluster, so that the user can very directly view the status of the high-performance cluster, thereby facilitating the fast positioning of the fault point, and effectively improving the accuracy and efficiency of the fault troubleshooting.