300 results about "Database clustering" patented technology

The invention relates to the technical field of databases, and discloses a multi-type database performance optimization and operation and maintenance management method and system.The multi-type database performance optimization and operation and maintenance management method comprises the steps that real-time performance indexes of a heterogeneous database cluster are collected, and a noise reduction data set is obtained; generating a cross-library performance coupling degree matrix through the resource competition coupling degree and the data dependence coupling degree; constructing a coupling relation graph; identifying a bottleneck node set based on the coupling relation graph and the abnormal level; generating an optimization strategy; and performing strategy conflict identification according to the optimization strategy, generating a processing strategy, and executing the processing strategy optimization strategy. Through the improved weighting centrality algorithm, the core bottleneck node which has the greatest influence on the whole cluster is accurately identified, a differential optimization strategy is adopted according to the database type, a perfect strategy conflict detection and avoidance mechanism is established, mutual interference in the optimization process is effectively prevented, the optimization success rate is improved, and the optimization efficiency is improved. And the optimization time is shortened.

The invention belongs to the technical field of computer software, and particularly relates to a heterogeneous database unified deployment management method based on a cluster management platform, which comprises the following steps: predefining metadata of various database services; the method comprises the following steps: pre-constructing a multi-version program package warehouse, and establishing a structured warehouse directory system; obtaining a chip architecture and an operating system type of the target node, and generating environment fingerprint information; dynamically selecting a database installation package matched with the current environment and a dependency item set of the database installation package by mapping a warehouse directory system; through a unified database adaptation layer, a standardized operation interface is called, and installation, configuration and start-stop operation of the database are executed; a visual deployment and arrangement function based on component roles is provided by expanding a Web service interface of the management platform; and according to a deployment arrangement result and the environment fingerprint information, calling a deployment script to complete automatic deployment and configuration of the database cluster. According to the invention, unified adaptation, deployment and full-life-cycle management of various domestic databases are realized.

The invention relates to the technical field of distributed data storage, and discloses a database cluster partition operation method and device under a network partition scene and a medium, and the method comprises the steps that a multi-master distributed database cluster is constructed, and each node continuously captures a database native transaction log; obtaining change event data, and performing unified data storage; analyzing the original log into structured data, and executing uniform fragmentation and parallelization processing; processing the block log change record, and arranging change events of the primary key according to a rule; executing conflict processing and data synchronization, and synchronizing all nodes after the conflict is solved; through exception handling and fault tolerance, self-healing is realized and data consistency is maintained. According to the method, millisecond conflict identification and data consistency convergence after network disconnection can be realized, continuous and stable operation of a cluster in a partition state is ensured, and self-healing and continuous availability of a system are realized.

The invention discloses a database connection pool management method and device, equipment and a storage medium, and relates to the technical field of database management, the method comprises the following steps: creating a corresponding cluster connection pool for each cluster according to cluster parameter information and cluster configuration information, the cluster connection pool comprising a plurality of connection clusters, the connection cluster comprises a plurality of target connections, and the target connections carry first tags and timers; creating a database cluster pool according to the corresponding relationship between the cluster identifier and the cluster connection pool; and managing the target connection based on the database cluster pool, the first tag and the timer. The database connection resources are managed in a unified manner through the two-stage pool, so that the overhead of connection establishment and disconnection is reduced, and the processing efficiency is improved. And assigning a label to each connection, and upgrading the database connection pool into unequal stateful connections. And the timer is used for marking and tracking the use condition of each connection, so that the life cycle of the connection can be managed, and the management efficiency of the connection pool is improved.

The invention discloses an intelligent database switching method and device, computer equipment and a storage medium, belongs to the technical field of big data, and is applied to a financial database downtime processing scene. According to the method, the node state of the database is monitored in real time, multi-dimensional load evaluation is combined, the optimal main and standby database combination is automatically selected, second-level fault sensing and rapid switching are achieved, and a minute-level time window needed by traditional database recovery is remarkably shortened. The database identifier is embedded in the service main key, so that the request can accurately position the corresponding database node, complex cross-database query and routing overhead are avoided, and the system response efficiency is improved. In addition, through data seamless transmission and automatic abnormal switching between the main database and the standby database, the risk of service interruption caused by database faults is reduced. According to the scheme, the high availability and the fault recovery speed of the database are improved, repeated construction of a multi-product-line database cluster is reduced through resource integration, and a large amount of hardware and operation and maintenance cost are saved.

The invention relates to the technical field of data processing, in particular to a database pre-downsampling method and device, electronic equipment and a storage medium, which can be applied to various scenes such as cloud technology, artificial intelligence, intelligent traffic, auxiliary driving and the like. The method comprises the steps that after time series data are written into all storage fragments of a storage node, a plurality of preset pre-downsampling tasks are obtained, and each pre-downsampling task is used for executing corresponding pre-downsampling processing on the time series data; merging a plurality of fragment tasks associated with the same storage fragment in the plurality of pre-downsampling tasks to obtain each merged task; and for each merging task, instructing the computing node to pull the written target data from the storage fragment associated with the merging task, executing corresponding pre-downsampling processing on the target data according to the merging task, and writing the obtained pre-downsampling data into the storage node. According to the invention, the memory of the storage node can be saved, and the availability of the time sequence database cluster is improved.

The invention discloses a distributed file management system and method, and belongs to the technical field of computer software, the system adopts a tree hierarchical structure to organize files, and the system comprises an application layer used for receiving file operation requests from a plurality of independent clients, and each request carries a unique application identifier; the file system layer comprises a configuration module, an operation module, a metadata management module, a day module, a timed task module and a gateway module, and the request is routed to a corresponding logic storage partition in the data layer according to an application identifier; the data layer comprises a database cluster and an object storage cluster, and performs persistent storage and management of logic isolation on metadata and file data according to application identifiers; and the infrastructure layer provides an extensible server and network resources. Based on a distributed architecture, load balancing and efficient access are realized through data storage partitioning, a metadata management and index technology and a client load balancing technology, and the expandability, fault tolerance and performance of the system are improved.

The invention discloses a database cluster log collection method, system and device and a medium, is applied to the technical field of databases, and is used for solving the problem of relatively low fault reason analysis efficiency of a database cluster in the prior art. The method specifically comprises the steps of determining to-be-investigated database node information corresponding to a database cluster and to-be-investigated log type information corresponding to the to-be-investigated database node information according to a to-be-investigated fault type based on a first configuration file; based on a second configuration file, determining target log collection component information according to the to-be-checked log type information; and calling the target log collection component corresponding to the target log collection component information, and collecting the log file of the to-be-checked database node represented by the to-be-checked database node information based on the to-be-checked log type information, thereby realizing automatic log collection when the database cluster fails, and improving the log collection efficiency. The log collection efficiency when the database cluster fails is improved, and then the fault reason analysis efficiency of the database cluster is improved.

The invention relates to an intelligent optimization method and device for a database cluster, equipment and a storage medium. The method comprises the following steps: generating a heterogeneous feature vector based on multi-modal data of the database cluster; inputting the heterogeneous feature vector and the constructed parameter dependent directed graph into a graph neural network model, outputting a high-priority parameter, inputting the heterogeneous feature vector into a parameter adjustment strategy network, generating a parameter adjustment action of the high-priority parameter, and updating the parameter adjustment strategy network on the basis of recording performance feedback data after parameter adjustment; uploading the encrypted encryption model gradient update vectors of the parameter adjustment strategy network to a federated learning server, performing weighted aggregation processing on the encryption model gradient update vectors of the plurality of database clusters to obtain a global model, and broadcasting the decrypted global model to the plurality of database clusters; according to the technical scheme provided by the invention, the precision, dynamic and knowledge sharing of database cluster parameter tuning can be realized, and the performance and operation and maintenance efficiency of the database cluster are remarkably improved.

The invention discloses a database cluster management method, device and equipment and a computer program product, and relates to the technical field of basic setting and IT supporting, and the method comprises the steps: monitoring the operation state of a first main server node; when the operation state of the first master server node has an operation fault, acquiring a node state of a slave server node cluster; and based on the node state of the slave server node cluster, determining a second master server node from the slave server node cluster through a pushing algorithm. The new master server node is determined between the slave server nodes through the pushing algorithm, automatic switching between the master server node and the slave server nodes is achieved, and the technical problem that a database deployed in a single-point mode cannot be used when having a fault is solved.

The invention relates to a distributed high-availability code warehouse management method and system and a medium, and relates to the technical field of version control and code management.The method comprises the steps that a pre-configured external load balancer, a plurality of GitLab Web nodes, an internal load balancer, a PostgreSQL database cluster, a Redis cluster, a Gitaly cluster and object storage or file storage are obtained; carrying out load balancing on the plurality of GitLab Web nodes through the external load balancer; performing load balancing on the internal connection of the GitLab application program through the internal load balancer; the data of the GitLab Web and the data of the GitLab Pracect are stored through the PostgreSQL database cluster, and the data of the GitLab Web and the data of the GitLab Pracect are And providing access to a Git warehouse by using the Gitaly cluster. According to the invention, high availability, high performance and expandability of code warehouse management are realized through a distributed architecture and redundancy configuration.

The invention discloses a log synchronization method, device, system and equipment, a medium and a program product, and relates to the technical field of databases. The method comprises the following steps: receiving a to-be-synchronized log packet sent by a main library through a standby library in a database cluster; adding the to-be-synchronized log packet into the asynchronous analysis reconstruction task, and sending a response message to the main library, wherein the response message is used for indicating the main library to write the to-be-synchronized log packet into an online log file of the main library; and executing the asynchronous analysis reconstruction task to obtain a reconstructed log packet of the to-be-synchronized log packet, and performing log replay according to the reconstructed log packet. The standby library adds the received to-be-synchronized log packet into the asynchronous analysis reconstruction task, and the response message is returned to the main library without waiting for the completion of the execution of the asynchronous analysis reconstruction task, so that the response waiting time of the main library is effectively shortened, the transaction submission speed of the main library is accelerated, and the log synchronization efficiency between the main library and the standby library is improved.

The invention relates to the field of data transmission, and provides a time series data cross-gatekeeper two-way transmission method, device and equipment, and the method comprises the steps: configuring an Internet of Things database cluster and an isolation gatekeeper, and determining a configuration result; according to the configuration result, the sending end is started to package the time sequence data, a data packet containing redundancy check information is generated, and the redundancy check information comprises a double-length field and a check code; transmitting the packaged data packet to a receiving end through an isolation gatekeeper; after the receiving end receives the data packet, verifying the dual-length field and the check code, if the verification is passed, returning a success signal, otherwise, returning a failure signal; and if the sending end receives the failure signal, automatically retransmitting the data packet until a successful signal is received or a preset retry condition is met. The problem that in the prior art, cross-gatekeeper time sequence data transmission cannot meet the requirements for high safety and real-time performance at the same time is solved, and safe and reliable time sequence data two-way real-time transmission under the strict network isolation condition is achieved.

The invention provides a database cluster inspection control method and system and a server, and relates to the field of database operation and maintaining.According to the method, health examination, performance analysis and fault diagnosis can be conducted on a database cluster in real time through multi-modal data fusion analysis, automatic repair can be conducted on the basis of a dynamic strategy selection mechanism when problems are found, and the problem is solved. And the disposal progress can be synchronously pushed through the multi-channel alarm platform, so that the inspection control effect of the database cluster is remarkably improved.

The invention discloses a data processing method and device, equipment and a medium, and relates to the technical field of computers, and the method comprises the steps: receiving real-time data streams sent by a plurality of clients through a target server cluster; wherein the target server cluster comprises an asynchronous socket server based on a non-blocking network, and the target server cluster realizes event-driven network data processing based on a reactor mode; according to the real-time load of each partition in a target message queue, distributing the received real-time data flow to the corresponding partition in the target message queue; cleaning the real-time data flow of each partition in the target message queue by using a target rule engine with a hot loading function to obtain cleaned data; and storing the cleaned data to a distributed memory database cluster. Therefore, resource tension and response delay caused by connection congestion are reduced, and data consistency and service continuity are guaranteed.

The invention relates to the technical field of databases, and discloses a database pre-synchronization implementation method and device, equipment, a medium and a product. The method comprises the steps of obtaining synchronization task configuration after a pre-synchronization process is started, and obtaining metadata corresponding to each to-be-synchronized table according to the synchronization task configuration; generating a file path corresponding to each to-be-synchronized table according to the metadata and a preset directory structure rule, and performing directional reading according to the file path to obtain a global event data file corresponding to each to-be-synchronized table; and according to the global event data file, obtaining a system change number corresponding to each to-be-synchronized table to complete database pre-synchronization. The accurate path positioning based on the metadata is adopted to replace full disk traversal, so that the disk input / output performance bottleneck can be avoided, the pre-synchronization efficiency can be improved, the pre-synchronization time consumption can be remarkably reduced, the normal operation of a database cluster can be prevented from being interfered, and the system stability can be improved.

The invention discloses a resource allocation method, a computer program product and electronic equipment. The method comprises the following steps: receiving an application data operation request, wherein the application data operation request carries a namespace identifier matched with a first application for triggering operation; determining a node list of a node database occupied by the application data of the first application in the first storage database cluster based on the namespace identifier; obtaining a node demand resource load of a node database in the node list in a second processing period; and determining a first node database from the second storage database cluster and migrating the application data of the first application to the first node database under the condition that the first application is determined to meet the resource redistribution condition based on the node demand resource load. The technical problem of low resource utilization rate caused by neglecting the resource use demand change condition of the application is solved.

The invention discloses a sub-health node switching method and device of a database cluster, equipment and a medium. Comprising the steps of collecting multi-dimensional resource indexes of all database nodes in a database cluster based on a preset period, and calculating a real-time health score of each database node by using a health assessment model; when the real-time health score is lower than a sub-health judgment threshold in a preset step length, marking the real-time health score as a sub-health master node; acquiring the real-time health score, the resource margin and the data synchronization delay of the standby node, and selecting a target standby node; switching the data service access from the sub-health master node to the target standby node, and changing the role identifier of the target standby node as the master node; and taking the sub-health main node as a standby node, completing state reconstruction of the database cluster, and putting into a next acquisition period. According to the embodiment of the invention, active identification and seamless switching of the sub-health nodes are realized, service interruption is effectively avoided, and high availability and service continuity of a distributed database cluster are improved.

The invention provides a double-cluster storage method based on time series data, and belongs to the technical field of time series data storage.The time series data to be stored are written into a standard database cluster through a flow forwarding component, and the time series data to be stored are subjected to data compression in an accuracy reduction mode and written into an accuracy reduction database cluster; detecting time information of standard time sequence data in the standard database cluster, and deleting the standard time sequence data when the time information exceeds a preset time period; acquiring data request time information of the time sequence data access request through a traffic forwarding component, and when the data request time information does not exceed a preset time period, querying standard time sequence data corresponding to the time sequence data access request through a standard database cluster; and when the data request time information exceeds a preset time period, querying the accuracy reduction time sequence data corresponding to the time sequence data access request through the accuracy reduction database cluster. According to the method, the storage cost of the time series data is reduced, and the availability of the accuracy reduction database cluster is improved.

The invention provides a data replication strategy adjustment method and device, electronic equipment and a storage medium, and relates to the technical field of data management. The method comprises the following steps: in response to a state monitoring request for a database cluster, obtaining an operation index data set of the database cluster at the current moment; multiple pieces of operation index data included in the operation index data set represent the motion states of a master node and multiple slave nodes included in the database cluster at the current moment; constructing a first environment state vector based on the operation index data set and an initial data replication strategy of the database cluster; determining a target data replication strategy based on the first environment state vector and a second environment state vector corresponding to the initial data replication strategy; the second environment state vector represents the operation performance of the database cluster at the historical moment when the initial data replication strategy is used; and adjusting data replication policies of the plurality of slave nodes based on the target data replication policy. Therefore, the data replication strategy of the database cluster can be better adjusted.

The invention discloses a real-time quality detection method for Internet of Things big data of a new energy power grid. The method comprises the following steps: firstly, collecting equipment time sequence data in a new energy power grid by a lower-layer Internet of Things platform, and uniformly pushing the equipment time sequence data to a Kafka platform; a user needs to deliver quality inspection standard data to the Flink to be loaded in a broadcast stream mode; meanwhile, the system can realize automatic synchronization after change of measuring point grouping, detection rules and parameter thresholds by means of the characteristics of Flink broadcast streams according to metadata reloaded in a period agreed by a user; the method comprises the following steps: firstly, using an asynchronous IO to consume kafka, then indicating Flink to use asynchronous IO to consume kafka, using configuration in a quality inspection standard data broadcast stream, then carrying out data detection, and finally dropping the detected data into a distributed time sequence database cluster by virtue of Flink Sink. The problems that an existing offline scheme is poor in timeliness, insufficient in adaptability and difficult to cope with super-large-scale concurrency are solved.

The invention discloses a method for collecting time series data statistical information of a distributed database, and belongs to the technical field of distributed databases in computer science, the method is implemented by comprising the following steps: a database running cluster is provided, and the database cluster of one application comprises a plurality of database running nodes; the database operation node operates data change processing, data change monitoring, statistical information refreshing monitoring and optimizer service; the data change processing module is responsible for processing data operation and synchronizing changes to the database storage engine; the data change monitoring module is responsible for recording the increasing and decreasing quantity of data records of each table; the statistical information refreshing monitoring module is responsible for regularly monitoring the data change condition and determining whether to trigger the updating of the statistical information; a query plan optimization module; a database execution module; a database stores an engine. According to the method, the collection performance of the time sequence data statistical information under the large data volume can be greatly improved, and meanwhile, the service query performance of the application is ensured.

The application discloses a data page recovery method and device, electronic equipment and a storage medium. The data page recovery method comprises the following steps: determining a data page cache of a first data page to which a target waiting event is directed, the target waiting event is a waiting event waiting for a target node in a database cluster to respond, and the target node comprises a fault node and / or an active node; releasing other data page caches in the active node except the data page cache of the first data page; and recovering a data page cache of a second data page according to a first redo log of the second data page, the second data page being a data page taking the active node as a recovery node, and the first redo log being recorded in a first log file in a disk. According to the method, other data page caches except the data page cache of the first data page are released, and then data page recovery is performed according to the first redo log of the second data page, so that the recovery efficiency of the data page in the shared storage database cluster is improved.

This application relates to a database cluster recovery method and related equipment. The method includes: obtaining metadata information for each node in the cluster to be recovered by extracting cluster configuration information and node status information; acquiring backup set data of the cluster to be recovered that matches the cluster recovery requirements; determining the node role layout of the target cluster based on its overall operating status; using the target cluster to recover a failed cluster in a cross-environment environment; reconstructing the cluster structure according to the node role layout and the metadata information of each node in the cluster to be recovered, distributing the backup set data to each node in the target cluster, and performing physical recovery in parallel; and using a cluster management tool to associate cluster relationships and perform cluster registration operations based on the nodes in the recovered target cluster to obtain the new recovered cluster corresponding to the cluster to be recovered. This method enables rapid equivalent recovery of database clusters across environments, improving cluster recovery efficiency.

The application relates to the technical field of computers, in particular to a distributed cloud native database management method and system. The method comprises the following steps: deploying multiple distributed database clusters based on a containerized deployment K8S framework; creating a unique database for each database cluster, and performing permission isolation at a user level; providing access to the database in a single cluster based on a memfire-cloud server provided by the K8S framework; performing user access and database access in a load balancing manner; expanding and shrinking the database cluster resources and storage space on demand; when a node fails and cannot be accessed, a K8S framework starts a new node, and database data information of the failed node is restored to the new node. The application can construct multiple distributed clusters on a K8S platform, and can add a database cluster in one key mode; the clusters are isolated from each other and do not interfere with each other, so that resource isolation between users is achieved, and database level isolation is achieved.

This application provides a data processing method, apparatus, device, and storage medium, relating to the field of data processing technology. When the performance parameters of a data repository are determined to be lower than a preset value, the performance parameters of the index cluster and the business database cluster of the data repository are obtained respectively. The index cluster stores the correspondence between business keywords and shard keys. If the performance parameters of the index cluster are lower than a first parameter corresponding to the preset value, an index database is added to the index cluster, and the correspondence is synchronized to the newly added index database. And / or, if the performance parameters of the business database cluster are lower than a second parameter corresponding to the preset value, a business database is added to the business database cluster, and the storage location of the business data is adjusted. This method determines how to expand the database by evaluating the performance parameters of the data repository, reducing expansion costs while ensuring database performance and avoiding business downtime.

The application provides an intelligent diagnosis method and related device of a database cluster. The intelligent diagnosis method of the database cluster comprises: collecting running state data of each database instance in the database cluster at a preset time interval; selecting a target database cluster in a working state from the database cluster according to the running state data; determining an abnormal score of each target database in the target database cluster according to a preset weighting rule and a running index parameter of the target database cluster; when the abnormal score is higher than an abnormal threshold, collecting abnormal event data in a first target database corresponding to the abnormal score according to a preset collection rule, calling a pre-trained intelligent diagnosis model to analyze the abnormal event data, and generating a diagnosis report and a stop-loss suggestion set. The application reduces the experience dependence on experienced database administrators.

The invention relates to a database cluster recovery method and related equipment. The method comprises the following steps: acquiring metadata information of each node in a to-be-recovered cluster by extracting cluster configuration information and node state information of the to-be-recovered cluster; acquiring backup set data of the to-be-recovered cluster matched with the cluster recovery demand information; determining a node role layout of the target cluster according to the overall operation state of the target cluster; the target cluster is used for recovering the failed to-be-recovered cluster in a cross-environment manner; reconstructing a cluster structure according to the node role layout and the metadata information of each node in the to-be-recovered cluster, distributing backup set data to each node in a target cluster, and executing physical recovery in parallel; and based on each node in the recovered target cluster, adopting a cluster management tool to associate a cluster relationship and executing a cluster registration operation to obtain a recovered new cluster corresponding to the to-be-recovered cluster. By adopting the method, quick equivalent recovery of the database cluster in a cross-environment can be realized, and the cluster recovery efficiency is improved.

The application discloses a method for deploying a cross-site dual-active cluster disaster recovery system and the cross-site dual-active cluster disaster recovery system, and comprises the following steps: building a master database cluster and a backup database cluster in different geographical positions, wherein the master database cluster and the backup database cluster are both based on a multi-master architecture; configuring an external port for a first database instance, wherein the first database instance is a database instance in the master database cluster; selecting the first database instance corresponding to each second database instance as a synchronous data source, and configuring the external port of the first database instance corresponding to the second database instance to the second database instance, so as to synchronize data from the first database instance to the second database instance based on the external port, wherein the second database instance is a database instance in the backup database cluster. In this way, the write performance can be expanded while avoiding the problem of high network delay.

The application discloses a data access method and device in a cloud service multi-tenant environment, and relates to the technical field of cloud computing, wherein the method comprises the following steps: obtaining a characteristic value of a tenant when the tenant accesses an application; determining a function type of the user accessing the application; and according to the function type of the user accessing the application, entering a corresponding type of database for data query through a database address returned according to the characteristic value of the tenant. The application can dynamically select a data source according to the characteristic value of different tenants, that is, different database clusters are dynamically called for data query when the application is queried to achieve the purpose of accessing different database clusters, without wasting many machine resources, so that the data isolation problem in the cloud service multi-tenant environment is fundamentally solved, the safety of data access in the cloud service multi-tenant environment is improved, and the cost of deploying machine resources is saved.