208 results about "Database clustering" patented technology

The invention relates to the technical field of databases, and discloses a multi-type database performance optimization and operation and maintenance management method and system.The multi-type database performance optimization and operation and maintenance management method comprises the steps that real-time performance indexes of a heterogeneous database cluster are collected, and a noise reduction data set is obtained; generating a cross-library performance coupling degree matrix through the resource competition coupling degree and the data dependence coupling degree; constructing a coupling relation graph; identifying a bottleneck node set based on the coupling relation graph and the abnormal level; generating an optimization strategy; and performing strategy conflict identification according to the optimization strategy, generating a processing strategy, and executing the processing strategy optimization strategy. Through the improved weighting centrality algorithm, the core bottleneck node which has the greatest influence on the whole cluster is accurately identified, a differential optimization strategy is adopted according to the database type, a perfect strategy conflict detection and avoidance mechanism is established, mutual interference in the optimization process is effectively prevented, the optimization success rate is improved, and the optimization efficiency is improved. And the optimization time is shortened.

The invention belongs to the technical field of computer software, and particularly relates to a heterogeneous database unified deployment management method based on a cluster management platform, which comprises the following steps: predefining metadata of various database services; the method comprises the following steps: pre-constructing a multi-version program package warehouse, and establishing a structured warehouse directory system; obtaining a chip architecture and an operating system type of the target node, and generating environment fingerprint information; dynamically selecting a database installation package matched with the current environment and a dependency item set of the database installation package by mapping a warehouse directory system; through a unified database adaptation layer, a standardized operation interface is called, and installation, configuration and start-stop operation of the database are executed; a visual deployment and arrangement function based on component roles is provided by expanding a Web service interface of the management platform; and according to a deployment arrangement result and the environment fingerprint information, calling a deployment script to complete automatic deployment and configuration of the database cluster. According to the invention, unified adaptation, deployment and full-life-cycle management of various domestic databases are realized.

The invention discloses an intelligent database switching method and device, computer equipment and a storage medium, belongs to the technical field of big data, and is applied to a financial database downtime processing scene. According to the method, the node state of the database is monitored in real time, multi-dimensional load evaluation is combined, the optimal main and standby database combination is automatically selected, second-level fault sensing and rapid switching are achieved, and a minute-level time window needed by traditional database recovery is remarkably shortened. The database identifier is embedded in the service main key, so that the request can accurately position the corresponding database node, complex cross-database query and routing overhead are avoided, and the system response efficiency is improved. In addition, through data seamless transmission and automatic abnormal switching between the main database and the standby database, the risk of service interruption caused by database faults is reduced. According to the scheme, the high availability and the fault recovery speed of the database are improved, repeated construction of a multi-product-line database cluster is reduced through resource integration, and a large amount of hardware and operation and maintenance cost are saved.

The invention discloses a distributed file management system and method, and belongs to the technical field of computer software, the system adopts a tree hierarchical structure to organize files, and the system comprises an application layer used for receiving file operation requests from a plurality of independent clients, and each request carries a unique application identifier; the file system layer comprises a configuration module, an operation module, a metadata management module, a day module, a timed task module and a gateway module, and the request is routed to a corresponding logic storage partition in the data layer according to an application identifier; the data layer comprises a database cluster and an object storage cluster, and performs persistent storage and management of logic isolation on metadata and file data according to application identifiers; and the infrastructure layer provides an extensible server and network resources. Based on a distributed architecture, load balancing and efficient access are realized through data storage partitioning, a metadata management and index technology and a client load balancing technology, and the expandability, fault tolerance and performance of the system are improved.

The invention relates to an intelligent optimization method and device for a database cluster, equipment and a storage medium. The method comprises the following steps: generating a heterogeneous feature vector based on multi-modal data of the database cluster; inputting the heterogeneous feature vector and the constructed parameter dependent directed graph into a graph neural network model, outputting a high-priority parameter, inputting the heterogeneous feature vector into a parameter adjustment strategy network, generating a parameter adjustment action of the high-priority parameter, and updating the parameter adjustment strategy network on the basis of recording performance feedback data after parameter adjustment; uploading the encrypted encryption model gradient update vectors of the parameter adjustment strategy network to a federated learning server, performing weighted aggregation processing on the encryption model gradient update vectors of the plurality of database clusters to obtain a global model, and broadcasting the decrypted global model to the plurality of database clusters; according to the technical scheme provided by the invention, the precision, dynamic and knowledge sharing of database cluster parameter tuning can be realized, and the performance and operation and maintenance efficiency of the database cluster are remarkably improved.

The invention relates to a distributed high-availability code warehouse management method and system and a medium, and relates to the technical field of version control and code management.The method comprises the steps that a pre-configured external load balancer, a plurality of GitLab Web nodes, an internal load balancer, a PostgreSQL database cluster, a Redis cluster, a Gitaly cluster and object storage or file storage are obtained; carrying out load balancing on the plurality of GitLab Web nodes through the external load balancer; performing load balancing on the internal connection of the GitLab application program through the internal load balancer; the data of the GitLab Web and the data of the GitLab Pracect are stored through the PostgreSQL database cluster, and the data of the GitLab Web and the data of the GitLab Pracect are And providing access to a Git warehouse by using the Gitaly cluster. According to the invention, high availability, high performance and expandability of code warehouse management are realized through a distributed architecture and redundancy configuration.

The invention relates to the field of data transmission, and provides a time series data cross-gatekeeper two-way transmission method, device and equipment, and the method comprises the steps: configuring an Internet of Things database cluster and an isolation gatekeeper, and determining a configuration result; according to the configuration result, the sending end is started to package the time sequence data, a data packet containing redundancy check information is generated, and the redundancy check information comprises a double-length field and a check code; transmitting the packaged data packet to a receiving end through an isolation gatekeeper; after the receiving end receives the data packet, verifying the dual-length field and the check code, if the verification is passed, returning a success signal, otherwise, returning a failure signal; and if the sending end receives the failure signal, automatically retransmitting the data packet until a successful signal is received or a preset retry condition is met. The problem that in the prior art, cross-gatekeeper time sequence data transmission cannot meet the requirements for high safety and real-time performance at the same time is solved, and safe and reliable time sequence data two-way real-time transmission under the strict network isolation condition is achieved.

The invention relates to the technical field of databases, and discloses a database pre-synchronization implementation method and device, equipment, a medium and a product. The method comprises the steps of obtaining synchronization task configuration after a pre-synchronization process is started, and obtaining metadata corresponding to each to-be-synchronized table according to the synchronization task configuration; generating a file path corresponding to each to-be-synchronized table according to the metadata and a preset directory structure rule, and performing directional reading according to the file path to obtain a global event data file corresponding to each to-be-synchronized table; and according to the global event data file, obtaining a system change number corresponding to each to-be-synchronized table to complete database pre-synchronization. The accurate path positioning based on the metadata is adopted to replace full disk traversal, so that the disk input / output performance bottleneck can be avoided, the pre-synchronization efficiency can be improved, the pre-synchronization time consumption can be remarkably reduced, the normal operation of a database cluster can be prevented from being interfered, and the system stability can be improved.

The invention discloses a sub-health node switching method and device of a database cluster, equipment and a medium. Comprising the steps of collecting multi-dimensional resource indexes of all database nodes in a database cluster based on a preset period, and calculating a real-time health score of each database node by using a health assessment model; when the real-time health score is lower than a sub-health judgment threshold in a preset step length, marking the real-time health score as a sub-health master node; acquiring the real-time health score, the resource margin and the data synchronization delay of the standby node, and selecting a target standby node; switching the data service access from the sub-health master node to the target standby node, and changing the role identifier of the target standby node as the master node; and taking the sub-health main node as a standby node, completing state reconstruction of the database cluster, and putting into a next acquisition period. According to the embodiment of the invention, active identification and seamless switching of the sub-health nodes are realized, service interruption is effectively avoided, and high availability and service continuity of a distributed database cluster are improved.

The invention provides a data replication strategy adjustment method and device, electronic equipment and a storage medium, and relates to the technical field of data management. The method comprises the following steps: in response to a state monitoring request for a database cluster, obtaining an operation index data set of the database cluster at the current moment; multiple pieces of operation index data included in the operation index data set represent the motion states of a master node and multiple slave nodes included in the database cluster at the current moment; constructing a first environment state vector based on the operation index data set and an initial data replication strategy of the database cluster; determining a target data replication strategy based on the first environment state vector and a second environment state vector corresponding to the initial data replication strategy; the second environment state vector represents the operation performance of the database cluster at the historical moment when the initial data replication strategy is used; and adjusting data replication policies of the plurality of slave nodes based on the target data replication policy. Therefore, the data replication strategy of the database cluster can be better adjusted.

The invention discloses a real-time quality detection method for Internet of Things big data of a new energy power grid. The method comprises the following steps: firstly, collecting equipment time sequence data in a new energy power grid by a lower-layer Internet of Things platform, and uniformly pushing the equipment time sequence data to a Kafka platform; a user needs to deliver quality inspection standard data to the Flink to be loaded in a broadcast stream mode; meanwhile, the system can realize automatic synchronization after change of measuring point grouping, detection rules and parameter thresholds by means of the characteristics of Flink broadcast streams according to metadata reloaded in a period agreed by a user; the method comprises the following steps: firstly, using an asynchronous IO to consume kafka, then indicating Flink to use asynchronous IO to consume kafka, using configuration in a quality inspection standard data broadcast stream, then carrying out data detection, and finally dropping the detected data into a distributed time sequence database cluster by virtue of Flink Sink. The problems that an existing offline scheme is poor in timeliness, insufficient in adaptability and difficult to cope with super-large-scale concurrency are solved.

The application discloses a data page recovery method and device, electronic equipment and a storage medium. The data page recovery method comprises the following steps: determining a data page cache of a first data page to which a target waiting event is directed, the target waiting event is a waiting event waiting for a target node in a database cluster to respond, and the target node comprises a fault node and / or an active node; releasing other data page caches in the active node except the data page cache of the first data page; and recovering a data page cache of a second data page according to a first redo log of the second data page, the second data page being a data page taking the active node as a recovery node, and the first redo log being recorded in a first log file in a disk. According to the method, other data page caches except the data page cache of the first data page are released, and then data page recovery is performed according to the first redo log of the second data page, so that the recovery efficiency of the data page in the shared storage database cluster is improved.

This application relates to a database cluster recovery method and related equipment. The method includes: obtaining metadata information for each node in the cluster to be recovered by extracting cluster configuration information and node status information; acquiring backup set data of the cluster to be recovered that matches the cluster recovery requirements; determining the node role layout of the target cluster based on its overall operating status; using the target cluster to recover a failed cluster in a cross-environment environment; reconstructing the cluster structure according to the node role layout and the metadata information of each node in the cluster to be recovered, distributing the backup set data to each node in the target cluster, and performing physical recovery in parallel; and using a cluster management tool to associate cluster relationships and perform cluster registration operations based on the nodes in the recovered target cluster to obtain the new recovered cluster corresponding to the cluster to be recovered. This method enables rapid equivalent recovery of database clusters across environments, improving cluster recovery efficiency.

The application relates to the technical field of computers, in particular to a distributed cloud native database management method and system. The method comprises the following steps: deploying multiple distributed database clusters based on a containerized deployment K8S framework; creating a unique database for each database cluster, and performing permission isolation at a user level; providing access to the database in a single cluster based on a memfire-cloud server provided by the K8S framework; performing user access and database access in a load balancing manner; expanding and shrinking the database cluster resources and storage space on demand; when a node fails and cannot be accessed, a K8S framework starts a new node, and database data information of the failed node is restored to the new node. The application can construct multiple distributed clusters on a K8S platform, and can add a database cluster in one key mode; the clusters are isolated from each other and do not interfere with each other, so that resource isolation between users is achieved, and database level isolation is achieved.

This application provides a data processing method, apparatus, device, and storage medium, relating to the field of data processing technology. When the performance parameters of a data repository are determined to be lower than a preset value, the performance parameters of the index cluster and the business database cluster of the data repository are obtained respectively. The index cluster stores the correspondence between business keywords and shard keys. If the performance parameters of the index cluster are lower than a first parameter corresponding to the preset value, an index database is added to the index cluster, and the correspondence is synchronized to the newly added index database. And / or, if the performance parameters of the business database cluster are lower than a second parameter corresponding to the preset value, a business database is added to the business database cluster, and the storage location of the business data is adjusted. This method determines how to expand the database by evaluating the performance parameters of the data repository, reducing expansion costs while ensuring database performance and avoiding business downtime.

The application provides an intelligent diagnosis method and related device of a database cluster. The intelligent diagnosis method of the database cluster comprises: collecting running state data of each database instance in the database cluster at a preset time interval; selecting a target database cluster in a working state from the database cluster according to the running state data; determining an abnormal score of each target database in the target database cluster according to a preset weighting rule and a running index parameter of the target database cluster; when the abnormal score is higher than an abnormal threshold, collecting abnormal event data in a first target database corresponding to the abnormal score according to a preset collection rule, calling a pre-trained intelligent diagnosis model to analyze the abnormal event data, and generating a diagnosis report and a stop-loss suggestion set. The application reduces the experience dependence on experienced database administrators.

The invention relates to a database cluster recovery method and related equipment. The method comprises the following steps: acquiring metadata information of each node in a to-be-recovered cluster by extracting cluster configuration information and node state information of the to-be-recovered cluster; acquiring backup set data of the to-be-recovered cluster matched with the cluster recovery demand information; determining a node role layout of the target cluster according to the overall operation state of the target cluster; the target cluster is used for recovering the failed to-be-recovered cluster in a cross-environment manner; reconstructing a cluster structure according to the node role layout and the metadata information of each node in the to-be-recovered cluster, distributing backup set data to each node in a target cluster, and executing physical recovery in parallel; and based on each node in the recovered target cluster, adopting a cluster management tool to associate a cluster relationship and executing a cluster registration operation to obtain a recovered new cluster corresponding to the to-be-recovered cluster. By adopting the method, quick equivalent recovery of the database cluster in a cross-environment can be realized, and the cluster recovery efficiency is improved.

The application discloses a method for deploying a cross-site dual-active cluster disaster recovery system and the cross-site dual-active cluster disaster recovery system, and comprises the following steps: building a master database cluster and a backup database cluster in different geographical positions, wherein the master database cluster and the backup database cluster are both based on a multi-master architecture; configuring an external port for a first database instance, wherein the first database instance is a database instance in the master database cluster; selecting the first database instance corresponding to each second database instance as a synchronous data source, and configuring the external port of the first database instance corresponding to the second database instance to the second database instance, so as to synchronize data from the first database instance to the second database instance based on the external port, wherein the second database instance is a database instance in the backup database cluster. In this way, the write performance can be expanded while avoiding the problem of high network delay.

The application discloses a data access method and device in a cloud service multi-tenant environment, and relates to the technical field of cloud computing, wherein the method comprises the following steps: obtaining a characteristic value of a tenant when the tenant accesses an application; determining a function type of the user accessing the application; and according to the function type of the user accessing the application, entering a corresponding type of database for data query through a database address returned according to the characteristic value of the tenant. The application can dynamically select a data source according to the characteristic value of different tenants, that is, different database clusters are dynamically called for data query when the application is queried to achieve the purpose of accessing different database clusters, without wasting many machine resources, so that the data isolation problem in the cloud service multi-tenant environment is fundamentally solved, the safety of data access in the cloud service multi-tenant environment is improved, and the cost of deploying machine resources is saved.

The application relates to the technical field of distributed data storage, and discloses a database cluster partition running method in a network partition scenario, equipment and a medium, wherein the method comprises the following steps: constructing a multi-master distributed database cluster, continuously capturing database native transaction logs by each node; obtaining change event data and uniformly storing the data; analyzing the original logs into structured data, performing uniform sharding and parallel processing; processing block log change records, arranging the change events of primary keys according to rules; performing conflict processing and data synchronization, synchronizing all nodes after solving the conflicts; and through abnormal processing and fault tolerance, realizing self-healing and maintaining data consistency. The application can realize millisecond-level conflict identification and data consistency convergence after network disconnection, guarantees the continuous and stable operation of the cluster in a partition state, realizes self-healing and continuous availability of the system.

The invention provides a retrieval enhancement method which can be applied to the technical field of big data and artificial intelligence. The method is applied to a distributed cluster, the distributed cluster comprises a document analysis cluster, a vector database cluster and a retrieval type generation cluster, and the method comprises the following steps: querying a preset knowledge base in the vector database cluster based on a received retrieval request from a user to obtain a query result, the preset knowledge base is established by cutting an image-text document on a document analysis cluster based on a preset text cutting strategy; and generating a retrieval formula based on the retrieval request and the query result in a retrieval formula generation cluster so as to use the retrieval formula as the input of a large language model service when the large language model service is called. The invention further provides a retrieval enhancement device and equipment, a storage medium and a program product.

The invention provides a database cluster monitoring method and device and electronic equipment, and relates to the technical field of database management. The method comprises the steps that a query request for user behavior data in a first database cluster is sent to a query server, so that a user behavior log is obtained in real time based on the query request through the query server, and the user behavior log is stored in a second database cluster; wherein the user behavior log is a log for recording user behavior data; sending a timed sampling request for the user behavior data in the first database cluster to a query server, so as to obtain metadata in a specified quantity range through the query server based on the timed sampling request, and generating a data grade corresponding to each piece of user behavior data according to the user behavior data in the metadata by adopting a preset grading strategy, and the data level corresponding to each piece of user behavior data is stored in the second database cluster, so that the resource occupation scale can be reduced, and the risk of influencing the normal use of the big data cluster is reduced.

This invention discloses a conflict resolution method, apparatus, and system for distributed databases, belonging to the field of distributed databases. The method comprises: acquiring consumer update information fed back by a consumer database cluster, wherein the consumer database cluster extracts information from change unit information when cross-datacenter replication is triggered based on change unit information sent by a producer database cluster; generating a replicable MP transaction based on the consumer update information, and determining whether a transaction conflict exists based on the primary key version snapshot corresponding to the replicable MP transaction and the consumer update information; if a conflicting transaction is determined to exist, adjusting the conflicting transaction based on its timestamp or priority to eliminate the conflict. This invention can both prevent data discrepancies between two database clusters due to conflicts, ensuring data consistency between the two clusters, and prevent data loss due to conflicts in one cluster.

The invention discloses a dynamic diet risk assessment method and query system for pancreatitis, and the method comprises the steps: carrying out the multi-source heterogeneous collaborative analysis of feedback data collected by a mobile terminal and hospital discharge guide data through employing a distributed database cluster architecture, and constructing a dynamic risk prediction model based on a Spark Streaming real-time calculation engine and an LSTM neural network. And generating taboo period suggestions and risk compensation coefficients by quantitatively processing time sequence interaction association characteristics among the discharge days, the eating response types and the individual health parameters, and designing a closed-loop feedback mechanism to drive dynamic iterative optimization of the model. According to the system, automatic synchronization of user feedback data and a database cluster is achieved through a Kafka message queue, cooperative processing of MongoDB document rules and Redis real-time sorting is achieved depending on a differential storage engine, and personalized diet risk quantitative evaluation for dynamic changes of pancreatitis patients in the rehabilitation stage is achieved. The taboo period prediction accuracy is improved, the recurrence risk is reduced, and the generalization ability of the system is continuously optimized through a sliding window triggering mechanism.

The application discloses a data library distribution method and device, equipment, a storage medium and a program product. The method comprises the following steps: extracting a target field value from a target message to be processed based on a preconfigured target field, and converting the target field value into a first hash code with a specified length; determining a target hash algorithm corresponding to the target field; combining the high-order element and the low-order element of the first hash code by using the target hash algorithm to generate a second hash code; and determining a target database to be distributed based on the second hash code. The second hash code integrates high-order features and low-order features, which can fully reflect the randomness of the overall hash result, improve the hash dispersion, balance the overall access pressure of the database cluster, reduce the situation that a single machine is shut down due to overload, and reduce resource waste caused by the fact that some machines receive less access.

The invention relates to a database operation and maintenance management method, system and device, computer equipment, a storage medium and a computer program product. The method comprises the following steps: in response to an operation and maintenance operation instruction triggered for a database cluster, determining an operation and maintenance operation type and operation and maintenance expectation information; obtaining target metadata matched with the operation and maintenance operation type and the operation and maintenance expectation information from the metadata storage service, and sharing the metadata of the database cluster by at least two central control machines deployed for the database cluster through the metadata storage service; executing operation and maintenance operation on the database cluster based on the target metadata and the operation and maintenance expectation information; and after the operation and maintenance operation is completed, updating the target metadata in the metadata storage service according to the operation and maintenance operation type. By adopting the method, the reliability of database operation and maintenance management can be improved, and the risk of operation and maintenance interruption of the database is reduced.

The invention discloses a bidirectional replication consistency checking and repairing method based on a record source identifier, particularly relates to the technical field of data consistency of a distributed memory database, and is used for solving the problem that a bidirectional replication cluster is difficult to accurately identify a record source, missing and conflicts in a non-stop scene, so that the reliability of the bidirectional replication cluster is improved. Therefore, the problems of difficulty in consistency verification, high possibility of false difference and repeated modification are solved. A deletion mark, a lock mark and a source mark are coded in a record control area in a centralized manner, a primary key field and a source pseudo column are used as anchor points, and an idempotent identifier constructed based on a primary key and a version field, a fragment comparison process divided according to a verification window and a primary key range and an automatic repair decision oriented to missing records and records with inconsistent fields are combined; therefore, the real difference can still be accurately recognized under the condition of bidirectional replication continuous operation, replication loopback and false inconsistency are avoided, and the consistency of the business data of the memory database clusters on the two sides is kept on the premise that business writing is not stopped.

The invention relates to the technical field of database optimization, in particular to a master-slave data synchronization optimization method and system.The method comprises the steps that a one-master multi-slave cluster of a database is created, and the cluster is provided with at least two slave nodes; creating a monitoring program, wherein the monitoring program can count the master-slave synchronization delay time of each pair; when the master node has a large data volume write-in operation or executes a large transaction, the slave nodes can be triggered to pull a large amount of data from the master node, at the moment, a synchronous delay threshold value is set, master-slave data synchronization is performed according to the threshold value, and the slave nodes with low synchronous delay are preferentially synchronized; according to the method, the influence of master-slave synchronization can be reduced to the greatest extent, and master-slave synchronization can be quickly completed by the slave nodes.

The invention relates to the technical field of energy information, and discloses a regional building group source network load storage information security interaction technology, which comprises the following steps: S1, each energy participant collects exclusive energy data locally, and constructs a local database cluster for storage and security protection; s2, each energy participant generates a zero-knowledge proof based on the data in the local database cluster, and submits the zero-knowledge proof to an alliance block chain network; s3, the alliance block chain network performs consensus and verification on the received zero-knowledge proof; and S4, based on a verification result of the zero-knowledge proof, performing collaborative operation of the regional energy system. By introducing the zero-knowledge proof, each energy participant only needs to generate and submit a proof that the data of the energy participant accords with a specific rule, and each party is allowed to participate in a complex energy optimization and decision process on the premise of completely protecting own privacy.