105 results about "Distributed database" patented technology

The embodiments of this application relate to the field of data processing technology, specifically to a financial data processing method, apparatus, computer equipment, and storage medium. The main steps of the aforementioned method include: storing reference data sequences in vector form in a distributed database; for each moment in each reference data sequence, associating and storing the sequence elements of the current moment with those of the previous moment in a wide table to generate fused data records, and dividing them into multiple data shards according to the time dimension; allocating target data to multiple computing nodes, calculating the differences between the target data and each data shard in parallel to obtain a difference array; and determining the reference result of the target data in each reference data sequence based on the minimum difference value in each difference array, used to quantify the valuation status of the object to be evaluated. Through the above method, the processing efficiency of massive amounts of financial data can be significantly improved, meeting the timeliness requirements of high-frequency trading in data processing.

The application relates to the field of distributed databases in computer science, and particularly provides a distributed database recommended index verification method and device, which comprises a database node, a candidate index verification module and a data warehouse; the database node performs index verification analysis on a database system where an environment is located; the candidate index verification module finds relevant SQL statements according to indexes generated by an index recommendation module, calculates the benefits of a single index and the benefits of an index combination, verifies the effectiveness of the recommended indexes, and eliminates indexes that are useless to application load; and the data warehouse is used to collect and store running data in the database. Compared with the prior art, the application greatly reduces the calculation amount of index verification, and enhances the reliability of index verification, so as to avoid recommending indexes that are useless or even harmful to application load.

This invention discloses a conflict resolution method, apparatus, and system for distributed databases, belonging to the field of distributed databases. The method comprises: acquiring consumer update information fed back by a consumer database cluster, wherein the consumer database cluster extracts information from change unit information when cross-datacenter replication is triggered based on change unit information sent by a producer database cluster; generating a replicable MP transaction based on the consumer update information, and determining whether a transaction conflict exists based on the primary key version snapshot corresponding to the replicable MP transaction and the consumer update information; if a conflicting transaction is determined to exist, adjusting the conflicting transaction based on its timestamp or priority to eliminate the conflict. This invention can both prevent data discrepancies between two database clusters due to conflicts, ensuring data consistency between the two clusters, and prevent data loss due to conflicts in one cluster.

Systems and methods for enabling availability of backup data during restoration of a distributed database system are provided. An external storage system stores a plurality of backup files each comprising backup data and one or more backup metadata files. A plurality of ranges are generated and stored among a plurality of computing devices based on the backup files and metadata files. A plurality of virtual data files are generated and stored in the ranges, where each range of the ranges comprises one or more of the virtual data files, where the virtual data files for a respective range identify physical data configured to be stored by the respective range comprising at least a portion of the backup data of one or more of the backup files. The ranges are configured to serve, based on the virtual data files, a request directed to reading or writing the physical data.

The embodiment of the present specification provides a data query method and device of a distributed database, wherein the data query method of the distributed database comprises the following steps: obtaining a query statement for querying data in the distributed database; extracting features of query keywords contained in the query statement to obtain query keyword features; determining keyword data distribution corresponding to the query keywords in historical data distribution of the distributed database; generating keyword density features based on the obtained keyword data distribution; inputting the query keyword features and the keyword density features into a base estimation model to estimate data base, and obtaining a base estimation value of target data corresponding to the query keywords as a query result of the query statement.

The application provides a bill transaction query method, device, equipment, medium and program product, and relates to the field of financial technology. The method queries a first bill transaction detail wide table in a distributed database to obtain a bill transaction corresponding to the bill transaction query request by calling an interface gateway in response to a bill transaction query request of a user, and outputs the bill transaction to the user. The first bill transaction detail wide table is obtained by using a stream computing engine to perform stream batch association calculation on incremental data and a static dimension table. The incremental data is collected from a business system database through a distributed message queue. The scheme combines the distributed message queue and the stream computing engine to collect incremental data, dynamically associates the incremental data with the static dimension table, and realizes incremental expansion of bill transactions. The interface gateway provides a query interface of the distributed database to support complex logical queries and realize high-concurrency and low-latency query services.

This application provides a method and system for implementing page access, and a server. A plurality of preset storage spaces used to store page directories are provided in the server, and statuses of the page directories are marked when the page directories in the preset storage spaces are migrated, to maintain availability of a user service when a server is added to and / or deleted from a distributed database cluster, thereby helping improve user experience. The method includes: When the server is added to and / or deleted from the distributed database cluster, after sending a first page directory in a first preset storage space to a second server, a first server marks the first preset storage space in the first server as being in an invalid state, and when receiving a first query request from a third server, the first server returns a query failure response based on the invalid state, so that the third server performs redirection until a page directory requested by the first query request is found.

Systems and methods are provided for implementing cloud distributed database capacity planning and / or adjustment, using time-series data analysis. In various embodiments, a computing system may be used to analyze collected throughput data associated with consumption of provisioned throughput resources of a distributed cloud database over one or more past periods by an entity. Based on the analysis, a set of predicted throughput data may be determined or generated for the entity over a future upcoming period. In some cases, based on a determination that adding physical partitions would be required, the computing system may adjust the set of predicted throughput data to reduce or minimize a number of physical partitions to be added. The provisioned capacity of the distributed cloud database may then be dynamically adjusted based at least in part on one of the unadjusted or adjusted set of predicted throughput data.

The application discloses a kind of distributed database two-stage transaction submission method and device based on RDMA, it is related to distributed database high-performance transaction processing technical field, comprising: in pre-commitment stage, data jump is completed using RDMA unilateral writing operation;Through polling, transaction redo log data is written successfully;In the submission stage, the atomic update submission state flag bit is carried out through the RDMA atomic comparison and exchange operation;When the number of successful times reaches the majority threshold value through polling RDMA completion queue, exit waiting loop and execute consensus achievement processing. It can eliminate the multiple network round trip and software processing delay in traditional protocol, so that the end-to-end transaction delay is reduced from the software protocol level to the hardware network level, the system CPU resources are released so that it does not need to participate in the protocol processing, so that the computing resources can be efficiently utilized, and the overall throughput and concurrent processing capacity of the distributed cluster are significantly improved.

The application discloses a distributed database index recommendation monitoring system and method, belongs to the field of distributed databases, and aims to solve the technical problem of how to perform real-time feedback on analysis recommendation functions, establish a closed loop index, and guarantee the health and efficiency of database indexes. The technical scheme adopted is as follows: the structure comprises an index analysis recommendation module, an index suggestion modification application module, an index usage monitoring module, an index usage analysis module, an index report generation module, and a self-adaptive index suggestion optimization module; the index analysis recommendation module is used to start from unhealthy statements, check related tables, access patterns, predicate categories and related fields, and propose candidate indexes based on each SQL statement according to a candidate index generation algorithm; after the generation of the candidate indexes that need to be created, the effectiveness of the candidate indexes is verified through two steps; the two steps specifically refer to verifying the benefits of the indexes on a single SQL statement and verifying the benefits of the indexes on all affected statements under the overall application load.

The application discloses a kind of distributed complex event processing systems using super heuristic algorithm, belong to distributed computer system technical field.The system includes cluster control component, distributed database, message middleware, event collection and analysis middleware, event distribution middleware and CEP processing node.It is in at core, by event collection and analysis middleware call super heuristic algorithm, CEP rule and historical data are trained, and optimal event fragment division method and Gower distance model based on attribute weight are automatically obtained;At runtime, event distribution middleware is based on the distance model to calculate the similarity of new event and existing partition, and in combination with the real-time load state of CEP processing node, dynamic, reasonable load distribution and balance are realized.The application realizes the intelligent adaptation and zero business invasion of CEP processing distribution, significantly improves the adaptability, universality and resource utilization of system in dynamic scene.

An abnormal transaction processing method and device in a distributed database and a storage medium, the distributed database comprising a computing layer and a storage layer, the method being applied to the computing layer and comprising: obtaining a transaction operation connection of a target transaction between the computing layer and the storage layer according to a SQL operation request of the target transaction issued by a business end, executing the SQL operation to process the target transaction, and generating a SQL operation record of the target transaction according to the successfully executed SQL operation; when a network interruption occurs in the processing process, establishing a transaction management connection with the storage layer and checking a transaction state of the target transaction in the storage layer; when the transaction state is a suspended state, destroying the target transaction and starting a new target transaction; re-obtaining a transaction operation connection of the new target transaction between the computing layer and the storage layer, and re-processing the new target transaction. The embodiments of the present disclosure enable the business end to no longer repeatedly issue SQL operation information of the target transaction, simplifies the transaction processing process, and improves the transaction processing efficiency.

In some embodiments, a method includes defining an address book for a distributed database. The address book includes an identifier for each computing device from a set of computing devices that implement the distributed database over a network. The method further includes receiving, from a computing device from the set of computing devices, an event including a transaction for updating the address book, and calculating a reception round of the event based on a consensus protocol for the distributed database that uses the address book. The method further includes updating the address book based on the transaction to define an updated address book after a predetermined number of rounds of at least one of reception rounds of events or updates to the address book.

In an example embodiment, a transaction log index is used to index log records. The transaction log index comprises an in-memory transaction log index and an on-disk transaction log index. The log records are temporarily indexed in the in-memory transaction log index until the storage server can ensure that the corresponding log records are stored without any missing log records in a sequence, as well as possibly based on other correctness and performance considerations.

This invention belongs to the field of agricultural information decision-making, and provides a method and system for supporting agricultural fertilization and irrigation decisions based on privacy intersection technology. The method includes: receiving raw agricultural data sent by farmer terminals, agricultural research institution terminals, and agricultural input supplier terminals; encrypting the raw agricultural data and storing the encrypted agricultural data in a distributed database; upon receiving an agricultural fertilization and irrigation decision request, determining key agricultural data matching the request based on privacy intersection technology and the encrypted agricultural data stored in the distributed database; establishing an agricultural fertilization and irrigation decision model based on the key agricultural data; and sending the agricultural fertilization and irrigation decision results output by the model to the farmer terminals. By using encryption and privacy intersection technology to achieve secure data sharing among all parties, and combined with the agricultural fertilization and irrigation decision model, the security of data interaction and the reliability of decision-making are improved.

The data processing method, apparatus, server, and storage medium for a distributed database provided in this invention include: receiving a data processing request sent by a client, the processing request including a data processing template; after sending a processing response to the processing request to the client, receiving an execution request sent by the client, the execution request including a template identifier and template parameters corresponding to the data processing template; invoking a data query processing thread and a data insertion processing thread according to the template identifier and template parameters; streamwise acquiring a preset number of target data rows from query data obtained from a first database node based on the data query processing thread; and sending the target data to a second database node based on the data insertion processing thread, so that the second database node performs a data insertion operation on the target data until the query data insertion is completed. This application can realize simultaneous data query and target data insertion operations, shortening data processing time and improving data processing efficiency.

Embodiments of the present disclosure provide a data recovery method and device, electronic equipment and storage medium, relating to the field of databases, which can be used in the field of distributed databases. The method comprises: obtaining replica information reported by each surviving node; if it is determined that an abnormal data shard based on the replica information reported by each surviving node, determining a target replica from at least one replica of the abnormal data shard in each surviving node; and in the case that a preset recovery trigger condition is met, processing log recovery for the abnormal data shard through the target replica. Embodiments of the present disclosure ensure that the log can be advanced and consistency can be maintained when the majority is unavailable, realize non-stop recovery, do not need to suspend services or manual intervention, the recovery process is completely automated, reduce the risk of manual operation, improve the recovery efficiency and stability, and thus improve the high availability of the distributed system.

This application provides a method, apparatus, server, and medium for batch data import using a distributed database, relating to the field of data processing technology. The method employs a data loading executor to obtain task configuration information for batch data loading tasks targeting a target business database using a stream-batch integrated computing engine; the data loading executor generates a control file based on the task configuration information; the data loading executor pulls batch business data from the target business database based on the task configuration information and generates data files based on the batch business data; and the data loading executor executes preset data loading commands according to the control file to write the data files to the target distributed database. This saves resources and improves data import efficiency in batch task scenarios.

This invention discloses a method for tracking and tracing sensitive network behaviors, relating to the field of network security technology. The specific steps of this method are as follows: collecting and preprocessing multi-source heterogeneous data in the network environment and storing it in a distributed database; constructing a cross-modal semantic graph using a large language model; training an intent classification model to predict sensitive behavior intents and forming an association set; locating intent trigger points and constructing a complete tracing chain through an intent trigger point causal contribution algorithm; integrating data to generate a standardized tracing report, and presenting it through a visualization tool, supporting interaction and export. This invention first integrates and analyzes multi-source heterogeneous data to construct a cross-modal semantic graph, accurately identifies sensitive behavior intents through model training and filtering, then locates intent trigger points through algorithms, constructs a tracing chain, and finally outputs a visualized report, forming an easy-to-understand report, thus improving the comprehensiveness, accuracy, and convenience of sensitive behavior identification and tracing.

This application discloses a defragmentation method, device, and medium for GoldenDB databases, relating to the field of database data processing technology. The method includes: obtaining metadata information of all tables within the cluster from the management node of the GoldenDB distributed database; filtering tables that meet the defragmentation conditions from all tables based on preset dual thresholds to accurately determine the defragmentation targets; intelligently grouping the tables to be defragmented according to a preset defragmentation strategy and generating corresponding defragmentation tasks based on the grouping results; and distributing the defragmentation tasks to the computing nodes of the GoldenDB database for execution to complete the data defragmentation operation within the cluster. Through parallel execution by grouping, the total defragmentation time is reduced compared to existing random parallel schemes. The dual threshold filtering based on table size and fragmentation rate ensures that only tables requiring defragmentation are processed, saving system resources.

A database system includes a commit layer implemented using a first set of host computing devices and a storage layer implemented using a second set of host computing devices. A control plane of the distributed database system determines a first sharding scheme for the commit layer and a second sharding scheme for the storage layer, wherein the first and second sharding schemes are not required to be the same. Also, in some embodiments, the second sharding scheme used for the storage layer enables overlapping key spaces across the shards of the storage layer, wherein various ones of the shards are optimized for different types of workloads.

This application discloses a method, apparatus, and system for intelligently generating visual reports for distributed relational databases. It connects to the target database and parses its stored procedures, publishing them as callable data interfaces. It receives natural language indicator descriptions input by the user and, based on the published data interfaces, maps them to corresponding calling parameters using natural language processing technology. Reports are generated based on the calling parameters, and predictive dataset caching and updates are performed based on the user's historical access patterns and database source data change information. This application achieves deep integration with distributed databases, intelligent report configuration with zero code, and efficient data caching and updates, significantly improving report development efficiency, system response speed, and operation and maintenance management capabilities.

A method and device for scaling a distributed database and a storage medium, comprising: in the case of meeting a scaling condition, scaling the distributed database using a consistent hashing algorithm; during the scaling, suspending a write request from a client for a target physical node; obtaining feature information of the suspended write request according to a preset strategy; in the case of the obtained feature information reaching a feature threshold, suspending the scaling, and processing a new write request from the client for the target physical node and the suspended write request whose feature information reaches the feature threshold, until the scaling is resumed in the case of re-meeting a scaling preparation condition. The embodiments of the present disclosure enable the distributed database to still provide normal services during scaling, thereby avoiding service interruption of the distributed database and improving service availability of the business distributed database.

A method for handling de-duplication in a leaderless distributed database system is provided. The database system comprises a set of datacenters. No datacenter in the set of datacenters is configured as a dedicated backup datacenter. A datacenter is selected (902) from the set of datacenters in the distributed database system for a keyspace associated with a backup or restore operation. A replication factor for the keyspace is selected (903). Based on a replication factor for the keyspace, the backup and / or restore operation is assigned (904) to one or more nodes in the selected datacenter. The backup and / or restore operation is performed (905).

Embodiments of the present application provide a data processing method and device, electronic equipment and storage medium, and relate to the technical field of databases. The method comprises: receiving a data query request and determining a target data table corresponding to the data query request; determining a target index corresponding to the target data table from a plurality of redistribution indexes; wherein the data in the redistribution indexes is stored in a distributed database based on the distribution of index columns of the redistribution indexes; when the operation for the data query request is directed to the index column of the target index, the original execution plan is optimized according to the target index to generate a target execution plan; the operation for the data query request comprises a single-table aggregation operation and / or a multi-table association operation; and the target execution plan is run to obtain a query result corresponding to the data query request. The embodiments of the present application realize query optimization, reduce cross-node data exchange operations, improve the efficiency of data processing, and improve the performance of the entire distributed database.

This invention provides a method and system for tracking user behavior in text and image content, comprising: splitting text and image content into basic element units and assigning unique identifiers, embedding lightweight collection scripts and binding listeners; collecting effective interaction behavior data on user terminals; preprocessing the data in the backend to obtain a single user access behavior sequence; storing the data using a distributed database architecture and building a multi-dimensional index; performing multi-dimensional analysis and calculation of element interaction popularity through models to mine user preferences and behavioral paths; visually outputting analysis reports and synchronizing interest data to downstream business systems. The system has six functional modules to implement the above method. This invention achieves element-level accurate full-link tracking, providing rich and efficient data processing, and can provide data support for content optimization, user profile construction, and precision marketing, while also protecting privacy and adapting to large-scale application scenarios.

The specification discloses a processing method, device, medium and equipment of a data table structure change operation. In the method, a master node of a distributed database sends a table structure change command to a first node, so that the first node obtains a data table to be changed according to the received table structure change command, and performs a data table structure change operation to obtain a target data table after the table structure of the data table to be changed is changed. A checkpoint file generated by the first node in the process of performing the data table structure change operation is saved to a specified storage node, so that when the master node monitors that the first node is abnormal, the master node can reselect a second node from the distributed database to replace the first node, and continue to perform the data table structure change operation based on the checkpoint file through the second node.

A communication method, distributed database middleware and a computing device cluster are provided. The method comprises: obtaining an access request from a client, the access request being used to indicate a processing operation on batch data of a data table, wherein the data table belongs to a database system, and the database system comprises a plurality of database nodes; determining at least one target node related to the batch data in the plurality of database nodes, and determining data corresponding to each target node in the batch data; and sending an operation request corresponding to each target node to each target node based on a parsing-binding-execution (PBE) protocol, wherein the operation request corresponding to each target node is used to indicate that the processing operation is performed on the data corresponding to the target node.

Provided herein are systems, methods, and computer-storage media for managing data skew in hash join operations. A skew manager partitions build-side row data into multiple sets corresponding to hash-join-build (HJB) instances based on hash values. The skew manager detects skew in a build-side row set associated with a first HJB instance by analyzing the number of rows. Upon detecting skew, the skew manager redirects data rows to at least a second HJB instance. The method involves configuring skew caches, generating histograms, and detecting frequent hash values to identify skew. It also includes communicating skew notifications, broadcasting probe-side row data, and adjusting partitioning of probe-side data. The disclosed techniques further include buffering build-side row sets in streams and performing join operations based on these streams, enhancing efficiency in distributed computing environments.

The application relates to the technical field of database operation and maintenance, in particular to a large transaction statistics implementation method based on a distributed database binlog log, which is characterized in that target binlog log files and a statistics time range are specified, a shell script is combined with a log analysis tool and a data processing tool to perform directional analysis on binlog logs of a distributed database, transaction-related key information is extracted, including an operation table name, a transaction type, an affected row number, a transaction execution duration and a transaction ID, large transaction data of each table generated within a specific time period is counted, the historical running state of the database during an abnormal period can be quickly traced back, the table and related application of the large transaction causing the abnormality can be accurately located, the distributed database operation and maintenance efficiency and problem troubleshooting accuracy are improved, and the method is suitable for operation and maintenance scenes of open source or domestic distributed databases under an IOE background.