31 results about "Redo log" patented technology

The application discloses a data page recovery method and device, electronic equipment and a storage medium. The data page recovery method comprises the following steps: determining a data page cache of a first data page to which a target waiting event is directed, the target waiting event is a waiting event waiting for a target node in a database cluster to respond, and the target node comprises a fault node and / or an active node; releasing other data page caches in the active node except the data page cache of the first data page; and recovering a data page cache of a second data page according to a first redo log of the second data page, the second data page being a data page taking the active node as a recovery node, and the first redo log being recorded in a first log file in a disk. According to the method, other data page caches except the data page cache of the first data page are released, and then data page recovery is performed according to the first redo log of the second data page, so that the recovery efficiency of the data page in the shared storage database cluster is improved.

This invention relates to the field of data processing and discloses a database log archiving method, apparatus, device, and medium, comprising: identifying a target redo log file to be archived from the database; creating a preliminary redo log file with the same attributes as the target redo log file and empty content within the same logical disk as the target redo log file; moving the target redo log file to a preset archiving directory located within the same logical disk via MV operations; changing the filename and storage path of the preliminary redo log file to the original filename and storage path of the target redo log file via MV operations; and mounting the revised preliminary redo log file to the database. This invention solves the problems of high I / O consumption and low archiving efficiency in existing database log archiving technologies.

The invention discloses a duplication data deletion metadata management method based on a persistent memory. The method comprises the following steps: managing a global fingerprint table and a logic address to a fingerprint mapping cache in a DRAM (Dynamic Random Access Memory), and managing an additional redo log and a global mapping table in the persistent memory. The write-in path firstly writes metadata update into the DRAM and forms a transaction record, the transaction record is sequentially added to a persistent memory log in a group submission mode, persistence is aligned according to the read-write granularity of the persistent memory, and random write and lock competition is reduced; in the checkpoint stage, an aggregation strategy is dynamically selected according to the redundancy rate and the coverage write proportion obtained through online load detection, and therefore metadata write amplification is remarkably reduced on the premise that collapse consistency is guaranteed. Meanwhile, three self-adaptive deduplication modes are provided, fingerprint management is bypassed in the low redundancy stage, redundancy is sampled and estimated in the medium redundancy stage, and full deduplication is started in the high redundancy stage, so that the metadata management cost is stably reduced under different loads, and the system throughput is improved.

The invention provides a data processing method and device, electronic equipment and a storage medium, relates to the technical field of data processing such as data storage and data analysis, and is suitable for cloud computing and database scenes. According to the specific implementation scheme, the method comprises the steps that a column redo log is pulled from a log server through a column storage engine, and the column redo log is written into the log server through a main computing node; the column redo log is analyzed and played back, first target data for data operation in a data table is determined, and the data table comprises a column storage index; and storing the first target data according to columns.

The application discloses a kind of database continuous log backup method, system, device and storage medium, belong to log backup technical field, comprising: query archive log list, copy all archive log files;Copy all redo log files, generate redo log copy file, obtain the path of the redo log file being written currently and the offset of the file, read the current redo log content from the start position to the offset position, overwrite the corresponding copy file, to calibrate the current redo log copy file content;If offset position changes, then read the content between last offset and current offset from current redo log file, overwrite to the corresponding copy file, the database continuous log backup method, system, device and storage medium do not need to install drive, have no performance influence to system and application, will not cause system crash downtime, without additional preparation target machine in advance, reduce implementation cost.

A method and system for data synchronization between a standby site and a primary site in a distributed database system is provided. A first primary site included in a primary site group of the distributed database system generates a commit action redo log message for a commit action performed by the first primary site on a first database transaction, the commit action redo log message including a transaction ID of the first database transaction and a transaction order indicator representing an order of database transactions in a group of database transactions executed on one or more primary sites in the primary site group. The first primary site sends the commit action redo log message to a corresponding first standby site backing up the first primary site.

The application discloses a kind of distributed database two-stage transaction submission method and device based on RDMA, it is related to distributed database high-performance transaction processing technical field, comprising: in pre-commitment stage, data jump is completed using RDMA unilateral writing operation;Through polling, transaction redo log data is written successfully;In the submission stage, the atomic update submission state flag bit is carried out through the RDMA atomic comparison and exchange operation;When the number of successful times reaches the majority threshold value through polling RDMA completion queue, exit waiting loop and execute consensus achievement processing. It can eliminate the multiple network round trip and software processing delay in traditional protocol, so that the end-to-end transaction delay is reduced from the software protocol level to the hardware network level, the system CPU resources are released so that it does not need to participate in the protocol processing, so that the computing resources can be efficiently utilized, and the overall throughput and concurrent processing capacity of the distributed cluster are significantly improved.

In some embodiments, there may be provided extracting, from a redo log, a first chunk of data; checking whether a population indicator is set; in response to the first chunk of data, creating a main buffer; in response to a population indicator not set, writing the first chunk of data to the created main buffer and repeat the writing for one or more additional chunks until a last chunk is extracted; and after the last chunk is written to the main buffer, reading the main buffer to write the first chunk of data, the one or more additional chunks, and the last chunk to main memory. Related systems, methods, and articles of manufacture are also disclosed.

Herein is database transaction throttling to compensate for excessive redo generation and excessive database replication lag. Each time interval in a sequence of contiguous time intervals is assigned to a distinct respective redo counter in a circular buffer. While executing database statements in a database transaction, some of the redo counters are adjusted to reflect a fluctuating amount of redo generated by the transaction. When the redo generation rate of the database transaction is detected as exceeding a threshold, execution of the database statements in the database transaction is paused.

The invention provides a virtual file system, a binlog-in-redo implementation method, electronic equipment and a readable medium, and relates to the technical field of databases. According to the scheme, a file management service responds to a redo log persistence request transparently transmitted by a database service layer through a virtual redo log service, and persistence operation is executed on physical files which are written into a redo log file block and a binlog file block in a staggered mode; the file management service also manages a shared memory, and the shared memory comprises a redo log cache and a binlog cache; the virtual redo log service provides an access path of the redo log cache, and the virtual binlog service provides an access path of the binlog cache. According to the scheme, writing of the redo log file block and writing of the binlog file block are combined into writing of the physical file, the transaction submission operation efficiency is improved, when it is confirmed that persistence of the binlog file block is completed, master-slave synchronization is triggered in an asynchronous notification mode, and the risk of master-slave inconsistency is avoided; the original storage format can be displayed to the outside through the virtual service, seamless transparent compatibility is achieved, the customization requirement is lowered, and the deployment cost is reduced.

The invention discloses a redo log parallel analysis method and device based on heterogeneous database data synchronization. According to the method, parallel acquisition load balancing is realized through an archived file granularity self-adaptive splitting and cross-boundary transaction detection mechanism; a dynamic task scheduling strategy sorted according to FIRSTCHANGE # is adopted, an event chain table is maintained in combination with a memory mapping file, and accurate merging of multi-thread event streams is achieved through SCN global comparison; and dynamically switching a serial-parallel analysis mode according to redo update frequency, automatically degenerating serial and continuously monitoring a log switching state in a high-frequency change scene, and ensuring data continuity through a site backspacing and SKIP condition filtering mechanism. According to the method, the throughput capacity and the real-time performance of incremental synchronization of the heterogeneous database are remarkably improved, data loss and repetition caused by log switching are effectively avoided, and the data consistency and the time sequence accuracy are ensured while system parallel computing resources are fully utilized.

The invention is suitable for the technical field of databases, and relates to a log analysis method and device, computer equipment and a computer readable medium. Wherein a logic log record corresponding to a transaction is obtained in response to analysis of the redo log, and whether the current time reaches a check point triggering period or not is recognized; if the current time reaches the check point triggering period, determining a target active transaction which is started earliest from the active transactions, and taking a redo log starting point corresponding to the target active transaction and a system logic clock corresponding to the logic log record as candidate check points for associative storage; responding to a received application system logic clock returned by the log application end, and determining a target candidate check point matched with the application system logic clock in the stored candidate check points; and configuring a redo log starting point in the target candidate check points as a formal check point of the redo log. According to the invention, the memory overhead required for promoting the check point can be reduced.

Provided in the present application are a log backup method and apparatus, and a storage medium and a computer program product. The method comprises: on the basis of number information of a copied redo log, determining first number information expected to be read next time; sequentially reading number information of redo logs before the redo logs are written into a disk, and when it is found, by means of reading, that second number information of a first redo log is greater than the first number information, storing the first redo log in an archive file; and after the storage is completed, backing up redo logs in the archive file to a backup node.

The embodiment of the invention provides a redo log processing method and device, a storage medium and a program product, and relates to the field of distribution. The method comprises the steps that metadata information of a redo log file in an ASM storage layer is obtained through an ASM API interface of an automatic storage management system; wherein the ASM API interface is used for accessing an ASM storage layer, the redo log file comprises a log file used for recording database change operation, and the metadata information at least comprises a file path, a block number, a block size and a file type; according to the metadata information, performing block reading on the redo log file through an ASM API interface to obtain redo log data read in a block manner; and converting the redo log data read in blocks into a local redo log format to obtain a local redo log. According to the method, the capability of reading and analyzing the redo log file from the ASM storage layer is increased, and the efficiency of fault recovery in database operation and maintenance is improved.

The application provides a data page processing method and device, electronic equipment and a storage medium, wherein the method comprises: parsing redo logs of a first batch to obtain a plurality of application tasks, a target copy page containing a target original page in a target memory space, the target original page being a data page associated with a target operation parsed from the redo logs of the first batch, the target operation being a split or merge operation of the data page; distributing the plurality of application tasks to a plurality of worker threads, each worker thread in the plurality of worker threads being used for processing part of the plurality of application tasks; executing the plurality of application tasks through the plurality of worker threads, an application task corresponding to the target original page in the plurality of application tasks being applied to the target copy page; and in the case that the plurality of application tasks are executed, updating data of the target copy page to the target original page by adding an exclusive lock to a target index, the target index being an index in which the target original page is located.

A data write minimization apparatus according to one embodiment may perform caching a page stored in a storage device into a buffer of a volatile memory; generating, when a data update transaction occurs for the page, a PPL block including a per-page log (PPL), which is a per-page redo log, for the data update transaction in a non-volatile memory, and storing, when the PPL block grows larger than a preset size, an entire page in the storage device; and maintaining, when a page write occurs for a PPL page in which the redo log is stored as a PPL block in the non-volatile memory, the PPL block stored in the non-volatile memory, and performing, when a page write occurs for a normal page, an entire page write to the storage device.

The application provides a database redo log parallel processing method and related products, wherein the database redo log parallel processing method comprises the following steps: a transaction process of a master database writes generated redo log data into a common log buffer; the transaction process awakens a log disk flushing process and a stream replication sending process of the master database in parallel; the log disk flushing process writes the redo log data to be persisted from the common log buffer to a disk log file of the master database; and the stream replication sending process sends the persisted redo log data from the common log buffer to a slave database. According to the scheme, the working starting point of the stream replication sending process is advanced from after the log disk flushing to during the log disk flushing, the overlap of network transmission time and disk I / O time is realized, the redo log data can flow to the slave database in real time, the replication delay between the master-slave cluster is shortened, and the real-time availability of data is improved.

The invention discloses an HNSW index data management method and device, equipment and a medium, and the method comprises the steps: obtaining a target node corresponding to a to-be-executed transaction, and carrying out the target operation of a storage position corresponding to the target node according to the to-be-executed transaction, and generating a target redo log and a target revocation log corresponding to the target operation and version transaction data of the to-be-executed transaction for performing the target operation on the target node, before the to-be-executed transaction is completed, executing a row lock on a corresponding storage position in the target node, and according to the to-be-executed transaction, performing an updating operation on a neighbor relationship between the neighbor node and the target node. And generating a neighbor redo log and / or a neighbor revocation log corresponding to the update operation, and before the to-be-executed transaction is completed, not limiting the access to the neighbor relationship of the neighbor node. The HNSW index data is decoupled and classified, and a differentiated transaction processing mechanism is adopted, so that good balance is achieved between guarantee of global strong consistency and reduction of neighbor relationship transaction management overhead, and performance and consistency are effectively considered.

Some embodiments provide a non-transitory machine-readable medium that stores a program to write a redo log for operations performed to modify data within a database. The program sets a throttle mass modification flag when a ring list of log buffers is full and unable to write an additional log entry at this point in time. When the throttle mass modification flag is set, the writing of the redo log for mass operations is blocked or delayed, thereby reducing the delay experienced when writing user operations into the redo log. This can improve the user experience.

The invention discloses a distributed database two-stage transaction submission method and device based on RDMA (Remote Direct Memory Access), and relates to the technical field of distributed database high-performance transaction processing. The method comprises the following steps: completing data jump by utilizing RDMA unilateral write operation in a pre-submission stage; confirming that the transaction redo log data is written successfully through polling; in the submission stage, the submission state flag bit is updated through RDMA atom comparison and operation atom exchange; and when the successful number of queue statistics completed by polling RDMA reaches a majority threshold, jumping out to wait for loop execution of consensus achievement processing. According to the method, multi-round network round-trip and software processing delay in a traditional protocol can be eliminated, end-to-end transaction delay is reduced to a hardware network level from a software protocol level, system CPU resources are released and do not need to participate in protocol processing, computing resources can be efficiently utilized, and the overall throughput and concurrent processing capacity of a distributed cluster are remarkably improved.

The application provides an automatic data recovery method of a database cluster and a storage medium. The automatic data recovery method comprises the following steps: acquiring a master database signal of the database cluster; acquiring a starting point of a redo log stream returned by a switching target database and a terminal point of a redo log stream returned by a switching source database in the case that the switching target database and the switching source database are both normally connected with a trusted gateway; setting a redo recovery point of a backup database; and determining a recovery data area according to the starting point of the redo log stream returned by the switching target database, the terminal point of the redo log stream returned by the switching source database and the redo recovery point of the backup database, so as to realize automatic data recovery.

The application discloses a database-oriented batch insertion transaction processing method and device, equipment and medium, relates to the technical field of database, and includes the following steps: when a database batch insertion transaction is monitored, a shared lock and an intention exclusive lock are applied to an index organization table, and a mark is set for the transaction and the index organization table; a first physical record ROWID and a rollback record of the transaction are generated, a data page is allocated in an index organization table storage space, batch data is inserted, only a redo log of the first record of the data page is generated, and batch insertion is completed. If the transaction is marked, the redo log and the data page are written to a disk, and the mark is reset to commit the transaction. When the transaction is rolled back, the ROWID is obtained according to the rollback record, the user record and the data page where the user record is located are located in the index organization table, the internal positioning record of the inner node is found according to the user record and is deleted, and the corresponding data page is released. The application can reduce the generation of redo logs to the maximum extent on the basis of guaranteeing data persistence.

The invention is suitable for the technical field of databases, and relates to a log processing method and device, computer equipment and a computer readable medium. The method comprises the following steps: counting a first check point advancing speed of a redo log corresponding to a database, and counting a first log generation speed of the redo log; if the first check point advancing speed is smaller than the first log generation speed, determining a target log generation speed smaller than the first check point advancing speed; and performing flow control on a service session thread corresponding to the database according to the target log generation speed. The stability of database performance can be improved.

The invention relates to the technical field of databases, and discloses a data persistence synchronization method, device and equipment based on a cloud environment, and the method comprises the steps: building a distributed database, and configuring a sharing architecture which comprises a sharing cache layer, a sharing persistence memory layer and a sharing storage layer; distributing corresponding transactions to each node, and synchronously writing data into a shared cache layer after the nodes update the data; allocating a timestamp for the transaction; judging whether a logic page exists in the shared cache layer or not; if the logic page exists, acquiring the current timestamp of the transaction as a read timestamp, reading each page requested by the transaction into a cache of a current node, executing the transaction locally and storing private data generated in transaction execution locally; and writing a redo log generated in execution of the transaction into a log area in the persistent memory, and performing playback by a background thread to generate page data in the shared persistent memory layer. The method and the device have the effects of reducing time delay and enhancing reliability.

The invention relates to the technical field of database resource management, in particular to an active-active database disaster recovery method and device based on shared storage, equipment and a medium. A database mounted on each database instance is connected with a local storage, a city-wide storage and a far-end storage, if the type of the write access is a new operation, new data is written into the local storage connected with the local database, a new redo log of the new data is sent to the city-wide storage, and the new redo log is asynchronously sent to the far-end storage; and the remote database reads the newly-added redo log from the remote storage and generates a copy of the newly-added data. Data storage is divided into local storage, intra-city storage and far-end storage, after redo is sent to the intra-city storage, it is considered that a transaction is completed, then a redo log is sent to the far-end storage in an asynchronous mode, the transaction does not need to wait for the redo log to be transmitted to the far-end storage and then submitted, and delay caused by transmission of the redo log between two database management systems is avoided.

The invention provides a fault recovery method and device, a medium and a product. The fault recovery method of the cluster file system comprises the following steps: acquiring all redo logs of a failed disk pack in the cluster file system after a newest check point; grouping all redo logs of the disk pack according to file granularity; combining the redo logs in the same group to obtain a combined redo operation of the file; and performing parallel playback on all the merging and redoing operations of the disk pack. Concurrent processing of the redo logs can be achieved, and the multi-core characteristic of hardware and the high throughput of a solid state disk are fully utilized. The recovery efficiency of the disk pack is effectively improved, so that the time that service cannot be provided for the outside due to fault recovery is shortened, and the availability of the cluster file system is greatly improved.

The application discloses a kind of HNSW index data management method, device, equipment and medium, comprising: obtaining the target node corresponding to the transaction to be executed, according to the transaction to be executed, target node corresponding storage location is operated, generates the target redo log corresponding to target operation, target undo log and the version transaction data of the target node for target operation of the transaction to be executed, before the transaction to be executed is completed, the line lock is executed to the corresponding storage location in target node, according to the transaction to be executed, the neighbor relationship of neighbor node and target node is updated, generates the neighbor redo log corresponding to update operation and / or neighbor undo log, before the transaction to be executed is completed, neighbor relationship of neighbor node is not limited to access.The HNSW index data is decoupled and classified, and a differentiated transaction processing mechanism is used, a good balance is achieved between ensuring global strong consistency and reducing neighbor relationship transaction management overhead, effectively balancing performance and consistency.

The present application relates to database technology, in particular to a redo log synchronization method, product, medium and equipment. The redo log synchronization method comprises: a control node receiving registration information from a normal node, wherein the registration information corresponds to metadata to be accessed in the normal node; the control node generates a corresponding redo log when the metadata changes, and obtains the normal node to be synchronized according to the registration information of each normal node; in the case that there is a normal node to be synchronized, the control node broadcasts the synchronization content of the redo log to the normal node to be synchronized; the normal node to be synchronized obtains and replays the redo log according to the synchronization content. The redo log synchronization method of the present application filters the nodes to be synchronized by the control node, and only synchronizes the redo log to the nodes with metadata access requirements, avoiding indiscriminate synchronization to all nodes in the cluster, reducing the number of synchronization objects, and thus improving the execution efficiency of the redo log synchronization.

The embodiment of the invention provides a write-in method and device based on a storage and calculation separation architecture database and a storage medium. The method comprises the following steps: receiving a first write operation instruction of a first client to a target data page through a calculation end of the storage and calculation separation architecture database; judging whether a buffer block of the target data page exists in a buffer pool of the computing end or not; if it is determined that the buffer block of the target data page does not exist in the buffer pool of the calculation end, the first write operation instruction is packaged into a first redo log, the first redo log is sent to a storage end of the storage and calculation separation architecture database, and the storage end executes the first write operation instruction in the first redo log on the target data page. According to the embodiment of the invention, the target data page does not need to be read from the storage end, the calculation end executes the write operation instruction on the target data page, and then the target data page is refreshed to the storage end, so that the I / O overhead between the calculation end and the storage end is reduced, and the write-in throughput capacity of the storage and calculation separation architecture database and the utilization efficiency of calculation resources are improved; and the performance of the storage and calculation separation architecture database is improved.

The application provides a database master-slave synchronization method, device, equipment, medium and product, wherein the method comprises the following steps: obtaining a first redo log including a first globally unique number generated by a master database end for a first log as a data definition language log; in the case that a log analysis thread judges that the log type of the first redo log is the first occurrence, obtaining a corresponding first binary log cached in a relay log file according to the first globally unique number, and applying the binary log to a slave database through a database language thread. The database master-slave synchronization is realized based on the binary log and the redo log, and the method of generating the log based on the globally unique number is supplemented, so that the replication delay problem caused by the method of realizing the database master-slave synchronization based on the binary log only is solved, and the synchronization of the data definition language log is realized.