43 results about "Redo log" patented technology

For database high availability and for accelerated recovery of a failed replica of a database, a storage computer is dynamically allocated and temporarily persists database content modifications until the database replica is ready to receive the modifications. The storage computer is not allocated storage that stores the database. The storage computer persists a recent portion of the database and later receives a request to synchronize the recovering replica. During recovery, the storage computer responsively sends the portion of the database to the recovering replica. For acceleration, recovery herein does not entail content interpretation such as replay of a redo log. For horizontally scaled acceleration involving two distinct storage computers per recovering replica, multiple replicas are concurrently recovered by respective storage computers that each receives recovered database content only from a respective distinct other storage computer.

The invention discloses a data processing method, a data processing device, electronic equipment and a readable storage medium, and belongs to the technical field of computers. The method comprises the following steps: in response to triggering of a data backup event aiming at a target database, backing up corresponding first incremental data from a first log serial number to a second log serial number, the first log serial number is a page tracking starting log serial number for starting record change data corresponding to the previous data backup operation of the target database, and the second log serial number is a log serial number of a first check point after the data backup event is triggered; and in response to completion of backup of the first incremental data, backing up the Redo log.

The application discloses a data page recovery method and device, electronic equipment and a storage medium. The data page recovery method comprises the following steps: determining a data page cache of a first data page to which a target waiting event is directed, the target waiting event is a waiting event waiting for a target node in a database cluster to respond, and the target node comprises a fault node and / or an active node; releasing other data page caches in the active node except the data page cache of the first data page; and recovering a data page cache of a second data page according to a first redo log of the second data page, the second data page being a data page taking the active node as a recovery node, and the first redo log being recorded in a first log file in a disk. According to the method, other data page caches except the data page cache of the first data page are released, and then data page recovery is performed according to the first redo log of the second data page, so that the recovery efficiency of the data page in the shared storage database cluster is improved.

This invention relates to the field of data processing and discloses a database log archiving method, apparatus, device, and medium, comprising: identifying a target redo log file to be archived from the database; creating a preliminary redo log file with the same attributes as the target redo log file and empty content within the same logical disk as the target redo log file; moving the target redo log file to a preset archiving directory located within the same logical disk via MV operations; changing the filename and storage path of the preliminary redo log file to the original filename and storage path of the target redo log file via MV operations; and mounting the revised preliminary redo log file to the database. This invention solves the problems of high I / O consumption and low archiving efficiency in existing database log archiving technologies.

The invention discloses a duplication data deletion metadata management method based on a persistent memory. The method comprises the following steps: managing a global fingerprint table and a logic address to a fingerprint mapping cache in a DRAM (Dynamic Random Access Memory), and managing an additional redo log and a global mapping table in the persistent memory. The write-in path firstly writes metadata update into the DRAM and forms a transaction record, the transaction record is sequentially added to a persistent memory log in a group submission mode, persistence is aligned according to the read-write granularity of the persistent memory, and random write and lock competition is reduced; in the checkpoint stage, an aggregation strategy is dynamically selected according to the redundancy rate and the coverage write proportion obtained through online load detection, and therefore metadata write amplification is remarkably reduced on the premise that collapse consistency is guaranteed. Meanwhile, three self-adaptive deduplication modes are provided, fingerprint management is bypassed in the low redundancy stage, redundancy is sampled and estimated in the medium redundancy stage, and full deduplication is started in the high redundancy stage, so that the metadata management cost is stably reduced under different loads, and the system throughput is improved.

The invention provides a data processing method and device, electronic equipment and a storage medium, relates to the technical field of data processing such as data storage and data analysis, and is suitable for cloud computing and database scenes. According to the specific implementation scheme, the method comprises the steps that a column redo log is pulled from a log server through a column storage engine, and the column redo log is written into the log server through a main computing node; the column redo log is analyzed and played back, first target data for data operation in a data table is determined, and the data table comprises a column storage index; and storing the first target data according to columns.

The application discloses a kind of database continuous log backup method, system, device and storage medium, belong to log backup technical field, comprising: query archive log list, copy all archive log files;Copy all redo log files, generate redo log copy file, obtain the path of the redo log file being written currently and the offset of the file, read the current redo log content from the start position to the offset position, overwrite the corresponding copy file, to calibrate the current redo log copy file content;If offset position changes, then read the content between last offset and current offset from current redo log file, overwrite to the corresponding copy file, the database continuous log backup method, system, device and storage medium do not need to install drive, have no performance influence to system and application, will not cause system crash downtime, without additional preparation target machine in advance, reduce implementation cost.

A method and system for data synchronization between a standby site and a primary site in a distributed database system is provided. A first primary site included in a primary site group of the distributed database system generates a commit action redo log message for a commit action performed by the first primary site on a first database transaction, the commit action redo log message including a transaction ID of the first database transaction and a transaction order indicator representing an order of database transactions in a group of database transactions executed on one or more primary sites in the primary site group. The first primary site sends the commit action redo log message to a corresponding first standby site backing up the first primary site.

Methods, systems, and computer-readable storage media for receiving, by a SSD from a host, a write command to write content to memory, the content including a redo log of a database system, the redo log including a set of redo entries, each redo entry recording one or more changes to data in the database system, writing, by a redo log processor executed by the SSD, redo entries of the redo log to a redo log buffer, the redo log buffer comprising a page identifier (page ID) ordered array and a set of bidirectional linked tables, the redo entries being written to the bidirectional linked tables in the set of bidirectional linked tables, and executing, by the redo log processor, one or more of creating a data page and updating a data page in memory of the SSD, the data page being stored in memory of the SSD.

The application discloses a kind of distributed database two-stage transaction submission method and device based on RDMA, it is related to distributed database high-performance transaction processing technical field, comprising: in pre-commitment stage, data jump is completed using RDMA unilateral writing operation;Through polling, transaction redo log data is written successfully;In the submission stage, the atomic update submission state flag bit is carried out through the RDMA atomic comparison and exchange operation;When the number of successful times reaches the majority threshold value through polling RDMA completion queue, exit waiting loop and execute consensus achievement processing. It can eliminate the multiple network round trip and software processing delay in traditional protocol, so that the end-to-end transaction delay is reduced from the software protocol level to the hardware network level, the system CPU resources are released so that it does not need to participate in the protocol processing, so that the computing resources can be efficiently utilized, and the overall throughput and concurrent processing capacity of the distributed cluster are significantly improved.

In some embodiments, there may be provided extracting, from a redo log, a first chunk of data; checking whether a population indicator is set; in response to the first chunk of data, creating a main buffer; in response to a population indicator not set, writing the first chunk of data to the created main buffer and repeat the writing for one or more additional chunks until a last chunk is extracted; and after the last chunk is written to the main buffer, reading the main buffer to write the first chunk of data, the one or more additional chunks, and the last chunk to main memory. Related systems, methods, and articles of manufacture are also disclosed.

As with database tables in general, a streaming table and its columns are defined within by a database. DML commands are issued against a streaming table that specify to insert rows and values into their defined columns. Change records in a change log (e.g., redo log) are generated to record the changes specified by DML commands. The change log may be used to stream data captured by the change log. However, unlike with database tables in general, the rows of a streaming table are not persistently stored in the database at commit time. Alternatively, rows are stored persistently but without values for columns defined as streaming columns. Columns defined as streaming columns may be set to values specified by DML commands but are not stored in the database; streaming column values are captured in change logs, however.

Methods and apparatus in a cloud-based database management system are described. Data in a database is stored in a plurality of pages within a page storage area of the database. A plurality of redo log records are received to apply to the database. The redo log records within a predefined boundary are parsed to determine, for each given redo log record, a corresponding page to which the given log record is to be applied. The redo log records are reordered by corresponding page. The reordered redo log records are stored for application to the page storage area of the database.

Herein is database transaction throttling to compensate for excessive redo generation and excessive database replication lag. Each time interval in a sequence of contiguous time intervals is assigned to a distinct respective redo counter in a circular buffer. While executing database statements in a database transaction, some of the redo counters are adjusted to reflect a fluctuating amount of redo generated by the transaction. When the redo generation rate of the database transaction is detected as exceeding a threshold, execution of the database statements in the database transaction is paused.

The invention provides a virtual file system, a binlog-in-redo implementation method, electronic equipment and a readable medium, and relates to the technical field of databases. According to the scheme, a file management service responds to a redo log persistence request transparently transmitted by a database service layer through a virtual redo log service, and persistence operation is executed on physical files which are written into a redo log file block and a binlog file block in a staggered mode; the file management service also manages a shared memory, and the shared memory comprises a redo log cache and a binlog cache; the virtual redo log service provides an access path of the redo log cache, and the virtual binlog service provides an access path of the binlog cache. According to the scheme, writing of the redo log file block and writing of the binlog file block are combined into writing of the physical file, the transaction submission operation efficiency is improved, when it is confirmed that persistence of the binlog file block is completed, master-slave synchronization is triggered in an asynchronous notification mode, and the risk of master-slave inconsistency is avoided; the original storage format can be displayed to the outside through the virtual service, seamless transparent compatibility is achieved, the customization requirement is lowered, and the deployment cost is reduced.

The invention discloses a redo log parallel analysis method and device based on heterogeneous database data synchronization. According to the method, parallel acquisition load balancing is realized through an archived file granularity self-adaptive splitting and cross-boundary transaction detection mechanism; a dynamic task scheduling strategy sorted according to FIRSTCHANGE # is adopted, an event chain table is maintained in combination with a memory mapping file, and accurate merging of multi-thread event streams is achieved through SCN global comparison; and dynamically switching a serial-parallel analysis mode according to redo update frequency, automatically degenerating serial and continuously monitoring a log switching state in a high-frequency change scene, and ensuring data continuity through a site backspacing and SKIP condition filtering mechanism. According to the method, the throughput capacity and the real-time performance of incremental synchronization of the heterogeneous database are remarkably improved, data loss and repetition caused by log switching are effectively avoided, and the data consistency and the time sequence accuracy are ensured while system parallel computing resources are fully utilized.

The invention is suitable for the technical field of databases, and relates to a log analysis method and device, computer equipment and a computer readable medium. Wherein a logic log record corresponding to a transaction is obtained in response to analysis of the redo log, and whether the current time reaches a check point triggering period or not is recognized; if the current time reaches the check point triggering period, determining a target active transaction which is started earliest from the active transactions, and taking a redo log starting point corresponding to the target active transaction and a system logic clock corresponding to the logic log record as candidate check points for associative storage; responding to a received application system logic clock returned by the log application end, and determining a target candidate check point matched with the application system logic clock in the stored candidate check points; and configuring a redo log starting point in the target candidate check points as a formal check point of the redo log. According to the invention, the memory overhead required for promoting the check point can be reduced.

Provided in the present application are a log backup method and apparatus, and a storage medium and a computer program product. The method comprises: on the basis of number information of a copied redo log, determining first number information expected to be read next time; sequentially reading number information of redo logs before the redo logs are written into a disk, and when it is found, by means of reading, that second number information of a first redo log is greater than the first number information, storing the first redo log in an archive file; and after the storage is completed, backing up redo logs in the archive file to a backup node.

The embodiment of the invention provides a redo log processing method and device, a storage medium and a program product, and relates to the field of distribution. The method comprises the steps that metadata information of a redo log file in an ASM storage layer is obtained through an ASM API interface of an automatic storage management system; wherein the ASM API interface is used for accessing an ASM storage layer, the redo log file comprises a log file used for recording database change operation, and the metadata information at least comprises a file path, a block number, a block size and a file type; according to the metadata information, performing block reading on the redo log file through an ASM API interface to obtain redo log data read in a block manner; and converting the redo log data read in blocks into a local redo log format to obtain a local redo log. According to the method, the capability of reading and analyzing the redo log file from the ASM storage layer is increased, and the efficiency of fault recovery in database operation and maintenance is improved.

The application provides a data page processing method and device, electronic equipment and a storage medium, wherein the method comprises: parsing redo logs of a first batch to obtain a plurality of application tasks, a target copy page containing a target original page in a target memory space, the target original page being a data page associated with a target operation parsed from the redo logs of the first batch, the target operation being a split or merge operation of the data page; distributing the plurality of application tasks to a plurality of worker threads, each worker thread in the plurality of worker threads being used for processing part of the plurality of application tasks; executing the plurality of application tasks through the plurality of worker threads, an application task corresponding to the target original page in the plurality of application tasks being applied to the target copy page; and in the case that the plurality of application tasks are executed, updating data of the target copy page to the target original page by adding an exclusive lock to a target index, the target index being an index in which the target original page is located.

The application discloses a database flashback implementation method based on a value log system, and comprises the following steps: a working thread / process receives a user library flashback request; the working thread / process executes a suspended database or data processing system; the suspended working thread stops providing services externally, and the current working thread / process is excepted; an instance is set as a suspended mode; the working thread / process determines a flashback point of the database flashback; the working thread / process truncates the value log based on the determined flashback point, and sets the tail of the value log block to zero, so that the new log reuses the value log storage space; the working thread / process cleans up invalid records in the index based on the determined flashback point, that is, the index records after the flashback point; the working thread / process brings other working threads / processes online, reopens the instance as a normal mode, and returns a user end flashback success. The application supports the flashback to a specified flashback point, does not need additional flashback logs and REDO logs, and the value log supports the library-level flashback originally.

A data write minimization apparatus according to one embodiment may perform caching a page stored in a storage device into a buffer of a volatile memory; generating, when a data update transaction occurs for the page, a PPL block including a per-page log (PPL), which is a per-page redo log, for the data update transaction in a non-volatile memory, and storing, when the PPL block grows larger than a preset size, an entire page in the storage device; and maintaining, when a page write occurs for a PPL page in which the redo log is stored as a PPL block in the non-volatile memory, the PPL block stored in the non-volatile memory, and performing, when a page write occurs for a normal page, an entire page write to the storage device.

The application provides a database redo log parallel processing method and related products, wherein the database redo log parallel processing method comprises the following steps: a transaction process of a master database writes generated redo log data into a common log buffer; the transaction process awakens a log disk flushing process and a stream replication sending process of the master database in parallel; the log disk flushing process writes the redo log data to be persisted from the common log buffer to a disk log file of the master database; and the stream replication sending process sends the persisted redo log data from the common log buffer to a slave database. According to the scheme, the working starting point of the stream replication sending process is advanced from after the log disk flushing to during the log disk flushing, the overlap of network transmission time and disk I / O time is realized, the redo log data can flow to the slave database in real time, the replication delay between the master-slave cluster is shortened, and the real-time availability of data is improved.

The invention discloses an HNSW index data management method and device, equipment and a medium, and the method comprises the steps: obtaining a target node corresponding to a to-be-executed transaction, and carrying out the target operation of a storage position corresponding to the target node according to the to-be-executed transaction, and generating a target redo log and a target revocation log corresponding to the target operation and version transaction data of the to-be-executed transaction for performing the target operation on the target node, before the to-be-executed transaction is completed, executing a row lock on a corresponding storage position in the target node, and according to the to-be-executed transaction, performing an updating operation on a neighbor relationship between the neighbor node and the target node. And generating a neighbor redo log and / or a neighbor revocation log corresponding to the update operation, and before the to-be-executed transaction is completed, not limiting the access to the neighbor relationship of the neighbor node. The HNSW index data is decoupled and classified, and a differentiated transaction processing mechanism is adopted, so that good balance is achieved between guarantee of global strong consistency and reduction of neighbor relationship transaction management overhead, and performance and consistency are effectively considered.

Some embodiments provide a non-transitory machine-readable medium that stores a program to write a redo log for operations performed to modify data within a database. The program sets a throttle mass modification flag when a ring list of log buffers is full and unable to write an additional log entry at this point in time. When the throttle mass modification flag is set, the writing of the redo log for mass operations is blocked or delayed, thereby reducing the delay experienced when writing user operations into the redo log. This can improve the user experience.

The invention discloses a distributed database two-stage transaction submission method and device based on RDMA (Remote Direct Memory Access), and relates to the technical field of distributed database high-performance transaction processing. The method comprises the following steps: completing data jump by utilizing RDMA unilateral write operation in a pre-submission stage; confirming that the transaction redo log data is written successfully through polling; in the submission stage, the submission state flag bit is updated through RDMA atom comparison and operation atom exchange; and when the successful number of queue statistics completed by polling RDMA reaches a majority threshold, jumping out to wait for loop execution of consensus achievement processing. According to the method, multi-round network round-trip and software processing delay in a traditional protocol can be eliminated, end-to-end transaction delay is reduced to a hardware network level from a software protocol level, system CPU resources are released and do not need to participate in protocol processing, computing resources can be efficiently utilized, and the overall throughput and concurrent processing capacity of a distributed cluster are remarkably improved.

The invention relates to a data processing method and device, computer equipment, a storage medium and a computer program product. The method comprises the following steps: when a main database node in a database node group is subjected to data change, determining a change result, and generating a redo log containing the change result; writing the redo log into a first log file in a disk of the main database node, and writing the redo log into the main database node for synchronous caching; searching at least one to-be-synchronized log corresponding to the slave database node from a synchronous cache; when the query fails, querying at least one to-be-synchronized log corresponding to the slave database node in the first log file; and sending the at least one to-be-synchronized log to the slave database node to indicate the slave database node to store the at least one to-be-synchronized log to the second log file, and updating the data file of the slave database node according to the second log file. By adopting the method, the synchronization efficiency can be improved.

The application provides an automatic data recovery method of a database cluster and a storage medium. The automatic data recovery method comprises the following steps: acquiring a master database signal of the database cluster; acquiring a starting point of a redo log stream returned by a switching target database and a terminal point of a redo log stream returned by a switching source database in the case that the switching target database and the switching source database are both normally connected with a trusted gateway; setting a redo recovery point of a backup database; and determining a recovery data area according to the starting point of the redo log stream returned by the switching target database, the terminal point of the redo log stream returned by the switching source database and the redo recovery point of the backup database, so as to realize automatic data recovery.

The application discloses a REDO log replay method and device, equipment and a storage medium. The method comprises the following steps: when a current node is restarted after a failure, a REDO log file in the current node is acquired, wherein the REDO log file comprises at least one REDO log package, and each REDO log package carries a database running mode identifier; the replay rule corresponding to each REDO log package is determined according to the database running mode identifier carried by each REDO log package, so as to replay the REDO log package according to the replay rule corresponding to each REDO log package. Through the technical scheme of the application, the problem of replaying the REDO log under a shared storage cluster data daemon environment is solved, the data can be restored to the state before the node failure, and the correctness of the data is ensured.

The application discloses a database-oriented batch insertion transaction processing method and device, equipment and medium, relates to the technical field of database, and includes the following steps: when a database batch insertion transaction is monitored, a shared lock and an intention exclusive lock are applied to an index organization table, and a mark is set for the transaction and the index organization table; a first physical record ROWID and a rollback record of the transaction are generated, a data page is allocated in an index organization table storage space, batch data is inserted, only a redo log of the first record of the data page is generated, and batch insertion is completed. If the transaction is marked, the redo log and the data page are written to a disk, and the mark is reset to commit the transaction. When the transaction is rolled back, the ROWID is obtained according to the rollback record, the user record and the data page where the user record is located are located in the index organization table, the internal positioning record of the inner node is found according to the user record and is deleted, and the corresponding data page is released. The application can reduce the generation of redo logs to the maximum extent on the basis of guaranteeing data persistence.