72 results about "Redo log" patented technology

For database high availability and for accelerated recovery of a failed replica of a database, a storage computer is dynamically allocated and temporarily persists database content modifications until the database replica is ready to receive the modifications. The storage computer is not allocated storage that stores the database. The storage computer persists a recent portion of the database and later receives a request to synchronize the recovering replica. During recovery, the storage computer responsively sends the portion of the database to the recovering replica. For acceleration, recovery herein does not entail content interpretation such as replay of a redo log. For horizontally scaled acceleration involving two distinct storage computers per recovering replica, multiple replicas are concurrently recovered by respective storage computers that each receives recovered database content only from a respective distinct other storage computer.

The invention provides a data processing method and a related device, which can be applied to a database flash-back scene. The method comprises the steps of receiving a first flashback instruction, and recording a first page to a flashback log before the first page in a database is modified for the first time in a target period in response to the first flashback instruction. And receiving a second flash-back instruction, and in response to the second flash-back instruction, restoring the database to a state at a specified moment based on the flash-back log and the redo log of the database. In the application, the first page is recorded in the flash-back log only before the first page is modified for the first time in the target period. In other words, for the first page with the modification operation, only one initial page is recorded in the target period, and the subsequent modification version in the target period is not recorded, so that the overhead of the disk space can be effectively reduced.

The invention discloses a data processing method, a data processing device, electronic equipment and a readable storage medium, and belongs to the technical field of computers. The method comprises the following steps: in response to triggering of a data backup event aiming at a target database, backing up corresponding first incremental data from a first log serial number to a second log serial number, the first log serial number is a page tracking starting log serial number for starting record change data corresponding to the previous data backup operation of the target database, and the second log serial number is a log serial number of a first check point after the data backup event is triggered; and in response to completion of backup of the first incremental data, backing up the Redo log.

The invention discloses a database request scheduling method and device and computer equipment. The method comprises the following steps: receiving a database request, and determining the type of the database request; the priority of the database request is determined according to the type of the database request, the priority of the database priority volume write request in the database request is the highest, and the priority volume at least comprises a redo log file; and under the condition that the database load exceeds a preset threshold value, scheduling the database request according to the priority. According to the method and the device, the technical problem of relatively low database log writing efficiency caused by relatively high database redo log writing time delay in related technologies is solved.

The embodiment of the invention belongs to the technical field of data processing and the field of medical treatment and health, and relates to a data management method applied to a distributed storage system and related equipment.The method comprises the steps that the storage system is constructed according to a Bloom filter and a consistent Hash algorithm, and when data needs to be updated, the Bloom filter and the consistent Hash algorithm are updated; and the storage layer is responsible for changing a stored database table application and performing distributed execution on a redo log. The Bloom filter and the consistency Hash algorithm are introduced, so that the storage space and the processing speed are optimized, and the consistency and the reliability of the data are enhanced through the version vector and a redo log mechanism.

The present application relates to the technical field of deduplication storage, and provides a fingerprint library recovery method, apparatus, computer device, readable storage medium and program product. The method includes: obtaining a target backup fingerprint library according to the backup fingerprint library closest to the target time at the target time; obtaining newly added data blocks from the closest backup time to the target time; obtaining invalid fingerprints from the closest backup time to the target time according to the fingerprints of the data blocks reclaimed by space in the space reclamation redo log after the closest backup time; processing the target backup fingerprint library according to the fingerprints of the newly added data blocks and the invalid fingerprints to obtain the target fingerprint library at the target time; the target fingerprint library is used for deduplication. By adopting this method, it is possible to make the target fingerprint library at the target time after recovery consistent with the data file at the target time and not contain invalid fingerprints under the condition of relatively high fingerprint library recovery efficiency.

The application discloses a data page recovery method and device, electronic equipment and a storage medium. The data page recovery method comprises the following steps: determining a data page cache of a first data page to which a target waiting event is directed, the target waiting event is a waiting event waiting for a target node in a database cluster to respond, and the target node comprises a fault node and / or an active node; releasing other data page caches in the active node except the data page cache of the first data page; and recovering a data page cache of a second data page according to a first redo log of the second data page, the second data page being a data page taking the active node as a recovery node, and the first redo log being recorded in a first log file in a disk. According to the method, other data page caches except the data page cache of the first data page are released, and then data page recovery is performed according to the first redo log of the second data page, so that the recovery efficiency of the data page in the shared storage database cluster is improved.

This invention relates to the field of data processing and discloses a database log archiving method, apparatus, device, and medium, comprising: identifying a target redo log file to be archived from the database; creating a preliminary redo log file with the same attributes as the target redo log file and empty content within the same logical disk as the target redo log file; moving the target redo log file to a preset archiving directory located within the same logical disk via MV operations; changing the filename and storage path of the preliminary redo log file to the original filename and storage path of the target redo log file via MV operations; and mounting the revised preliminary redo log file to the database. This invention solves the problems of high I / O consumption and low archiving efficiency in existing database log archiving technologies.

The invention discloses a duplication data deletion metadata management method based on a persistent memory. The method comprises the following steps: managing a global fingerprint table and a logic address to a fingerprint mapping cache in a DRAM (Dynamic Random Access Memory), and managing an additional redo log and a global mapping table in the persistent memory. The write-in path firstly writes metadata update into the DRAM and forms a transaction record, the transaction record is sequentially added to a persistent memory log in a group submission mode, persistence is aligned according to the read-write granularity of the persistent memory, and random write and lock competition is reduced; in the checkpoint stage, an aggregation strategy is dynamically selected according to the redundancy rate and the coverage write proportion obtained through online load detection, and therefore metadata write amplification is remarkably reduced on the premise that collapse consistency is guaranteed. Meanwhile, three self-adaptive deduplication modes are provided, fingerprint management is bypassed in the low redundancy stage, redundancy is sampled and estimated in the medium redundancy stage, and full deduplication is started in the high redundancy stage, so that the metadata management cost is stably reduced under different loads, and the system throughput is improved.

The invention provides a data processing method and device, electronic equipment and a storage medium, relates to the technical field of data processing such as data storage and data analysis, and is suitable for cloud computing and database scenes. According to the specific implementation scheme, the method comprises the steps that a column redo log is pulled from a log server through a column storage engine, and the column redo log is written into the log server through a main computing node; the column redo log is analyzed and played back, first target data for data operation in a data table is determined, and the data table comprises a column storage index; and storing the first target data according to columns.

Methods, systems, and computer-readable storage media for receiving, by a SSD from a host, a write command to write content to memory, the content including a redo log of a database system, the redo log including a set of redo entries, each redo entry recording one or more changes to data in the database system, writing, by a redo log processor executed by the SSD, redo entries of the redo log to a redo log buffer, the redo log buffer comprising a page identifier (page ID) ordered array and a set of bidirectional linked tables, the redo entries being written to the bidirectional linked tables in the set of bidirectional linked tables, and executing, by the redo log processor, one or more of creating a data page and updating a data page in memory of the SSD, the data page being stored in memory of the SSD.

The application discloses a kind of database continuous log backup method, system, device and storage medium, belong to log backup technical field, comprising: query archive log list, copy all archive log files;Copy all redo log files, generate redo log copy file, obtain the path of the redo log file being written currently and the offset of the file, read the current redo log content from the start position to the offset position, overwrite the corresponding copy file, to calibrate the current redo log copy file content;If offset position changes, then read the content between last offset and current offset from current redo log file, overwrite to the corresponding copy file, the database continuous log backup method, system, device and storage medium do not need to install drive, have no performance influence to system and application, will not cause system crash downtime, without additional preparation target machine in advance, reduce implementation cost.

A method and system for data synchronization between a standby site and a primary site in a distributed database system is provided. A first primary site included in a primary site group of the distributed database system generates a commit action redo log message for a commit action performed by the first primary site on a first database transaction, the commit action redo log message including a transaction ID of the first database transaction and a transaction order indicator representing an order of database transactions in a group of database transactions executed on one or more primary sites in the primary site group. The first primary site sends the commit action redo log message to a corresponding first standby site backing up the first primary site.

Methods, systems, and computer-readable storage media for receiving, by a SSD from a host, a write command to write content to memory, the content including a redo log of a database system, the redo log including a set of redo entries, each redo entry recording one or more changes to data in the database system, writing, by a redo log processor executed by the SSD, redo entries of the redo log to a redo log buffer, the redo log buffer comprising a page identifier (page ID) ordered array and a set of bidirectional linked tables, the redo entries being written to the bidirectional linked tables in the set of bidirectional linked tables, and executing, by the redo log processor, one or more of creating a data page and updating a data page in memory of the SSD, the data page being stored in memory of the SSD.

The application discloses a kind of distributed database two-stage transaction submission method and device based on RDMA, it is related to distributed database high-performance transaction processing technical field, comprising: in pre-commitment stage, data jump is completed using RDMA unilateral writing operation;Through polling, transaction redo log data is written successfully;In the submission stage, the atomic update submission state flag bit is carried out through the RDMA atomic comparison and exchange operation;When the number of successful times reaches the majority threshold value through polling RDMA completion queue, exit waiting loop and execute consensus achievement processing. It can eliminate the multiple network round trip and software processing delay in traditional protocol, so that the end-to-end transaction delay is reduced from the software protocol level to the hardware network level, the system CPU resources are released so that it does not need to participate in the protocol processing, so that the computing resources can be efficiently utilized, and the overall throughput and concurrent processing capacity of the distributed cluster are significantly improved.

The present application provides a data writing method, a database system, an electronic device, and a storage medium. According to an embodiment of the present application, when creating a redo log (Redo Log), physical storage space is pre-allocated for the redo log, initial data is configured, and a mapping relationship is established. After a mapping relationship is determined between the redo log and the physical storage space, when the redo log is subsequently modified and updated, on the one hand, the file size will not be frequently modified, and on the other hand, there is no need to frequently establish a mapping relationship, thereby reducing the time overhead when writing the redo log, improving the writing speed of the redo log, and further optimizing the overall performance of the database, thereby reducing the problem of slow data writing speed affecting the access experience of the database system.

The invention provides an incremental data acquisition method and device, a storage medium and a program product, and the method comprises the steps: determining the name of a target PDB of to-be-acquired incremental data by responding to an incremental data acquisition instruction for acquiring any PDB; according to the name of the target PDB, the incremental SQL data of the target PDB is extracted from the redo log of the source end CDB corresponding to the target PDB, so that useless incremental SQL data can be prevented from being obtained; according to the incremental SQL data of the target PDB, the meta-information of the target data table related to the incremental SQL data is obtained, the incremental data of the target PDB comprises the incremental SQL data and the meta-information of the target data table, only the target PDB needs to be connected, the meta-information of the target data table is obtained from the target PDB, the efficiency of obtaining the incremental data of the specified PDB is improved, and then the PDB incremental data synchronization efficiency is improved.

In some embodiments, there may be provided extracting, from a redo log, a first chunk of data; checking whether a population indicator is set; in response to the first chunk of data, creating a main buffer; in response to a population indicator not set, writing the first chunk of data to the created main buffer and repeat the writing for one or more additional chunks until a last chunk is extracted; and after the last chunk is written to the main buffer, reading the main buffer to write the first chunk of data, the one or more additional chunks, and the last chunk to main memory. Related systems, methods, and articles of manufacture are also disclosed.

The invention relates to the technical field of duplicated data deletion and storage, and provides a fingerprint database recovery method and device, computer equipment, a readable storage medium and a program product. The method comprises the following steps: obtaining a target backup fingerprint database according to a backup fingerprint database closest to a backup moment of a target moment; acquiring a newly added data block closest to the backup moment to the target moment; according to the fingerprints of the space recovery data blocks in the space recovery redo log closest to the backup moment, invalid fingerprints from the closest backup moment to the target moment are obtained; according to the fingerprints of the newly added data blocks and the invalid fingerprints, processing the target backup fingerprint database to obtain a target fingerprint database at the target moment; the target fingerprint database is used for deleting repeated data. By adopting the method, under the condition that the fingerprint database recovery efficiency is relatively high, the recovered target fingerprint database at the target moment is consistent with the data file at the target moment, and invalid fingerprints are not included.

As with database tables in general, a streaming table and its columns are defined within by a database. DML commands are issued against a streaming table that specify to insert rows and values into their defined columns. Change records in a change log (e.g., redo log) are generated to record the changes specified by DML commands. The change log may be used to stream data captured by the change log. However, unlike with database tables in general, the rows of a streaming table are not persistently stored in the database at commit time. Alternatively, rows are stored persistently but without values for columns defined as streaming columns. Columns defined as streaming columns may be set to values specified by DML commands but are not stored in the database; streaming column values are captured in change logs, however.

Methods and apparatus in a cloud-based database management system are described. Data in a database is stored in a plurality of pages within a page storage area of the database. A plurality of redo log records are received to apply to the database. The redo log records within a predefined boundary are parsed to determine, for each given redo log record, a corresponding page to which the given log record is to be applied. The redo log records are reordered by corresponding page. The reordered redo log records are stored for application to the page storage area of the database.

Herein is database transaction throttling to compensate for excessive redo generation and excessive database replication lag. Each time interval in a sequence of contiguous time intervals is assigned to a distinct respective redo counter in a circular buffer. While executing database statements in a database transaction, some of the redo counters are adjusted to reflect a fluctuating amount of redo generated by the transaction. When the redo generation rate of the database transaction is detected as exceeding a threshold, execution of the database statements in the database transaction is paused.

A method, device and system for building a master and standby database can be used in the financial field or other fields. The method includes: obtaining the IP address of the master database server and the standby database server, determining the file size and number of the master database instance information, directory list and online redo log; creating a standby database redo log in the master database according to the file size and the number of files; generating a connection string and adding it to the descriptor configuration file of the master database server; obtaining a password file, a parameter file and a descriptor configuration file from the master database, and synchronizing them to the standby database according to the standby database server IP; creating a standby database directory list in the standby database according to the master database directory list, and storing each file under the standby database directory list, so that the standby database obtains the standby database redo log from the master database and copies the data. The present invention reduces the professional threshold for building master and standby databases and improves the accuracy, efficiency and quality of daily operation and maintenance.

The invention relates to a dynamic data dictionary acquisition method based on Oracle incremental data. The method comprises the steps of querying a mapping relation between a table name and a table ID and a mapping relation between a table field number and a field name through a metadata table of a database, and constructing an initial data dictionary structure; carrying out Oracle incremental data acquisition through an SQL (Structured Query Language) interface of the logminer component, and extracting an incremental DDL (Double Data Language) redo statement; analyzing and processing the increment DDL redo statement by using an SQL grammar analyzer to obtain table structure change information; and issuing a structure change event according to the table structure change information, receiving the structure change event by the dictionary maintainer, and merging the original data dictionary with newly received change data to obtain a dynamically updated new data dictionary. According to the method, the defects that an online directory and an exported data dictionary file require that a data dictionary cannot be changed and a redo log file is low in performance are overcome, the limitation that the data dictionary cannot be changed can be avoided, and the performance of the data dictionary can be prevented from being greatly reduced.

The invention provides a virtual file system, a binlog-in-redo implementation method, electronic equipment and a readable medium, and relates to the technical field of databases. According to the scheme, a file management service responds to a redo log persistence request transparently transmitted by a database service layer through a virtual redo log service, and persistence operation is executed on physical files which are written into a redo log file block and a binlog file block in a staggered mode; the file management service also manages a shared memory, and the shared memory comprises a redo log cache and a binlog cache; the virtual redo log service provides an access path of the redo log cache, and the virtual binlog service provides an access path of the binlog cache. According to the scheme, writing of the redo log file block and writing of the binlog file block are combined into writing of the physical file, the transaction submission operation efficiency is improved, when it is confirmed that persistence of the binlog file block is completed, master-slave synchronization is triggered in an asynchronous notification mode, and the risk of master-slave inconsistency is avoided; the original storage format can be displayed to the outside through the virtual service, seamless transparent compatibility is achieved, the customization requirement is lowered, and the deployment cost is reduced.

The invention discloses a redo log parallel analysis method and device based on heterogeneous database data synchronization. According to the method, parallel acquisition load balancing is realized through an archived file granularity self-adaptive splitting and cross-boundary transaction detection mechanism; a dynamic task scheduling strategy sorted according to FIRSTCHANGE # is adopted, an event chain table is maintained in combination with a memory mapping file, and accurate merging of multi-thread event streams is achieved through SCN global comparison; and dynamically switching a serial-parallel analysis mode according to redo update frequency, automatically degenerating serial and continuously monitoring a log switching state in a high-frequency change scene, and ensuring data continuity through a site backspacing and SKIP condition filtering mechanism. According to the method, the throughput capacity and the real-time performance of incremental synchronization of the heterogeneous database are remarkably improved, data loss and repetition caused by log switching are effectively avoided, and the data consistency and the time sequence accuracy are ensured while system parallel computing resources are fully utilized.

The invention is suitable for the technical field of databases, and relates to a log analysis method and device, computer equipment and a computer readable medium. Wherein a logic log record corresponding to a transaction is obtained in response to analysis of the redo log, and whether the current time reaches a check point triggering period or not is recognized; if the current time reaches the check point triggering period, determining a target active transaction which is started earliest from the active transactions, and taking a redo log starting point corresponding to the target active transaction and a system logic clock corresponding to the logic log record as candidate check points for associative storage; responding to a received application system logic clock returned by the log application end, and determining a target candidate check point matched with the application system logic clock in the stored candidate check points; and configuring a redo log starting point in the target candidate check points as a formal check point of the redo log. According to the invention, the memory overhead required for promoting the check point can be reduced.

Provided in the present application are a log backup method and apparatus, and a storage medium and a computer program product. The method comprises: on the basis of number information of a copied redo log, determining first number information expected to be read next time; sequentially reading number information of redo logs before the redo logs are written into a disk, and when it is found, by means of reading, that second number information of a first redo log is greater than the first number information, storing the first redo log in an archive file; and after the storage is completed, backing up redo logs in the archive file to a backup node.

Embodiments in the present description provide an operation request processing method and apparatus, and a device, and a readable storage medium, and a system. In the present embodiment, after obtaining an operation request for a data table, a read-write node can return a request processing completion message to a requester in time without waiting for all read-only nodes; a locking operation is performed on the data table, a log of locking the data table is written into a redo log file; and after status information of the log of locking is written into a status record, a monitoring processing process can achieve log playback monitoring of all the read-only nodes by using the status record and the log of locking. Thus, an asynchronous processing mechanism is achieved, and the stability and efficiency of operation request execution are improved.

The invention provides a cloud native database backup and recovery method and device, electronic equipment and a medium, and belongs to the technical field of databases, the method comprises the steps of generating a log event based on a data write-in statement, and writing the log event into an RODO log file; writing the corresponding data page into a buffer pool based on the data writing statement; refreshing the modified data page in the buffer pool into a data file on a data persistence layer of the cloud native database, and generating a data snapshot chain for backup based on the RODO log file and the data file on the data persistence layer; and under the condition that the cloud native database needs to be recovered, recovering the cloud native database to a state when the snapshot is created based on the flash-back snapshot. According to the method, a snapshot storage technology and support of a front-mounted and rear-mounted real-time notification mechanism for database REDO log real-time disk refreshing are adopted, a database snapshot flash-back function is achieved, and the technical problem that an existing database cannot solve the real-time problem and the technical problem that restoration at a specified time point cannot be achieved at the same time is solved.