35 results about "Log sequence number" patented technology

The invention provides a log data reading method and device, equipment, a medium and a program product, and relates to the field of big data. Obtaining a mapping relationship between a plurality of timeline identifiers in a database and corresponding start log serial numbers; the timeline identifier is an identifier generated when the main database and the standby database are switched, whether the log data cross different timeline identifiers or not is judged according to the mapping relation and the starting log serial number and the ending log serial number of the log data, and under the condition that the log data cross different timeline identifiers, the log data cross different timeline identifiers. Reconstructing a log reading process according to a start log serial number of a new timeline identifier, wherein the new timeline identifier is different from the plurality of timeline identifiers; according to the method, automatic reconstruction of the log reading process after main and standby switching is achieved, the problems of low efficiency and poor real-time performance caused by manual operation are avoided, and continuity and consistency of data synchronization of the source end and the target end are guaranteed.

Techniques for volume durable log sequence number movement in a multi-volume database environment are described. Blocks of database updates are persisted in an atomic and durable manner, where the blocks may include updates to a first volume, a second volume, or both. Backlinks from one block of updates to the previous block of updates may be set, in addition to backlinks between updates to the first volume and backlinks between updates to the second volume. Upon durably persisting a block, the block backlinks can be followed to help verify that all contiguous blocks—of changes to one or both volumes—have all been persisted. Thereafter, a volume durable log sequence number (VDL) pointer value can be updated to point to a known “last” durably persisted log update.

Support for distributed database backtracking, in which the database is reverted or iteratively reverted to a previous state, is provided. Log sequence number ranges and storage offset ranges for use in reconstructing the previous state are determined and used in the reconstruction. Clock sampling records can be generated indicating a feasible backtracking window with time points indicating allowable times to which backtracking can be performed. Appropriate log records, snapshots and checkpoints for database states are identified and organized with backtracking information record and timeline. A minimal and necessary set of log records, snapshots and checkpoints enabling database reconstruction within a time window is maintained. Measures are taken to maintain the necessary data to support backtracking even after prior backtrackings have occurred in the backtracking time window.

The invention discloses a pre-writing type log service method and device, equipment and a storage medium, relates to the technical field of log services, is applied to a log server, and comprises the following steps: receiving a log recording request for a target log concurrently submitted by a state machine; based on the log recording request, generating a logic log serial number and a physical log serial number of the target log; in response to a restart playback request sent by the state machine, obtaining a mapping relationship between the logic log serial number and the physical log serial number; and based on the mapping relationship between the logical log serial number and the physical log serial number, playing back and applying the target log to the state machine through a logical log sequence sliding window. According to the method, the uniqueness and the sequence of the target log are ensured through the logic log serial number and the physical log serial number, and the fault tolerance and the throughput capacity are improved.

The application discloses a database pre-write log adaptive disk brushing scheduling method and device and medium, belongs to the technical field of database system data storage, and is used for solving the technical problems that the pre-write log disk brushing strategy of the existing database cannot adapt to different transaction load scenes, the disk synchronization overhead is high, the throughput capacity is limited, and the resource utilization rate is low. The method comprises the following steps: acquiring the transaction submission rate of a target database system within a preset time window; determining the pre-write log generation rate according to the log sequence number change amount of the target database system; predicting the system load state of the target database system based on the transaction submission rate and the pre-write log generation rate; dynamically matching the corresponding pre-write log disk brushing strategy according to the system load state; and controlling the target database system to perform the disk brushing operation according to the pre-write log disk brushing strategy.

The invention provides a log data processing method, device and equipment for a memory access node. The processing method comprises the following steps: acquiring a plurality of log threads which are started at present; dividing the plurality of log threads into N log thread groups according to the number N of non-uniform memory access nodes; obtaining a first preset number of target log file data to be operated by at least one log thread of the N consistent memory access nodes; distributing a global log sequence number for the first preset number of target log file data, wherein the global log sequence number is a log sequence number generated according to all log data in the N log thread groups; and according to a global log sequence number corresponding to the target log file data, writing a first preset quantity of target log file data to be operated by the at least one log thread into a local memory of the non-uniform memory access node. According to the embodiment of the invention, cross-node remote access can be avoided; and the memory access delay is obviously reduced.

A system and method for reconciling data from a local database streamed to a Cloud storage service is disclosed. A data transaction is received from a structured query language (SQL) database service operated by a server. The transaction includes log sequence numbers and corresponding data records. The log sequence numbers are stored in a change data capture (CDC) table. The sent data transaction is streamed via a data integration service. A storage object service stores the sent data into an object storage service bucket. A data reconciliation service deploys a cloud trail target to generate a log file of log sequence numbers from the sent data stored in the object storage service bucket. The log file of log sequence numbers is compared with the log sequence numbers in the CDC table. A missing data record is flagged to an exception table based on the comparison. An alert is issued for the missing data record.

Ensuring exactly once data ingestion, the method comprising: receiving, from an event stream, event data comprising a plurality of first records; generating a plurality of second records each corresponding to a first record of the plurality of first records and comprising a log sequence number (LSN) describing a location in the event stream for a corresponding first record; and sending, to a data warehouse, the plurality of second records.

Embodiments of the present application provide a database migration method, device, equipment, medium and program product. The method comprises: at the starting moment of migrating the inventory data of a source database to a target database, for any one of a plurality of tables to be migrated, based on a change data capture table, obtaining a first target log sequence number of a migration transaction corresponding to the table to be migrated; recording the first target log sequence number as low-water information; at the end moment of migrating the inventory data of the source database to the target database, based on the change data capture table, obtaining a second target log sequence number of a migration transaction corresponding to the table to be migrated; recording the second target log sequence number as high-water information; and based on the high-water information and the low-water information, synchronizing the incremental data of the source database to the target database. The method is used to improve the efficiency of database migration.

Support for distributed database backtracking, in which the database is reverted or iteratively reverted to a previous state, is provided. Log sequence number ranges and storage offset ranges for use in reconstructing the previous state are determined and used in the reconstruction. Clock sampling records can be generated indicating a feasible backtracking window with time points indicating allowable times to which backtracking can be performed. Appropriate log records, snapshots and checkpoints for database states are identified and organized with backtracking information record and timeline. A minimal and necessary set of log records, snapshots and checkpoints enabling database reconstruction within a time window is maintained. Measures are taken to maintain the necessary data to support backtracking even after prior backtrackings have occurred in the backtracking time window.

In a fault-tolerant method for event information delivery within a file system cluster, one or more processors determine event information associated with file system activity performed by nodes of the cluster. The one or more processors add the event information to an event log buffer in memory. The one or more processors receive a first log sequence number (LSN) associated with flushing recovery information from a recovery log buffer. The one or more processors determine event information in the event log buffer that has a log sequence number less than or equal to the first log sequence number, and determine that the event information includes a log sequence number less than or equal to the first log sequence number, and the one or more processors flush the corresponding event information from the event log buffer to disk storage.

A durability marker may be advanced according to ordered batches of log records. Different batches of log records may be sent to a distributed data store to update a log stored in the distributed data store. The different batches may be ordered according to a minimum Log Sequence Number (LSN) value within the different batches. The ordered different batches may be used to determine whether and how much to advance a durability marker for the log.

The invention relates to the technical field of database synchronization, and provides a data synchronization method and device based on database restoration, and the method comprises the steps that a source end synchronization tool obtains a backup file generated by a source end database; sending the backup file to a destination synchronization tool to enable the destination synchronization tool to obtain a reference database; obtaining a minimum log serial number corresponding to an active transaction in a source end database at the backup starting moment and a maximum log serial number corresponding to the backup finishing moment; reading an operation log of which the log serial number is greater than the minimum log serial number in a log file of the source end database, and modifying a submission operation of which the log serial number is less than or equal to the maximum log serial number into a rollback operation to obtain a target operation log, the target end synchronization tool carries out data synchronization on the target end database according to the reference database and the target operation log. According to the method, the consistency of the data is effectively maintained on the premise that the active transaction is automatically rolled back after the backup of the destination end database is restored.

The present disclosure relates to a timestamp-based log collection method, device, equipment and medium, the method comprising: reading a timestamp collection starting point added in a configuration file of a collection task; finding a target transaction log file based on the timestamp collection starting point; reading and analyzing the target transaction log file to determine a log sequence number (LSN) set in the configuration file; starting a collection replication stream based on the LSN, and collecting log data based on the replication stream. The present disclosure can determine the corresponding LSN by using the timestamp collection starting point, greatly reducing the difficulty of obtaining the LSN, reducing the professional requirements for users, and thus facilitating the execution of real-time collection tasks.

Examples described herein provide systems and methods for replicating unlogged data changes in a database management system are disclosed. The method involves identifying a load operation in a transaction log, determining a log sequence number (LSN) associated with the load operation, and identifying data pages in a source database with headers containing the load LSN. Data from these pages is extracted and saved to an external file. The target database is updated by loading the extracted data from the external file, allowing for incremental replication. This approach optimizes replication efficiency by avoiding a full refresh of the target table, reducing resource consumption, and enhancing system performance.

This application discloses a unified database waiting method, device, and medium based on WAL refresh events, belonging to the field of database technology. It addresses the problems of resource waste and unstable event response delays caused by existing polling mechanisms waiting for WAL refresh events. The method includes: establishing a WAL refresh event center in the database kernel; child processes registering global wait items with the WAL refresh event center; the WAL refresh event center arranging the registered wait items to generate a global wait queue; when WAL disk flushing is completed and a new log refresh position is generated, the WAL refresh event center wakes up the child processes corresponding to wait items in the global wait queue whose target log sequence number is not greater than the latest log refresh position; the awakened child processes obtain the current log refresh position, re-check the waiting conditions based on the current log refresh position, and execute business logic or re-enter the waiting state based on the check results.

The present application provides systems and methods that can generate an efficient architecture and method for multiple database engines to write to a shared data store to eliminate global locks during the write process. The systems and methods can use a common log layer between the shared data store and the compute database nodes, where write conflict detection is implemented using write-ahead logging and log sequence numbers. The write-write conflict check for the write-ahead log records received from the database engines of multiple database engines can be performed by comparing the log sequence numbers received from the database engines together with the write-ahead log records with the global log sequence numbers in the hash table in the common log. After the write-ahead log record passes the write-write conflict check, the write-ahead log record can be sent to the shared data store.

The invention relates to the technical field of data processing, and particularly provides a data synchronization method, system and device based on an embedded database and a medium, and the method comprises the steps: directly obtaining a corresponding log record before writing as a synchronization data source when a transaction change occurs in the database; determining a synchronization priority according to the data type of the log record, and caching and packaging the time series data with the common priority in batches; for the to-be-synchronized data, determining a consistent snapshot based on the log serial number, and constructing a synchronous data unit; sending the information to a cloud; and receiving and executing a data merging instruction issued by the cloud to update local data. According to the invention, high-reliability data synchronization with lower delay, higher network and storage efficiency and zero interference to services is realized on the Internet of Things terminal with limited resources.

The application relates to the technical field of databases, in particular to a data processing method and a data processing device, the data processing method comprising the following steps: when it is determined that a data processing request is a data reading request, first polling a plurality of slave servers and determining one of the slave servers as a target slave server, obtaining a first log sequence number stored in the target slave server, then obtaining a second log sequence number stored in a master server, and determining a synchronization delay difference between the master server and the target slave server according to the first log sequence number and the second log sequence number; when the synchronization delay difference is less than or equal to a preset threshold value and the polling frequency is less than or equal to a preset maximum polling frequency, stopping the polling and distributing the data reading request to the target slave server, and returning a query result obtained by the target slave server according to the data reading request; the query result obtained in this way guarantees that consistency and real-time performance both meet preset requirements, and the accuracy of the query result is guaranteed.

The invention discloses a transaction persistence method and device. The transaction persistence method comprises the steps of allocating a transaction number to a transaction based on the fact that a structured query language (SQL) list corresponding to the transaction is converted into a key-value pair list comprising at least one key-value pair; assigning a log sequence number (LSN) to each key value pair of the list of key value pairs; storing transaction metadata of the transaction in a metadata list stored in a dynamic random access memory (DRAM) that computes a high speed link (CXL) memory module (CMM-H) memory; and removing a pre-write log (WAL) from the LSM-tree relational database. The transaction metadata includes the transaction number, a first LSN of a first key-value pair of the list of key-value pairs, and one or more LSNs of the list of key-value pairs.

The application provides a database cluster physical backup method and related products. The database cluster physical backup method comprises the following steps: an initiating node reads a configuration file stored by itself, initiates a physical backup process, and acquires a starting log sequence number; a backup set label of this backup is constructed; the starting log sequence number and the backup set label are synchronized to other nodes, and the other nodes are guided to verify whether they participate in this backup; the initiating node acquires meta information of all files in a database directory, and distributes a required copy file list to each node participating in this backup according to the configuration file of the initiating node; each node is guided to copy files according to the distributed file list, so that each node records the meta information of the copied files in the copying process; and the initiating node receives the meta information of the copied files returned by each node. The application has the advantage that all available nodes can participate in backup, and resource consumption and occupation caused by single node backup are avoided.

A method for performing logging of modifications of a database includes, for each backend process of a plurality of backend processes simultaneously, writing a respective log entry to a write-ahead log buffer, submitting a respective commit request requesting the respective log entry be committed to a write-ahead log, and sleeping the respective backend process. The method also includes writing, using a dedicated writing process and direct asynchronous input / output, one or more of the respective log entries in the write-ahead log buffer to the write-ahead log. The dedicated writing process is different from each respective backend process of the plurality of backend processes. The method also includes updating a log sequence number pointer based on the respective log sequence numbers of the one or more of the respective log entries and waking, based on the log sequence number pointer, one or more of the respective backend processes.

Aspects of the disclosure are directed to a parallel recovery mode that applies log records while allowing read queries on read-replica databases. The parallel recovery mode can include applying log records in log sequence number (LSN) order for a block or for multiple blocks, and managing log records affecting multiple blocks. The parallel recovery mode can further manage dependency between different log records and maintain transactional consistency on read queries.

The present application discloses a pre-written log service method, apparatus, device and storage medium, relating to the technical field of log services and applied to a log server, including: receiving a log record request for a target log submitted concurrently by a state machine; generating a logical log sequence number and a physical log sequence number of the target log based on the log record request; in response to a restart replay request sent by the state machine, obtaining a mapping relationship between the logical log sequence number and the physical log sequence number; and applying the target log to the state machine through a logical log sequence sliding window based on the mapping relationship between the logical log sequence number and the physical log sequence number. The present application ensures the uniqueness and orderliness of the target log through the logical log sequence number and the physical log sequence number, and improves the fault tolerance and throughput capacity.

The invention provides a transaction log processing method of a database, a storage medium and equipment. The page mirror data of the transaction log is stored in a storage file configured in a disk. The transaction log processing method of the database comprises the steps of obtaining table position information of a table to which a data page corresponding to page mirror image data to be searched belongs in a disk, a block number of the data page corresponding to the page mirror image data to be searched in the table to which the data page belongs, and a log serial number of a transaction log to which the page mirror image data to be searched belongs, respectively recording as a target table position, a target block number and a target serial number; traversing storage files in the disk; obtaining a storage file of which the table position information in the file name is the same as the target table position in the disk, and recording the storage file as a preliminary target file; and obtaining page mirror image data with the same block number as the target block number and the same log serial number as the target serial number in the preliminary target file. The convenience of using the transaction log is improved from multiple aspects, and the operation efficiency of the database is improved.

The invention provides a database change data analysis method and related products, and the database change data analysis method comprises the steps: reading part of change data of a source database, and obtaining a plurality of analysis data corresponding to different source tables of the source database; the minimum serial number data in each piece of analysis data is searched for, whether at least one piece of analysis data and the unread change data comprise the same submission transaction or not is judged, and the minimum serial number data is the data with the minimum start log serial number of the submission transaction on the transaction log in the analysis data to which the minimum serial number data belongs; and if not, brushing the minimum serial number data according to the sequence of the starting log serial numbers. Under the condition that the data of the source database is changed, the consistency of the data after the data of the source database is changed and the actually changed data is ensured, and the data consistency of the target database and the source database is ensured.

The embodiment of the present specification provides a method, device and system for processing pre-written logs, wherein the method for processing pre-written logs comprises: providing a publishing interface and a subscribing interface based on a message queue communication protocol; receiving a pre-written log writing request through the publishing interface; obtaining a pre-written log record according to data carried by the pre-written log writing request; storing the pre-written log record into a message queue, so that a subscribing end consumes the log record through the subscribing interface; wherein a log serial number in the pre-written log record is determined based on a corresponding rule between a message offset and the log serial number, the message offset is a message offset of the pre-written log record in the message queue, and the log serial number is used to make the subscribing end determine a corresponding message offset of the pre-written log record to be played back based on the corresponding rule between the message offset and the log serial number.

The invention provides a log synchronization system, method and device of a database and a medium, and relates to the technical field of databases, in order to relieve the technical problem that logs are not synchronized after main and standby switching in the prior art, the method comprises the steps that a pre-written log directory of an old host is obtained; determining the latest log position of the old host based on the pre-write log directory of the old host; based on the latest log position, determining a log bifurcation position between the new host and the old host, determining a to-be-synchronized transaction set from the old host to the new host, and determining start log serial numbers of all transactions in the to-be-synchronized transaction set; performing logic decoding on the to-be-synchronized transaction set to obtain structured change data; on the basis of the structured change data, the new host is triggered to execute log synchronization operation, so that log synchronization of the new host and the original host is achieved after the host and the standby are switched, and consistency and reliability of data are ensured.

The application provides a log synchronization system, method, device and medium of a database, and relates to the technical field of databases, to alleviate the technical problem of log asynchronization after master-slave switching in the prior art, by obtaining a pre-write log directory of an old host; determining a latest log position of the old host based on the pre-write log directory of the old host; determining a log fork position between a new host and the old host based on the latest log position, determining a to-be-synchronized transaction set from the old host to the new host, and determining a starting log sequence number of all transactions in the to-be-synchronized transaction set; logically decoding the to-be-synchronized transaction set to obtain structured change data; and triggering the new host to perform a log synchronization operation based on the structured change data, to realize log synchronization between the new host and the original host after master-slave switching, and ensure data consistency and reliability.

This disclosure describes a technique for providing data consistency for hybrid transactional and analytical processing. A logical log and a Log Sequence Number (LSN) associated with the logical log can be generated based on data captured by a first processing engine. The logical log and LSN can be propagated to a storage subsystem configured to communicate with both the first and second processing engines. The LSN and information indicating the LSN mode version can be stored and distributed by a metadata service. The first processing engine, the second processing engine, the storage subsystem, and the metadata service are modularized and support an LSN mechanism for maintaining data consistency.