24 results about "Log sequence number" patented technology

The invention provides a log data reading method and device, equipment, a medium and a program product, and relates to the field of big data. Obtaining a mapping relationship between a plurality of timeline identifiers in a database and corresponding start log serial numbers; the timeline identifier is an identifier generated when the main database and the standby database are switched, whether the log data cross different timeline identifiers or not is judged according to the mapping relation and the starting log serial number and the ending log serial number of the log data, and under the condition that the log data cross different timeline identifiers, the log data cross different timeline identifiers. Reconstructing a log reading process according to a start log serial number of a new timeline identifier, wherein the new timeline identifier is different from the plurality of timeline identifiers; according to the method, automatic reconstruction of the log reading process after main and standby switching is achieved, the problems of low efficiency and poor real-time performance caused by manual operation are avoided, and continuity and consistency of data synchronization of the source end and the target end are guaranteed.

Support for distributed database backtracking, in which the database is reverted or iteratively reverted to a previous state, is provided. Log sequence number ranges and storage offset ranges for use in reconstructing the previous state are determined and used in the reconstruction. Clock sampling records can be generated indicating a feasible backtracking window with time points indicating allowable times to which backtracking can be performed. Appropriate log records, snapshots and checkpoints for database states are identified and organized with backtracking information record and timeline. A minimal and necessary set of log records, snapshots and checkpoints enabling database reconstruction within a time window is maintained. Measures are taken to maintain the necessary data to support backtracking even after prior backtrackings have occurred in the backtracking time window.

The application discloses a database pre-write log adaptive disk brushing scheduling method and device and medium, belongs to the technical field of database system data storage, and is used for solving the technical problems that the pre-write log disk brushing strategy of the existing database cannot adapt to different transaction load scenes, the disk synchronization overhead is high, the throughput capacity is limited, and the resource utilization rate is low. The method comprises the following steps: acquiring the transaction submission rate of a target database system within a preset time window; determining the pre-write log generation rate according to the log sequence number change amount of the target database system; predicting the system load state of the target database system based on the transaction submission rate and the pre-write log generation rate; dynamically matching the corresponding pre-write log disk brushing strategy according to the system load state; and controlling the target database system to perform the disk brushing operation according to the pre-write log disk brushing strategy.

A system and method for reconciling data from a local database streamed to a Cloud storage service is disclosed. A data transaction is received from a structured query language (SQL) database service operated by a server. The transaction includes log sequence numbers and corresponding data records. The log sequence numbers are stored in a change data capture (CDC) table. The sent data transaction is streamed via a data integration service. A storage object service stores the sent data into an object storage service bucket. A data reconciliation service deploys a cloud trail target to generate a log file of log sequence numbers from the sent data stored in the object storage service bucket. The log file of log sequence numbers is compared with the log sequence numbers in the CDC table. A missing data record is flagged to an exception table based on the comparison. An alert is issued for the missing data record.

Ensuring exactly once data ingestion, the method comprising: receiving, from an event stream, event data comprising a plurality of first records; generating a plurality of second records each corresponding to a first record of the plurality of first records and comprising a log sequence number (LSN) describing a location in the event stream for a corresponding first record; and sending, to a data warehouse, the plurality of second records.

Embodiments of the present application provide a database migration method, device, equipment, medium and program product. The method comprises: at the starting moment of migrating the inventory data of a source database to a target database, for any one of a plurality of tables to be migrated, based on a change data capture table, obtaining a first target log sequence number of a migration transaction corresponding to the table to be migrated; recording the first target log sequence number as low-water information; at the end moment of migrating the inventory data of the source database to the target database, based on the change data capture table, obtaining a second target log sequence number of a migration transaction corresponding to the table to be migrated; recording the second target log sequence number as high-water information; and based on the high-water information and the low-water information, synchronizing the incremental data of the source database to the target database. The method is used to improve the efficiency of database migration.

Support for distributed database backtracking, in which the database is reverted or iteratively reverted to a previous state, is provided. Log sequence number ranges and storage offset ranges for use in reconstructing the previous state are determined and used in the reconstruction. Clock sampling records can be generated indicating a feasible backtracking window with time points indicating allowable times to which backtracking can be performed. Appropriate log records, snapshots and checkpoints for database states are identified and organized with backtracking information record and timeline. A minimal and necessary set of log records, snapshots and checkpoints enabling database reconstruction within a time window is maintained. Measures are taken to maintain the necessary data to support backtracking even after prior backtrackings have occurred in the backtracking time window.

In a fault-tolerant method for event information delivery within a file system cluster, one or more processors determine event information associated with file system activity performed by nodes of the cluster. The one or more processors add the event information to an event log buffer in memory. The one or more processors receive a first log sequence number (LSN) associated with flushing recovery information from a recovery log buffer. The one or more processors determine event information in the event log buffer that has a log sequence number less than or equal to the first log sequence number, and determine that the event information includes a log sequence number less than or equal to the first log sequence number, and the one or more processors flush the corresponding event information from the event log buffer to disk storage.

A durability marker may be advanced according to ordered batches of log records. Different batches of log records may be sent to a distributed data store to update a log stored in the distributed data store. The different batches may be ordered according to a minimum Log Sequence Number (LSN) value within the different batches. The ordered different batches may be used to determine whether and how much to advance a durability marker for the log.

The invention relates to the technical field of database synchronization, and provides a data synchronization method and device based on database restoration, and the method comprises the steps that a source end synchronization tool obtains a backup file generated by a source end database; sending the backup file to a destination synchronization tool to enable the destination synchronization tool to obtain a reference database; obtaining a minimum log serial number corresponding to an active transaction in a source end database at the backup starting moment and a maximum log serial number corresponding to the backup finishing moment; reading an operation log of which the log serial number is greater than the minimum log serial number in a log file of the source end database, and modifying a submission operation of which the log serial number is less than or equal to the maximum log serial number into a rollback operation to obtain a target operation log, the target end synchronization tool carries out data synchronization on the target end database according to the reference database and the target operation log. According to the method, the consistency of the data is effectively maintained on the premise that the active transaction is automatically rolled back after the backup of the destination end database is restored.

The present disclosure relates to a timestamp-based log collection method, device, equipment and medium, the method comprising: reading a timestamp collection starting point added in a configuration file of a collection task; finding a target transaction log file based on the timestamp collection starting point; reading and analyzing the target transaction log file to determine a log sequence number (LSN) set in the configuration file; starting a collection replication stream based on the LSN, and collecting log data based on the replication stream. The present disclosure can determine the corresponding LSN by using the timestamp collection starting point, greatly reducing the difficulty of obtaining the LSN, reducing the professional requirements for users, and thus facilitating the execution of real-time collection tasks.

Examples described herein provide systems and methods for replicating unlogged data changes in a database management system are disclosed. The method involves identifying a load operation in a transaction log, determining a log sequence number (LSN) associated with the load operation, and identifying data pages in a source database with headers containing the load LSN. Data from these pages is extracted and saved to an external file. The target database is updated by loading the extracted data from the external file, allowing for incremental replication. This approach optimizes replication efficiency by avoiding a full refresh of the target table, reducing resource consumption, and enhancing system performance.

This application discloses a unified database waiting method, device, and medium based on WAL refresh events, belonging to the field of database technology. It addresses the problems of resource waste and unstable event response delays caused by existing polling mechanisms waiting for WAL refresh events. The method includes: establishing a WAL refresh event center in the database kernel; child processes registering global wait items with the WAL refresh event center; the WAL refresh event center arranging the registered wait items to generate a global wait queue; when WAL disk flushing is completed and a new log refresh position is generated, the WAL refresh event center wakes up the child processes corresponding to wait items in the global wait queue whose target log sequence number is not greater than the latest log refresh position; the awakened child processes obtain the current log refresh position, re-check the waiting conditions based on the current log refresh position, and execute business logic or re-enter the waiting state based on the check results.

The invention relates to the technical field of data processing, and particularly provides a data synchronization method, system and device based on an embedded database and a medium, and the method comprises the steps: directly obtaining a corresponding log record before writing as a synchronization data source when a transaction change occurs in the database; determining a synchronization priority according to the data type of the log record, and caching and packaging the time series data with the common priority in batches; for the to-be-synchronized data, determining a consistent snapshot based on the log serial number, and constructing a synchronous data unit; sending the information to a cloud; and receiving and executing a data merging instruction issued by the cloud to update local data. According to the invention, high-reliability data synchronization with lower delay, higher network and storage efficiency and zero interference to services is realized on the Internet of Things terminal with limited resources.

The application relates to the technical field of databases, in particular to a data processing method and a data processing device, the data processing method comprising the following steps: when it is determined that a data processing request is a data reading request, first polling a plurality of slave servers and determining one of the slave servers as a target slave server, obtaining a first log sequence number stored in the target slave server, then obtaining a second log sequence number stored in a master server, and determining a synchronization delay difference between the master server and the target slave server according to the first log sequence number and the second log sequence number; when the synchronization delay difference is less than or equal to a preset threshold value and the polling frequency is less than or equal to a preset maximum polling frequency, stopping the polling and distributing the data reading request to the target slave server, and returning a query result obtained by the target slave server according to the data reading request; the query result obtained in this way guarantees that consistency and real-time performance both meet preset requirements, and the accuracy of the query result is guaranteed.

The invention discloses a transaction persistence method and device. The transaction persistence method comprises the steps of allocating a transaction number to a transaction based on the fact that a structured query language (SQL) list corresponding to the transaction is converted into a key-value pair list comprising at least one key-value pair; assigning a log sequence number (LSN) to each key value pair of the list of key value pairs; storing transaction metadata of the transaction in a metadata list stored in a dynamic random access memory (DRAM) that computes a high speed link (CXL) memory module (CMM-H) memory; and removing a pre-write log (WAL) from the LSM-tree relational database. The transaction metadata includes the transaction number, a first LSN of a first key-value pair of the list of key-value pairs, and one or more LSNs of the list of key-value pairs.

The application provides a database cluster physical backup method and related products. The database cluster physical backup method comprises the following steps: an initiating node reads a configuration file stored by itself, initiates a physical backup process, and acquires a starting log sequence number; a backup set label of this backup is constructed; the starting log sequence number and the backup set label are synchronized to other nodes, and the other nodes are guided to verify whether they participate in this backup; the initiating node acquires meta information of all files in a database directory, and distributes a required copy file list to each node participating in this backup according to the configuration file of the initiating node; each node is guided to copy files according to the distributed file list, so that each node records the meta information of the copied files in the copying process; and the initiating node receives the meta information of the copied files returned by each node. The application has the advantage that all available nodes can participate in backup, and resource consumption and occupation caused by single node backup are avoided.

A method for performing logging of modifications of a database includes, for each backend process of a plurality of backend processes simultaneously, writing a respective log entry to a write-ahead log buffer, submitting a respective commit request requesting the respective log entry be committed to a write-ahead log, and sleeping the respective backend process. The method also includes writing, using a dedicated writing process and direct asynchronous input / output, one or more of the respective log entries in the write-ahead log buffer to the write-ahead log. The dedicated writing process is different from each respective backend process of the plurality of backend processes. The method also includes updating a log sequence number pointer based on the respective log sequence numbers of the one or more of the respective log entries and waking, based on the log sequence number pointer, one or more of the respective backend processes.

The embodiment of the present specification provides a method, device and system for processing pre-written logs, wherein the method for processing pre-written logs comprises: providing a publishing interface and a subscribing interface based on a message queue communication protocol; receiving a pre-written log writing request through the publishing interface; obtaining a pre-written log record according to data carried by the pre-written log writing request; storing the pre-written log record into a message queue, so that a subscribing end consumes the log record through the subscribing interface; wherein a log serial number in the pre-written log record is determined based on a corresponding rule between a message offset and the log serial number, the message offset is a message offset of the pre-written log record in the message queue, and the log serial number is used to make the subscribing end determine a corresponding message offset of the pre-written log record to be played back based on the corresponding rule between the message offset and the log serial number.

The application provides a log synchronization system, method, device and medium of a database, and relates to the technical field of databases, to alleviate the technical problem of log asynchronization after master-slave switching in the prior art, by obtaining a pre-write log directory of an old host; determining a latest log position of the old host based on the pre-write log directory of the old host; determining a log fork position between a new host and the old host based on the latest log position, determining a to-be-synchronized transaction set from the old host to the new host, and determining a starting log sequence number of all transactions in the to-be-synchronized transaction set; logically decoding the to-be-synchronized transaction set to obtain structured change data; and triggering the new host to perform a log synchronization operation based on the structured change data, to realize log synchronization between the new host and the original host after master-slave switching, and ensure data consistency and reliability.

This disclosure describes a technique for providing data consistency for hybrid transactional and analytical processing. A logical log and a Log Sequence Number (LSN) associated with the logical log can be generated based on data captured by a first processing engine. The logical log and LSN can be propagated to a storage subsystem configured to communicate with both the first and second processing engines. The LSN and information indicating the LSN mode version can be stored and distributed by a metadata service. The first processing engine, the second processing engine, the storage subsystem, and the metadata service are modularized and support an LSN mechanism for maintaining data consistency.

The present application relates to the technical field of database management, and particularly relates to a database recoverable time interval acquisition method, medium and equipment, which determines start and end sequence numbers through a current log sequence number and a timeline identifier, avoids calculation deviation of a recoverable interval caused by scanning range error, performs continuity analysis through a file naming sequence and a sequence number, automatically identifies log integrity before recovery, avoids recovery failure caused by log loss, analyzes a timeline history file, checks legality based on a branch point and an associated sequence number, automatically identifies a legal log segment in a timeline branch scene, avoids recovery failure caused by using an incorrect timeline log, calculates a recoverable interval through fusion of continuity state, time mapping, legality result and an archive delay threshold value, can generate an explicit effective time range before recovery, avoids recovery failure caused by blind operation, and dynamically adjusts a termination time to ensure matching with a master node log synchronization state.

Support for distributed database backtracking, in which the database is reverted or iteratively reverted to a previous state, is provided. Log sequence number ranges and storage offset ranges for use in reconstructing the previous state are determined and used in the reconstruction. Clock sampling records can be generated indicating a feasible backtracking window with timepoints indicating allowable times to which backtracking can be performed. Appropriate log records, snapshots and checkpoints for database states are identified and organized with backtracking information record and timeline. A minimal and necessary set of log records, snapshots and checkpoints enabling database reconstruction within a time window is maintained. Measures are taken to maintain the necessary data to support backtracking even after prior backtrackings have occurred in the backtracking time window.

The invention relates to the technical field of distributed file systems, in particular to a write-once read-many distributed file system implementation method, device and medium, and the method comprises the steps: constructing a write-once read-many architecture which comprises a write node with a unique write permission and a plurality of read nodes, and enabling all the nodes to share and access the same shared storage layer, each node is locally configured with a B-tree data structure, the write operation content of the write node is stored in a shared storage layer, the updated operation log is synchronized to each read node, the received operation log is applied to a local B-tree data structure copy of the read node according to a log sequence number sequence, and when the read node receives a read operation request, the read operation request is sent to the read node, and the read operation request is sent to the read node. Adopting a corresponding read operation strategy according to the data consistency level of the request; according to the method, the storage cost and the read-write delay are remarkably reduced while the strong consistency of the data can be ensured.