28 results about "Change data capture" patented technology

The invention provides a multi-database automatic synchronization method under an AI platform, and belongs to the technical field of database automatic synchronization. The captured change records are converted into synchronous messages in a unified format, the synchronous messages are input into an intelligent scheduling optimization model for dependency analysis and transaction reordering based on dynamic programming, data conversion rules are dynamically generated according to mode version differences, and data consistency is verified through a Merkle tree hierarchical structure; the multi-stage self-adaptive compression transmission and data writing are completed after the target node executes conflict detection and circular dependency processing, and finally the final consistency among the multiple databases is guaranteed through a periodic full-amount account checking task, so that the technical problem of synchronization performance bottleneck caused by low transaction scheduling efficiency in the automatic synchronization process of the multiple databases is solved.

The invention provides a data synchronization desensitization method and device, computer equipment and a readable storage medium. The method comprises the following steps: pre-configuring synchronization parameters; receiving a change event of the source database output by the change data capture engine; performing desensitization processing on the event load of the change event according to a preset desensitization function and a desensitization parameter; judging the event type of the change event; according to the event type and the dialect identifier, mapping an abstract type of a corresponding field in the event load into a physical type which can be understood by the target database to obtain type information of the field; generating a parameterized equivalent SQL template compatible with the target database dialect according to the dialect identifier, the event type and the type information of the field; the event load is bound to the parameterized equivalent SQL template, and a bound parameterized SQL statement is obtained; and executing the parameterized SQL statement in the target database. According to the invention, a set of general data synchronization and desensitization mechanism is constructed.

This invention discloses a low-latency materialized view method, comprising the following steps: S1, after the application server initiates a database transaction to the source database and executes data operation instructions, before the database transaction is committed, a state intent event is constructed. This relates to the field of database technology. This application uses heavy data change information as the state intent event and a lightweight commit confirmation signal as the commit state event. By separating the state intent event and the commit state event and processing them asynchronously in parallel, the update latency of the materialized view no longer depends on the entire CDC link, but only on the transmission latency of the commit confirmation signal. This reduces transmission latency, achieves lower-latency data synchronization, and alleviates the high latency problem inherent in traditional materialized view updates that rely on change data capture technology.

The invention provides a device for automatic metadata management and data knitting treatment, and the device comprises a semantic mounting unit which is used for carrying out the semantic tag mounting of a structured metadata set of a data factory based on a semantic model of a data knitting architecture, so as to generate enhanced semantic metadata and a complete data line record; the rule binding unit is used for carrying out governance rule binding on the enhanced semantic metadata and the complete data line record on the basis of a data knitting federation governance rule so as to generate a metadata governance list bound with the governance rule; the change updating unit is used for synchronously updating a data set state field and a flow path recorded by a corresponding complete data line to a metadata governance list based on a change data capture mechanism when it is monitored that a data factory has a data set change, so as to generate unified metadata caalog; the interface configuration unit is used for configuring a metadata retrieval interface and a governance rule execution engine for the unified metadata caalog so that the unified metadata caalog can be accessed in the data weaving environment.

A method, system, and computer program product configured to perform operations including: detecting, using a first thread associated with a first change data capture (CDC) replication mode, that a database transaction associated with a source database exceeds a threshold time value; based on the detecting, spawning a second thread that is configured to identify log entries associated with the database transaction; confirming the database transaction qualifies as a long transaction based on a commit operation in the log entries associated with the database transaction; based on the confirming, switching to a second replication mode for a table associated with the database transaction; updating the table in a target database using values obtained by executing queries on the source database; and based on the updating the table in the target database, terminating the second thread and switching back to the first CDC replication mode for the table.

The application discloses a method and device for sensing, judging and processing data structure change. The method comprises the following steps: collecting a data structure change event based on a change data capture tool; constructing a data structure change abstract syntax tree through a SQL parser to accurately identify the change operation type and content; obtaining structure snapshots before and after the change; executing compatibility judgment in combination with a compatibility rule library to output the compatibility state and risk level; adopting a hierarchical message notification strategy according to the risk level; and automatically triggering or arranging corresponding processing actions, such as automatically synchronizing fields, generating a compatible view or suspending a high-risk task. The application realizes semantic-level analysis of data structure change, automatic compatibility judgment, refined risk grading and automatic processing, significantly reduces the cost of manual intervention, effectively avoids downstream synchronization interruption and data quality abnormalities caused by structure change, and improves the stability and automatic operation and maintenance level of a medium or large data platform.

Embodiments of the present application provide a database migration method, device, equipment, medium and program product. The method comprises: at the starting moment of migrating the inventory data of a source database to a target database, for any one of a plurality of tables to be migrated, based on a change data capture table, obtaining a first target log sequence number of a migration transaction corresponding to the table to be migrated; recording the first target log sequence number as low-water information; at the end moment of migrating the inventory data of the source database to the target database, based on the change data capture table, obtaining a second target log sequence number of a migration transaction corresponding to the table to be migrated; recording the second target log sequence number as high-water information; and based on the high-water information and the low-water information, synchronizing the incremental data of the source database to the target database. The method is used to improve the efficiency of database migration.

The application discloses a real-time fusion and incremental synchronization method and system of heterogeneous data sources, and belongs to the technical field of data synchronization. The method comprises the following steps: a change data capture step is used to parse a plurality of database logs through a source connector to obtain change events; an event standardization step is used to convert the change events into a unified format; a stream processing step is used for deduplication, sorting and transaction boundary identification; an adaptive scheduling step is used to dynamically adjust the consumption parallelism according to the load; and a consistency checking step is used to compare the data digest of the source end and the target end and repair inconsistencies. The application solves the problems of high synchronization delay, complex integration and difficult consistency guarantee of the heterogeneous data sources, controls the end-to-end delay in the order of seconds, and supports the processing of millions of events per second.

A method and system for updating backup data for a database is presented. The method includes initiating a merging operation for a defined time interval; obtaining change data capture (CDC) data generated during the defined time interval; aggregating and ordering the CDC data by a key value; identifying backup data associated with the ordered CDC data; generating a candidate data object that represents data modified during the defined time interval; deduplicating the candidate data object by comparing a hash identifier of the candidate data object with a plurality of hash identifiers stored in a repository; storing the candidate data object when the comparison determines that no matching hash identifier exists in the repository; and updating backup metadata to associate the key value with the candidate data object.

The application relates to the field of e-government informatization and archive information management, and discloses an archive coupling processing method and system based on business flow and event driving, which identifies a business settlement event through complex event processing; linkage change data capture extracts an anchor change set in a transaction consistent range and obtains a candidate fragment through difference; conclusion data is screened by adopting a lightweight Transformer, and an archiving package containing field tracing is generated by extracting and checking according to a predefined data mode; a Merkel tree is respectively constructed for content and metadata, and a ledger abstract is generated to be written into an archiving ledger, entity information is associated under a dual-track system, settlement and archiving are realized, and traceability and verification are realized.

The invention discloses a project full life cycle cost control method and system, and relates to the technical field of enterprise project management and cost control. The method comprises the following steps: accessing heterogeneous cost data from a plurality of service systems through a multi-source data acquisition framework, and realizing quasi-real-time data synchronization by adopting a change data capture technology; performing real-time processing on the accessed data by using a stream processing engine and a memory computing framework, and constructing a distributed cost knowledge graph based on a graph database, the knowledge graph being used for modeling a causal relationship among cost elements; based on the knowledge graph and the real-time data, at least one intelligent analysis operation is executed, and the intelligent analysis operation comprises self-adaptive process optimization based on reinforcement learning, self-adaptive cost anomaly detection based on multi-model fusion and high-performance cost analysis based on a multi-dimensional data cube engine. The technical problems that all links of a traditional cost management system are separated, and data flow is not smooth are solved.

The application relates to the fields of CAE simulation technology and distributed architecture, and discloses a service-oriented CAE simulation platform micro-service architecture construction method and system, which comprises the following steps: constructing a micro-service architecture, wherein the micro-service architecture comprises a core micro-service cluster, a unified service bus, a dynamic resource scheduling center and a distributed data layer; data generated by each micro-service is stored through the distributed data layer; the distributed data layer cuts unstructured data into data shards and stores the data shards in the form of multiple copies into an object storage cluster, stores structured data into a relational database, stores business metadata into a cache cluster, listens to change logs of the relational database by using a change data capture technology, and synchronizes change events to the cache cluster. The traditional monomer platform needs 3 hours to complete the same task, the application only needs 1 hour and 30 minutes, the efficiency is improved by 50%, and the task queuing time is shortened from 1 hour to 10 minutes.

The application provides a database log analysis method, device and equipment, belongs to the technical field of computer information processing, and solves the problems of resource waste, structure change blockage and low scanning efficiency caused by the traditional change data capture mechanism when monitoring multiple tables. The method comprises the following steps: obtaining a metadata table and log data of a target database; performing query grouping processing on the metadata table to obtain a monitoring table; performing analysis and row record extraction processing on the monitoring table to obtain row record cache data; performing grabbing and updating processing on the log data to obtain log records; performing grouping filtering processing on the log records to obtain business change logs; performing cutting and conversion processing on the business change logs according to the monitoring table and the row record cache data to obtain structured data; and determining log change data according to the structured data. The scheme realizes millisecond-level change capture of multiple tables through single scanning, and significantly reduces resource consumption and data delay.

A computer-implemented method for maintaining unified guest identity records across heterogeneous hospitality data systems. The method receives guest interaction data from independent data sources operating on different schemas, normalizes identifiers through standardization algorithms, and applies two-stage matching comprising deterministic cryptographic hashing followed by probabilistic Bayesian inference. Core attributes are stored in a relational database with strong consistency guarantees while behavioral data is maintained in a document database with eventual consistency, synchronized through a change data capture mechanism providing near real-time synchronization. The identity resolution engine operates continuously in real-time, reducing storage requirements compared to systems allowing duplicate records. The system further implements a predictive segmentation model using online learning to adjust parameters in real-time while preserving accuracy for previously learned patterns. An event-driven workflow orchestration architecture records state-changing guest actions as immutable events and dispatches them to independent CRM, marketing automation, event management, and analytics modules that autonomously respond without centralized orchestration, enabling high-throughput processing with low latency.

The invention provides a data synchronization configuration optimization method and device based on metadata automatic discovery and electronic equipment, and relates to the technical field of data optimizing.The method comprises the steps that metadata obtaining logic of heterogeneous data sources is abstracted through a uniform interface, and table structures, field types and primary key constraints of the data sources are automatically connected and analyzed; based on a declarative configuration engine, automatically generating a field mapping rule and a target end DDL statement according to a source library name, a table name and a target end address input by a user; a source end DDL event is monitored in real time by integrating a change data capture technology, and a target end table structure is updated by adopting a dual-stage submission protocol; and constructing a cross-data source type mapping knowledge base, and dynamically adapting a type conversion rule in combination with semantic reasoning. According to the method, the metadata acquisition logic of the heterogeneous data source is abstracted through the uniform interface, the zero configuration access of a traditional database, a big data component and an API interface is supported, the compatibility is improved, and the tedious process of manually compiling field mapping rules and type conversion scripts is eliminated.

Systems and methods herein combine the benefits of message passing through a message broker with the reliability of traditional databases for long-term data, storage. By utilizing a Change Data Capture (CDC) mechanism, various embodiments facilitate publishing of messages to the message broker while capturing the nature of the database changes. By establishing the database as the primary point of interaction, the two-transaction problem that arises from coordinating between disjoint systems, such as a database and a message broker is successfully avoided. Messages are generated directly from database interactions, captured by the CDC mechanism, and asynchronously transmitted. Advantageously, system components can receive messages and execute corresponding actions, without having to query the database. Asynchronous message processing thus reduces system latency and enhances efficiency by minimizing database load and the resources needed for tire database to keep connections open to various components.

The invention relates to the technical field of data synchronization, in particular to a MySQL and Elasticsearch data real-time synchronization method based on a message queue. The invention provides a data real-time synchronization scheme based on a message queue. Monitoring the MySQL binary log through the change data capture component, and obtaining database addition, deletion and change operation events and change information in real time; the change information is converted into a standard format and sent to a message queue; obtaining and analyzing the message content from the queue, performing data type mapping and structure conversion, and generating target data conforming to an Elasticsearch format; finally, index updating operation is executed, and consistency verification and conflict detection are carried out at the same time. According to the method, traditional timed task query is replaced by binary log monitoring, and the message queue is introduced as a middle layer, so that data real-time synchronization is realized, the influence on database performance is reduced, and the data accuracy is ensured.

The invention relates to the technical field of database security, and discloses a database field integrity protection method and system based on dynamic Hash check.The method comprises the steps that a database log file is monitored and analyzed based on a change data capturing method, data changes are captured in real time, and multi-level Hash chains are generated through layer-by-layer aggregation; and based on the generated multi-level hash chain, positioning key data tampering in combination with multi-database adaptation and a dynamic verification strategy mechanism, and triggering an alarm. According to the method, database log files are monitored and analyzed by adopting a change data capturing technology, the change of each piece of data is captured in real time, the dynamic hash chain is generated by layer-by-layer aggregation, and accurate positioning of key data tampering is realized by combining a multi-database adaptation layer and a dynamic verification strategy mechanism. When the data is written in, the hash value is stored in the independent check table, if the field in the strategy is tampered, an alarm is triggered, if the field is not configured, check is skipped, and the safety and the performance efficiency are both considered.

This invention provides a method for automatic synchronization of multiple databases under an AI platform, belonging to the field of automatic database synchronization technology. This invention initializes the multi-database synchronization topology and deploys a change data capture module to monitor data changes in real time. The captured change records are converted into unified format synchronization messages and input into an intelligent scheduling optimization model for dependency analysis and transaction reordering based on dynamic programming. Data conversion rules are dynamically generated based on mode version differences, and Merkle tree hierarchical structure is used to verify data consistency. After conflict detection and circular dependency handling are performed at the target node, multi-level adaptive compression transmission and data writing are completed. Finally, periodic full reconciliation tasks ensure eventual consistency among the multiple databases, solving the technical problem of low transaction scheduling efficiency leading to synchronization performance bottlenecks during automatic multi-database synchronization.

The invention provides a front-end framework rendering optimization method based on off-screen rendering and multiple threads. According to the method, non-interactive rendering elements in a single file component are automatically identified in a compiling stage and marks are added, so that a traditional process needing manual optimization is automated, and the development complexity is reduced; thread isolation of rendering logic is realized by replacing the marking element with the agent component and generating the multi-thread rendering script; in combination with a communication bridging mechanism of a double-buffer structure, an off-screen canvas control right is transferred to an independent rendering thread during operation, so that a main thread is only responsible for state management and change data capture, and the rendering thread focuses on an incremental rendering task. Therefore, decoupling of the main thread and the rendering thread avoids the problem of synchronous blocking, and a double-buffer data transmission mechanism reduces serialization overhead, so that the rendering efficiency and the page response speed in a high-frequency data updating scene are remarkably improved, and meanwhile seamless integration with a front-end frame is kept.

The invention discloses a real-time abnormal data detection method and system based on dynamic perception and learning, and belongs to the technical field of data warehouse and big data processing, and the system comprises a change data capturing module which is used for capturing change data of a data source in real time; the message queue module is used for receiving the real-time change data stream captured by the change data capturing module, and decoupling and buffering the real-time change data stream to obtain a buffered data stream; the rule sensing engine is used for analyzing the buffer data flow through an existing rule set and a sensing detection model and outputting a latest rule set; the dynamic learning engine is used for training and optimizing a machine learning model based on continuously inflowing buffer data flow, historical data and feedback of a detection result; according to the method and the system, data abnormity real-time detection, automatic sensing of data mode change and dynamic learning and updating of the detection model can be realized, so that the detection delay is remarkably reduced, false alarms and missing alarms are reduced, and the manual maintenance cost is reduced.

System and methods are provided for dynamically consolidating interaction points in a distribution ecosystem. The method involves integrating multiple touchpoints of communication between distributors, resellers, end-users, vendors, and suppliers into a unified interactive interface. This interface enables the management of end-to-end partner lifecycle, systematic data collection, analysis using advanced statistical algorithms, deployment of artificial intelligence and machine learning algorithms, and continuous updates based on user feedback. The system includes modules for communication integration, consolidation, lifecycle management, data collection, data analysis, and artificial intelligence. The disclosed method and system enhance supply chain operations, generate actionable insights, and provide personalized user experiences, ultimately driving business growth and efficiency.

A cloud-based data platform is disclosed, enabling storage-agnostic data management and analytics in a unified environment. The system may integrate a data lake implemented as a hyperscaler object store, a cloud-based database management system (DBMS), elastic compute resources, and analytics engines. Data may be stored in open table formats, such as Apache Parquet, Delta Lake, and Apache Iceberg, supporting ACID transactions, schema evolution, and efficient query processing. Virtual tables may map data stored in the data lake to the DBMS, enabling in-situ query processing via SQL interfaces. The platform may support advanced features, including change data capture (CDC), time travel, and lifecycle management using a SAGA pattern for atomic operations. Security may be ensured through X.509 certificates, web tokens, and role-based access controls. Elastic compute resources, such as Apache Spark, facilitate large-scale data transformations and analytics.

Systems and methods provide low-latency, ordered delivery of database change events to a watch client. In response to a database (DB) create, update, or delete (CUD) request, the system writes the request to a database. The database usually provides a change data capture stream of events to a publish / subscribe (pub / sub) system. This results in higher change event delivery latency and is referred to as a slow path. The system leverages a dual write approach, where the system asynchronously writes a corresponding change event directly to a fast path pub / sub system. A watch server consumes change events from the slow and fast path pub / sub systems and merges and orders the change events using a first-in first-out cache. An ordered event stream can be transmitted to the watch client with lower latency. The ordered event stream includes one or more batches, and each batch comprises a most recent change event version for each object that was changed.

The invention relates to the technical field of computers, and discloses a distributed database online redistribution method and device, a medium and computer device.The method comprises the steps that in response to a redistribution request, a target distribution strategy is determined, and a shadow table with the same structure as an original table is constructed; migrating the total data of the original table to the shadow table under the state that the original table keeps the online service; capturing data operation on the original table in real time based on a change data capturing mechanism, and synchronizing the data operation to the shadow table through a double-writing mechanism; and when it is detected that the data state of the shadow table is consistent with that of the original table, executing an atomization switching operation on the shadow table, and redirecting an access request for the original table to the shadow table. According to the method, on-line data redistribution without shutdown, lock blockage and service awareness is realized through core technical means such as construction of shadow tables with consistent structures, parallel execution of total data migration, increment double-write synchronization based on change data capture, atomization metadata switching and the like.

The subject technology dynamically decides between two approaches to produce change data capture (CDC) records depending on estimated cost. One approach corresponds to a time-travel based approach. In the other approach, the subject technology builds off the mutations recorded by the underlying linearizable store between the two timestamps to produce CDC.

A procedure (400) involves accessing a distributed database (140) containing a plurality of clusters (210), each cluster containing a plurality of partitions (212), and each cluster being configured to receive read and write operations (20, 21). The procedure involves receiving, on a second cluster (210B), a plurality of changes (350) for a second table (250B) and storing the plurality of changes in a replication log (351). The procedure also involves asynchronously replicating the plurality of changes from the second table to a first table (250A) and maintaining a respective change stream position (214A) that tracks a respective position in the replication log indicating one or more changes that have been replicated.The procedure involves receiving a read request at the first cluster (210A) requesting one or more rows of the first table, and returning the respective change stream position.

The application relates to a full-incremental synchronization method and system based on a change data real-time capturing technology, and belongs to the field of data synchronization. The method comprises the following steps: full-incremental synchronization accesses a power grid resource business middle station database through a Rest interface to obtain full-incremental graphic data; further, a synchronization conversion operation is performed on the full-incremental graphic data, and the converted GIS graphic data is batch-synchronized to a GIS database; incremental synchronization captures change data from the power grid resource business middle station through a preset data synchronization service, encapsulates the change data into a standardized data format, and pushes the change data to a message middleware; the message middleware acquires the change data in the standardized data format, inputs the change data into a synchronization conversion link, performs a synchronization conversion operation, and synchronizes the converted change data to the GIS database; and the application can realize data synchronization of the power grid resource business middle station database and the GIS database, and solves the problem of inconsistent graphics of the two sets of systems.