43 results about "Change data capture" patented technology

The invention provides a multi-database automatic synchronization method under an AI platform, and belongs to the technical field of database automatic synchronization. The captured change records are converted into synchronous messages in a unified format, the synchronous messages are input into an intelligent scheduling optimization model for dependency analysis and transaction reordering based on dynamic programming, data conversion rules are dynamically generated according to mode version differences, and data consistency is verified through a Merkle tree hierarchical structure; the multi-stage self-adaptive compression transmission and data writing are completed after the target node executes conflict detection and circular dependency processing, and finally the final consistency among the multiple databases is guaranteed through a periodic full-amount account checking task, so that the technical problem of synchronization performance bottleneck caused by low transaction scheduling efficiency in the automatic synchronization process of the multiple databases is solved.

Described herein are methods and systems for updating knowledge bases for Retrieval-Augmented Generation (RAG) applications. The methods employ Change Data Capture (CDC) to efficiently detect modifications in source data. These CDC techniques may enable targeted updates to semantic indexing tables by traversing data models from leaf tables to root entities, ensuring that only affected embeddings are regenerated rather than reprocessing entire document collections.

The invention provides a data synchronization desensitization method and device, computer equipment and a readable storage medium. The method comprises the following steps: pre-configuring synchronization parameters; receiving a change event of the source database output by the change data capture engine; performing desensitization processing on the event load of the change event according to a preset desensitization function and a desensitization parameter; judging the event type of the change event; according to the event type and the dialect identifier, mapping an abstract type of a corresponding field in the event load into a physical type which can be understood by the target database to obtain type information of the field; generating a parameterized equivalent SQL template compatible with the target database dialect according to the dialect identifier, the event type and the type information of the field; the event load is bound to the parameterized equivalent SQL template, and a bound parameterized SQL statement is obtained; and executing the parameterized SQL statement in the target database. According to the invention, a set of general data synchronization and desensitization mechanism is constructed.

This invention discloses a low-latency materialized view method, comprising the following steps: S1, after the application server initiates a database transaction to the source database and executes data operation instructions, before the database transaction is committed, a state intent event is constructed. This relates to the field of database technology. This application uses heavy data change information as the state intent event and a lightweight commit confirmation signal as the commit state event. By separating the state intent event and the commit state event and processing them asynchronously in parallel, the update latency of the materialized view no longer depends on the entire CDC link, but only on the transmission latency of the commit confirmation signal. This reduces transmission latency, achieves lower-latency data synchronization, and alleviates the high latency problem inherent in traditional materialized view updates that rely on change data capture technology.

The invention provides a device for automatic metadata management and data knitting treatment, and the device comprises a semantic mounting unit which is used for carrying out the semantic tag mounting of a structured metadata set of a data factory based on a semantic model of a data knitting architecture, so as to generate enhanced semantic metadata and a complete data line record; the rule binding unit is used for carrying out governance rule binding on the enhanced semantic metadata and the complete data line record on the basis of a data knitting federation governance rule so as to generate a metadata governance list bound with the governance rule; the change updating unit is used for synchronously updating a data set state field and a flow path recorded by a corresponding complete data line to a metadata governance list based on a change data capture mechanism when it is monitored that a data factory has a data set change, so as to generate unified metadata caalog; the interface configuration unit is used for configuring a metadata retrieval interface and a governance rule execution engine for the unified metadata caalog so that the unified metadata caalog can be accessed in the data weaving environment.

A method, system, and computer program product configured to perform operations including: detecting, using a first thread associated with a first change data capture (CDC) replication mode, that a database transaction associated with a source database exceeds a threshold time value; based on the detecting, spawning a second thread that is configured to identify log entries associated with the database transaction; confirming the database transaction qualifies as a long transaction based on a commit operation in the log entries associated with the database transaction; based on the confirming, switching to a second replication mode for a table associated with the database transaction; updating the table in a target database using values obtained by executing queries on the source database; and based on the updating the table in the target database, terminating the second thread and switching back to the first CDC replication mode for the table.

A system and method for reconciling data from a local database streamed to a Cloud storage service is disclosed. A data transaction is received from a structured query language (SQL) database service operated by a server. The transaction includes log sequence numbers and corresponding data records. The log sequence numbers are stored in a change data capture (CDC) table. The sent data transaction is streamed via a data integration service. A storage object service stores the sent data into an object storage service bucket. A data reconciliation service deploys a cloud trail target to generate a log file of log sequence numbers from the sent data stored in the object storage service bucket. The log file of log sequence numbers is compared with the log sequence numbers in the CDC table. A missing data record is flagged to an exception table based on the comparison. An alert is issued for the missing data record.

The application discloses a method and device for sensing, judging and processing data structure change. The method comprises the following steps: collecting a data structure change event based on a change data capture tool; constructing a data structure change abstract syntax tree through a SQL parser to accurately identify the change operation type and content; obtaining structure snapshots before and after the change; executing compatibility judgment in combination with a compatibility rule library to output the compatibility state and risk level; adopting a hierarchical message notification strategy according to the risk level; and automatically triggering or arranging corresponding processing actions, such as automatically synchronizing fields, generating a compatible view or suspending a high-risk task. The application realizes semantic-level analysis of data structure change, automatic compatibility judgment, refined risk grading and automatic processing, significantly reduces the cost of manual intervention, effectively avoids downstream synchronization interruption and data quality abnormalities caused by structure change, and improves the stability and automatic operation and maintenance level of a medium or large data platform.

Embodiments of the present application provide a database migration method, device, equipment, medium and program product. The method comprises: at the starting moment of migrating the inventory data of a source database to a target database, for any one of a plurality of tables to be migrated, based on a change data capture table, obtaining a first target log sequence number of a migration transaction corresponding to the table to be migrated; recording the first target log sequence number as low-water information; at the end moment of migrating the inventory data of the source database to the target database, based on the change data capture table, obtaining a second target log sequence number of a migration transaction corresponding to the table to be migrated; recording the second target log sequence number as high-water information; and based on the high-water information and the low-water information, synchronizing the incremental data of the source database to the target database. The method is used to improve the efficiency of database migration.

The application discloses a real-time fusion and incremental synchronization method and system of heterogeneous data sources, and belongs to the technical field of data synchronization. The method comprises the following steps: a change data capture step is used to parse a plurality of database logs through a source connector to obtain change events; an event standardization step is used to convert the change events into a unified format; a stream processing step is used for deduplication, sorting and transaction boundary identification; an adaptive scheduling step is used to dynamically adjust the consumption parallelism according to the load; and a consistency checking step is used to compare the data digest of the source end and the target end and repair inconsistencies. The application solves the problems of high synchronization delay, complex integration and difficult consistency guarantee of the heterogeneous data sources, controls the end-to-end delay in the order of seconds, and supports the processing of millions of events per second.

A method and system for updating backup data for a database is presented. The method includes initiating a merging operation for a defined time interval; obtaining change data capture (CDC) data generated during the defined time interval; aggregating and ordering the CDC data by a key value; identifying backup data associated with the ordered CDC data; generating a candidate data object that represents data modified during the defined time interval; deduplicating the candidate data object by comparing a hash identifier of the candidate data object with a plurality of hash identifiers stored in a repository; storing the candidate data object when the comparison determines that no matching hash identifier exists in the repository; and updating backup metadata to associate the key value with the candidate data object.

The application relates to the field of e-government informatization and archive information management, and discloses an archive coupling processing method and system based on business flow and event driving, which identifies a business settlement event through complex event processing; linkage change data capture extracts an anchor change set in a transaction consistent range and obtains a candidate fragment through difference; conclusion data is screened by adopting a lightweight Transformer, and an archiving package containing field tracing is generated by extracting and checking according to a predefined data mode; a Merkel tree is respectively constructed for content and metadata, and a ledger abstract is generated to be written into an archiving ledger, entity information is associated under a dual-track system, settlement and archiving are realized, and traceability and verification are realized.

The invention discloses a real-time data analysis and visualization system and method based on big data. The system comprises a data acquisition module, a data transmission module, a data preprocessing module, a real-time analysis module, a data storage module and a visualization module. The data acquisition module is connected with the data transmission module, the data transmission module is connected with the data preprocessing module, the data preprocessing module is connected with the real-time architecture module, and the real-time analysis module is connected with the data storage module and the visualization module. According to the method, minute-level delay of traditional Sqoop'timed pulling 'is avoided, millisecond-level access of database change data is achieved, flexible expansion of'data source-computing node-storage capacity' is achieved through hierarchical decoupling and a distributed elastic architecture, and analysis result errors are controlled within 1% through combination of whole-process data quality management and control and multi-algorithm fusion optimization.

The invention discloses a project full life cycle cost control method and system, and relates to the technical field of enterprise project management and cost control. The method comprises the following steps: accessing heterogeneous cost data from a plurality of service systems through a multi-source data acquisition framework, and realizing quasi-real-time data synchronization by adopting a change data capture technology; performing real-time processing on the accessed data by using a stream processing engine and a memory computing framework, and constructing a distributed cost knowledge graph based on a graph database, the knowledge graph being used for modeling a causal relationship among cost elements; based on the knowledge graph and the real-time data, at least one intelligent analysis operation is executed, and the intelligent analysis operation comprises self-adaptive process optimization based on reinforcement learning, self-adaptive cost anomaly detection based on multi-model fusion and high-performance cost analysis based on a multi-dimensional data cube engine. The technical problems that all links of a traditional cost management system are separated, and data flow is not smooth are solved.

The application relates to the fields of CAE simulation technology and distributed architecture, and discloses a service-oriented CAE simulation platform micro-service architecture construction method and system, which comprises the following steps: constructing a micro-service architecture, wherein the micro-service architecture comprises a core micro-service cluster, a unified service bus, a dynamic resource scheduling center and a distributed data layer; data generated by each micro-service is stored through the distributed data layer; the distributed data layer cuts unstructured data into data shards and stores the data shards in the form of multiple copies into an object storage cluster, stores structured data into a relational database, stores business metadata into a cache cluster, listens to change logs of the relational database by using a change data capture technology, and synchronizes change events to the cache cluster. The traditional monomer platform needs 3 hours to complete the same task, the application only needs 1 hour and 30 minutes, the efficiency is improved by 50%, and the task queuing time is shortened from 1 hour to 10 minutes.

The application provides a database log analysis method, device and equipment, belongs to the technical field of computer information processing, and solves the problems of resource waste, structure change blockage and low scanning efficiency caused by the traditional change data capture mechanism when monitoring multiple tables. The method comprises the following steps: obtaining a metadata table and log data of a target database; performing query grouping processing on the metadata table to obtain a monitoring table; performing analysis and row record extraction processing on the monitoring table to obtain row record cache data; performing grabbing and updating processing on the log data to obtain log records; performing grouping filtering processing on the log records to obtain business change logs; performing cutting and conversion processing on the business change logs according to the monitoring table and the row record cache data to obtain structured data; and determining log change data according to the structured data. The scheme realizes millisecond-level change capture of multiple tables through single scanning, and significantly reduces resource consumption and data delay.

A computer-implemented method for maintaining unified guest identity records across heterogeneous hospitality data systems. The method receives guest interaction data from independent data sources operating on different schemas, normalizes identifiers through standardization algorithms, and applies two-stage matching comprising deterministic cryptographic hashing followed by probabilistic Bayesian inference. Core attributes are stored in a relational database with strong consistency guarantees while behavioral data is maintained in a document database with eventual consistency, synchronized through a change data capture mechanism providing near real-time synchronization. The identity resolution engine operates continuously in real-time, reducing storage requirements compared to systems allowing duplicate records. The system further implements a predictive segmentation model using online learning to adjust parameters in real-time while preserving accuracy for previously learned patterns. An event-driven workflow orchestration architecture records state-changing guest actions as immutable events and dispatches them to independent CRM, marketing automation, event management, and analytics modules that autonomously respond without centralized orchestration, enabling high-throughput processing with low latency.

The invention provides a data synchronization configuration optimization method and device based on metadata automatic discovery and electronic equipment, and relates to the technical field of data optimizing.The method comprises the steps that metadata obtaining logic of heterogeneous data sources is abstracted through a uniform interface, and table structures, field types and primary key constraints of the data sources are automatically connected and analyzed; based on a declarative configuration engine, automatically generating a field mapping rule and a target end DDL statement according to a source library name, a table name and a target end address input by a user; a source end DDL event is monitored in real time by integrating a change data capture technology, and a target end table structure is updated by adopting a dual-stage submission protocol; and constructing a cross-data source type mapping knowledge base, and dynamically adapting a type conversion rule in combination with semantic reasoning. According to the method, the metadata acquisition logic of the heterogeneous data source is abstracted through the uniform interface, the zero configuration access of a traditional database, a big data component and an API interface is supported, the compatibility is improved, and the tedious process of manually compiling field mapping rules and type conversion scripts is eliminated.

The invention discloses a change data capturing mechanism of a data synchronization framework program, belongs to the technical field of big data and databases, and particularly relates to the change data capturing mechanism of the data synchronization framework program. A storage directory of the mechanism distinguishes different WAL files according to namespace, each namespace creates a corresponding WAL file according to a node where the namespace is located, and the file directory can be abstracted into a tree structure. WAL writing is achieved through a Disruptor framework in the writing process, and the writing process comprises the four steps of writing data into Disruptor, writing Disruptor into a WAL file, writing data into a disk and returning a writing result. According to the mechanism, the log file is firstly written in before the data is written in, and then the data is written in the memory, and the writing process follows the sequence of'first disk and then memory ', so that even if the server crashes or the node is restarted, the data before crash can be recovered by fully reading the WAL log. And if WAL writing fails, the whole operation is judged to be failed, so that the integrity and consistency of the data are fully ensured.

The application relates to a business data processing method and device, computer equipment, a storage medium and a computer program product. The method comprises the following steps: receiving first changed business data in a source database; the first changed business data is determined by a business end through a change data capture tool; converting the data format of the first changed business data to obtain first target business data; obtaining target dimension business data having a first correlation relationship with the first target business data in a target database according to the data identifier of the first target business data; the business type corresponding to the target dimension business data is different from the business type corresponding to the first target business data; generating to-be-executed business data according to the first target business data and the target dimension business data, and executing a business processing operation corresponding to the to-be-executed business data. The method can improve the business data processing efficiency.

The invention discloses a data low-delay materialized view method, which comprises the following steps: S1, after an application server initiates a database transaction to a source database and executes a data operation instruction and before the database transaction is submitted, constructing a state intention event, and relates to the technical field of databases. According to the method, the data change information of the weight is taken as the state intention event, the submission confirmation signal of the light weight is taken as the submission state event, and the state intention event and the submission state event are separated and are subjected to asynchronous parallel processing, so that the update delay of the materialized view does not depend on the whole CDC link any more; the method only depends on the transmission delay of submitting the confirmation signal, so that the transmission delay is reduced, data synchronization with lower delay is realized, and the problem that the traditional materialized view update depends on the change data capture technology and has high delay is solved.

The invention provides a database log analysis method, device and equipment, belongs to the technical field of computer information processing, and solves the problems of resource waste, structural change blockage and low scanning efficiency caused by multi-table monitoring of a traditional change data capture mechanism. The method comprises the steps of obtaining a metadata table and log data of a target database; performing query grouping processing on the metadata table to obtain a monitoring table; performing analysis and row record extraction processing on the monitoring table to obtain row record cache data; capturing and updating the log data to obtain a log record; performing grouping filtering processing on the log records to obtain business change logs; cutting and converting the business change log according to the monitoring table and the row record cache data to obtain structured data; and determining log change data according to the structured data. According to the scheme, multi-table millisecond-level change capture is achieved through single scanning, and resource consumption and data delay are remarkably reduced.

Systems and methods herein combine the benefits of message passing through a message broker with the reliability of traditional databases for long-term data, storage. By utilizing a Change Data Capture (CDC) mechanism, various embodiments facilitate publishing of messages to the message broker while capturing the nature of the database changes. By establishing the database as the primary point of interaction, the two-transaction problem that arises from coordinating between disjoint systems, such as a database and a message broker is successfully avoided. Messages are generated directly from database interactions, captured by the CDC mechanism, and asynchronously transmitted. Advantageously, system components can receive messages and execute corresponding actions, without having to query the database. Asynchronous message processing thus reduces system latency and enhances efficiency by minimizing database load and the resources needed for tire database to keep connections open to various components.

The invention relates to the technical field of energy data synchronization, in particular to a multi-source heterogeneous energy data synchronization method and system, and the method comprises the steps: obtaining data through a plug-in connector, encrypting a data block through an AES algorithm, and transmitting and storing the data block to a distributed file system through an SSL / TLS protocol, processing the acquired data by adopting a protocol adapter, and analyzing unstructured data through a machine learning model; capturing a data change log in real time by adopting a change data capturing technology, dynamically synchronizing through a streaming processing engine, and storing based on a hot and cold data layering strategy; and monitoring a data synchronization process, and responding to an abnormal event through an automatic alarm mechanism. The method has the beneficial effects that the efficient access, processing, synchronization and monitoring of multi-source heterogeneous energy data are realized by creatively integrating the plug-in connector, the protocol adapter, the CDC technology and the SeaTunnel Zeta engine.

The invention relates to the technical field of data synchronization, in particular to a MySQL and Elasticsearch data real-time synchronization method based on a message queue. The invention provides a data real-time synchronization scheme based on a message queue. Monitoring the MySQL binary log through the change data capture component, and obtaining database addition, deletion and change operation events and change information in real time; the change information is converted into a standard format and sent to a message queue; obtaining and analyzing the message content from the queue, performing data type mapping and structure conversion, and generating target data conforming to an Elasticsearch format; finally, index updating operation is executed, and consistency verification and conflict detection are carried out at the same time. According to the method, traditional timed task query is replaced by binary log monitoring, and the message queue is introduced as a middle layer, so that data real-time synchronization is realized, the influence on database performance is reduced, and the data accuracy is ensured.

The present embodiments relate to implementing change data on no-master NoSQL data stores. An optimized node can be identified from a plurality of NoSQL data storage nodes and a specialized node can be connected (e.g., collocated) to the optimized node. The specialized node can maintain change data capture (CDC) data provided by client nodes in a hash map that can be used as a point of truth for coordinating CDC data across the plurality of NoSQL data storage nodes. The plurality of NoSQL data storage nodes can identify and coordinate all read / write data obtained from multiple client devices in a geographically separated large-scale (e.g., planet scale) system to identify change data in a distributed data store. The specialized data can provide read data to devices in the large-scale system to reconcile inconsistencies in change data across nodes in the large-scale system.

The invention relates to the technical field of database security, and discloses a database field integrity protection method and system based on dynamic Hash check.The method comprises the steps that a database log file is monitored and analyzed based on a change data capturing method, data changes are captured in real time, and multi-level Hash chains are generated through layer-by-layer aggregation; and based on the generated multi-level hash chain, positioning key data tampering in combination with multi-database adaptation and a dynamic verification strategy mechanism, and triggering an alarm. According to the method, database log files are monitored and analyzed by adopting a change data capturing technology, the change of each piece of data is captured in real time, the dynamic hash chain is generated by layer-by-layer aggregation, and accurate positioning of key data tampering is realized by combining a multi-database adaptation layer and a dynamic verification strategy mechanism. When the data is written in, the hash value is stored in the independent check table, if the field in the strategy is tampered, an alarm is triggered, if the field is not configured, check is skipped, and the safety and the performance efficiency are both considered.

The application discloses a student growth tracking data processing method based on activity flow, specifically comprising the following steps: training five activity flow description models, autonomously learning activity flow, testing learning activity flow, interactive learning activity flow, sports activity flow and popular science activity flow; collecting data generated by students in different regions; storing the collected data in an ODS (On-Demand System) and updating the data by using a log-based CDC (Change Data Capture); extracting activity flow information from unstructured data, text, images / videos and audio in the ODS by using a deep learning mode; polling and calling SQL Merge by using an encapsulated ETL Engine (Extract, Transform, Load) to send the updated data into a DW (Data Warehouse); performing feature-level data fusion and decision-level data fusion on the data in the DW according to the preset five activity flow description models; and the method can be used for student growth tracking analysis.

This invention provides a method for automatic synchronization of multiple databases under an AI platform, belonging to the field of automatic database synchronization technology. This invention initializes the multi-database synchronization topology and deploys a change data capture module to monitor data changes in real time. The captured change records are converted into unified format synchronization messages and input into an intelligent scheduling optimization model for dependency analysis and transaction reordering based on dynamic programming. Data conversion rules are dynamically generated based on mode version differences, and Merkle tree hierarchical structure is used to verify data consistency. After conflict detection and circular dependency handling are performed at the target node, multi-level adaptive compression transmission and data writing are completed. Finally, periodic full reconciliation tasks ensure eventual consistency among the multiple databases, solving the technical problem of low transaction scheduling efficiency leading to synchronization performance bottlenecks during automatic multi-database synchronization.

The invention discloses an entity library table generation method and system based on a dynamic data model, and the system comprises a dynamic materialization engine module which configures a server cluster, deploys a three-node ZooKeeper cluster for a metadata server, and is responsible for storing model version metadata; a difference calculation module of an FPGA acceleration card is configured for a calculation node, and the Levenshtein editing distance is calculated; the elastic storage distribution module is used for setting a storage distribution strategy of a mixed layout of column storage and row storage and recombining a data space; and the model storage consistency guarantee module is used for setting a transaction log capture layer at a cloud end to monitor a change event of the model definition library, generating a CDC change data capture log with a timestamp, and configuring a two-stage materialization protocol at the same time. According to the scheme, the problem of service interruption caused by the fact that a traditional relational database needs to be shut down to execute DDL operation is solved, and real-time synchronization from model change to table structure adjustment is achieved.