56 results about "Change data capture" patented technology

The invention provides a multi-database automatic synchronization method under an AI platform, and belongs to the technical field of database automatic synchronization. The captured change records are converted into synchronous messages in a unified format, the synchronous messages are input into an intelligent scheduling optimization model for dependency analysis and transaction reordering based on dynamic programming, data conversion rules are dynamically generated according to mode version differences, and data consistency is verified through a Merkle tree hierarchical structure; the multi-stage self-adaptive compression transmission and data writing are completed after the target node executes conflict detection and circular dependency processing, and finally the final consistency among the multiple databases is guaranteed through a periodic full-amount account checking task, so that the technical problem of synchronization performance bottleneck caused by low transaction scheduling efficiency in the automatic synchronization process of the multiple databases is solved.

An example computer system includes a transactional database configured to store multiple data sources each corresponding to one of multiple applications, a standby database configured to store multiple rule schemas and multiple data schemas, and processor hardware configured to execute instructions to monitor for a change to stored data in one of the multiple data sources, via a change data capture process, replicate data from the transactional database to the standby database in response to an update indicated by the change data capture process, identify encrypted data in at least one data object of an updated data table of one of the data schemas of the standby database, extract data from the at least one data object to store decrypted data, wherein the standby database is an external database for the data extraction engine, and reload the decrypted data to one of the data schemas of the standby database.

A computer-implemented method for determining a message in a message queue of an information technology system. The method may include receiving an information technology event data object in an object-relational database management system, the information technology event data object including metadata; determining a change to the object-relational database has occurred by implementing a change data capture platform; determining, using the change data capture platform, a log file of the change to the first table, the log file including the metadata from the information technology event data object; deploying the log file to a message queue; and processing, through a consumer module, the log file of the message queue to perform an action.

The invention discloses a database detection operation method of biological feedback equipment, and relates to the technical field of database detection operation, according to the database detection operation method, in combination with a change data capture CDC technology and a Kafka Streams streaming processing framework, myoelectricity reaction data are collected and cleaned in real time, redundant processing operation of full-amount data is effectively reduced, an incremental calculation algorithm is introduced to process dynamic data, and therefore the dynamic data processing efficiency is improved. The dynamic change of data can be quickly adapted by updating the average value and the standard deviation of the detection base line; meanwhile, a priority grouping mechanism is introduced, high-priority data with large deviation are sorted according to the abnormal amplitude from large to small, and the high-priority data are immediately transmitted to a visualization module for emphasized processing; low-priority data with small deviation is processed in a batch mode according to a detection sequence, and resource allocation and response speed are optimized; and high-priority data generates an exclusive thermodynamic diagram and a tendency diagram through an independent process, and low-priority data is integrated into a grouped thermodynamic diagram and a statistical distribution diagram, so that the intuition and practicability of data interpretation are remarkably improved.

The invention relates to the technical field of transaction data aggregation, and discloses a near-real-time aggregation method and system for heterogeneous transaction system data, and the method comprises the steps: obtaining real-time change data from a plurality of heterogeneous transaction systems through a change data capturing technology, and standardizing the change data into a uniform format; performing Hash routing calculation on the standardized change data based on the order number to generate a Hash value, and routing the change data of the same order to the same message queue according to the Hash value; an independent consumer thread is allocated to each message queue through a multi-queue parallel processing mechanism, and the number of the consumer threads is dynamically adjusted according to the message accumulation amount; according to the method, the processed change data is aggregated according to the preset core field template, the aggregated data is stored in the aggregation database, cross-system unified query and analysis are supported, and the problems that heterogeneous data query is difficult, order affiliation judgment is complex, cross-system data statistical analysis is obstructed, and data synchronization timeliness is not high are solved.

Described herein are methods and systems for updating knowledge bases for Retrieval-Augmented Generation (RAG) applications. The methods employ Change Data Capture (CDC) to efficiently detect modifications in source data. These CDC techniques may enable targeted updates to semantic indexing tables by traversing data models from leaf tables to root entities, ensuring that only affected embeddings are regenerated rather than reprocessing entire document collections.

Systems and methods for real-time data integration, analysis, and notification within a mobile application system are provided. Embodiments include initializing a data layer and a pre-processing phase to store structured and unstructured data with a distributed database policy. A real-time data grid (RTDM) continuously extracts data from various platforms using signal processing methods and machine learning techniques for noise removal and data feature extraction. An advanced analysis and machine learning (AAML) engine processes the data, while a decision construct derives appropriate actions. A push notification service is activated based on an event-driven architecture (EDA) or a publish-subscribe (publish-subscribe) system to deliver customized notifications. The technical precision ensures that relevant information reaches the end user in time. The system optimizes operation through a self-adaptive feedback mechanism and protects data through encryption.

The invention discloses a semiconductor device production data processing method and system, and the method comprises the steps: obtaining production process data generated in the production process of a semiconductor device, and the production process data comprises at least one of enterprise resource planning (ERP) data, production execution system (MES) data, and quality assurance system (QA) data, supplier production order online information WIP data; converting unstructured data in the production process data into structured data by adopting a heterogeneous data source off-line synchronous DATAX method; and carrying out data processing on all the structured data by adopting a change data capture (CDC) method, and synchronizing a data processing result to a distributed storage system database. According to the scheme, different formats and different types of data generated in the semiconductor production process can be integrated, and subsequent data analysis is facilitated.

The invention provides a data synchronization desensitization method and device, computer equipment and a readable storage medium. The method comprises the following steps: pre-configuring synchronization parameters; receiving a change event of the source database output by the change data capture engine; performing desensitization processing on the event load of the change event according to a preset desensitization function and a desensitization parameter; judging the event type of the change event; according to the event type and the dialect identifier, mapping an abstract type of a corresponding field in the event load into a physical type which can be understood by the target database to obtain type information of the field; generating a parameterized equivalent SQL template compatible with the target database dialect according to the dialect identifier, the event type and the type information of the field; the event load is bound to the parameterized equivalent SQL template, and a bound parameterized SQL statement is obtained; and executing the parameterized SQL statement in the target database. According to the invention, a set of general data synchronization and desensitization mechanism is constructed.

This invention discloses a low-latency materialized view method, comprising the following steps: S1, after the application server initiates a database transaction to the source database and executes data operation instructions, before the database transaction is committed, a state intent event is constructed. This relates to the field of database technology. This application uses heavy data change information as the state intent event and a lightweight commit confirmation signal as the commit state event. By separating the state intent event and the commit state event and processing them asynchronously in parallel, the update latency of the materialized view no longer depends on the entire CDC link, but only on the transmission latency of the commit confirmation signal. This reduces transmission latency, achieves lower-latency data synchronization, and alleviates the high latency problem inherent in traditional materialized view updates that rely on change data capture technology.

The invention provides a device for automatic metadata management and data knitting treatment, and the device comprises a semantic mounting unit which is used for carrying out the semantic tag mounting of a structured metadata set of a data factory based on a semantic model of a data knitting architecture, so as to generate enhanced semantic metadata and a complete data line record; the rule binding unit is used for carrying out governance rule binding on the enhanced semantic metadata and the complete data line record on the basis of a data knitting federation governance rule so as to generate a metadata governance list bound with the governance rule; the change updating unit is used for synchronously updating a data set state field and a flow path recorded by a corresponding complete data line to a metadata governance list based on a change data capture mechanism when it is monitored that a data factory has a data set change, so as to generate unified metadata caalog; the interface configuration unit is used for configuring a metadata retrieval interface and a governance rule execution engine for the unified metadata caalog so that the unified metadata caalog can be accessed in the data weaving environment.

A method, system, and computer program product configured to perform operations including: detecting, using a first thread associated with a first change data capture (CDC) replication mode, that a database transaction associated with a source database exceeds a threshold time value; based on the detecting, spawning a second thread that is configured to identify log entries associated with the database transaction; confirming the database transaction qualifies as a long transaction based on a commit operation in the log entries associated with the database transaction; based on the confirming, switching to a second replication mode for a table associated with the database transaction; updating the table in a target database using values obtained by executing queries on the source database; and based on the updating the table in the target database, terminating the second thread and switching back to the first CDC replication mode for the table.

A system and method for reconciling data from a local database streamed to a Cloud storage service is disclosed. A data transaction is received from a structured query language (SQL) database service operated by a server. The transaction includes log sequence numbers and corresponding data records. The log sequence numbers are stored in a change data capture (CDC) table. The sent data transaction is streamed via a data integration service. A storage object service stores the sent data into an object storage service bucket. A data reconciliation service deploys a cloud trail target to generate a log file of log sequence numbers from the sent data stored in the object storage service bucket. The log file of log sequence numbers is compared with the log sequence numbers in the CDC table. A missing data record is flagged to an exception table based on the comparison. An alert is issued for the missing data record.

The application discloses a method and device for sensing, judging and processing data structure change. The method comprises the following steps: collecting a data structure change event based on a change data capture tool; constructing a data structure change abstract syntax tree through a SQL parser to accurately identify the change operation type and content; obtaining structure snapshots before and after the change; executing compatibility judgment in combination with a compatibility rule library to output the compatibility state and risk level; adopting a hierarchical message notification strategy according to the risk level; and automatically triggering or arranging corresponding processing actions, such as automatically synchronizing fields, generating a compatible view or suspending a high-risk task. The application realizes semantic-level analysis of data structure change, automatic compatibility judgment, refined risk grading and automatic processing, significantly reduces the cost of manual intervention, effectively avoids downstream synchronization interruption and data quality abnormalities caused by structure change, and improves the stability and automatic operation and maintenance level of a medium or large data platform.

Embodiments of the present application provide a database migration method, device, equipment, medium and program product. The method comprises: at the starting moment of migrating the inventory data of a source database to a target database, for any one of a plurality of tables to be migrated, based on a change data capture table, obtaining a first target log sequence number of a migration transaction corresponding to the table to be migrated; recording the first target log sequence number as low-water information; at the end moment of migrating the inventory data of the source database to the target database, based on the change data capture table, obtaining a second target log sequence number of a migration transaction corresponding to the table to be migrated; recording the second target log sequence number as high-water information; and based on the high-water information and the low-water information, synchronizing the incremental data of the source database to the target database. The method is used to improve the efficiency of database migration.

The application discloses a real-time fusion and incremental synchronization method and system of heterogeneous data sources, and belongs to the technical field of data synchronization. The method comprises the following steps: a change data capture step is used to parse a plurality of database logs through a source connector to obtain change events; an event standardization step is used to convert the change events into a unified format; a stream processing step is used for deduplication, sorting and transaction boundary identification; an adaptive scheduling step is used to dynamically adjust the consumption parallelism according to the load; and a consistency checking step is used to compare the data digest of the source end and the target end and repair inconsistencies. The application solves the problems of high synchronization delay, complex integration and difficult consistency guarantee of the heterogeneous data sources, controls the end-to-end delay in the order of seconds, and supports the processing of millions of events per second.

A method and system for updating backup data for a database is presented. The method includes initiating a merging operation for a defined time interval; obtaining change data capture (CDC) data generated during the defined time interval; aggregating and ordering the CDC data by a key value; identifying backup data associated with the ordered CDC data; generating a candidate data object that represents data modified during the defined time interval; deduplicating the candidate data object by comparing a hash identifier of the candidate data object with a plurality of hash identifiers stored in a repository; storing the candidate data object when the comparison determines that no matching hash identifier exists in the repository; and updating backup metadata to associate the key value with the candidate data object.

The application relates to the field of e-government informatization and archive information management, and discloses an archive coupling processing method and system based on business flow and event driving, which identifies a business settlement event through complex event processing; linkage change data capture extracts an anchor change set in a transaction consistent range and obtains a candidate fragment through difference; conclusion data is screened by adopting a lightweight Transformer, and an archiving package containing field tracing is generated by extracting and checking according to a predefined data mode; a Merkel tree is respectively constructed for content and metadata, and a ledger abstract is generated to be written into an archiving ledger, entity information is associated under a dual-track system, settlement and archiving are realized, and traceability and verification are realized.

The invention discloses a real-time data analysis and visualization system and method based on big data. The system comprises a data acquisition module, a data transmission module, a data preprocessing module, a real-time analysis module, a data storage module and a visualization module. The data acquisition module is connected with the data transmission module, the data transmission module is connected with the data preprocessing module, the data preprocessing module is connected with the real-time architecture module, and the real-time analysis module is connected with the data storage module and the visualization module. According to the method, minute-level delay of traditional Sqoop'timed pulling 'is avoided, millisecond-level access of database change data is achieved, flexible expansion of'data source-computing node-storage capacity' is achieved through hierarchical decoupling and a distributed elastic architecture, and analysis result errors are controlled within 1% through combination of whole-process data quality management and control and multi-algorithm fusion optimization.

The invention discloses a project full life cycle cost control method and system, and relates to the technical field of enterprise project management and cost control. The method comprises the following steps: accessing heterogeneous cost data from a plurality of service systems through a multi-source data acquisition framework, and realizing quasi-real-time data synchronization by adopting a change data capture technology; performing real-time processing on the accessed data by using a stream processing engine and a memory computing framework, and constructing a distributed cost knowledge graph based on a graph database, the knowledge graph being used for modeling a causal relationship among cost elements; based on the knowledge graph and the real-time data, at least one intelligent analysis operation is executed, and the intelligent analysis operation comprises self-adaptive process optimization based on reinforcement learning, self-adaptive cost anomaly detection based on multi-model fusion and high-performance cost analysis based on a multi-dimensional data cube engine. The technical problems that all links of a traditional cost management system are separated, and data flow is not smooth are solved.

The application relates to the fields of CAE simulation technology and distributed architecture, and discloses a service-oriented CAE simulation platform micro-service architecture construction method and system, which comprises the following steps: constructing a micro-service architecture, wherein the micro-service architecture comprises a core micro-service cluster, a unified service bus, a dynamic resource scheduling center and a distributed data layer; data generated by each micro-service is stored through the distributed data layer; the distributed data layer cuts unstructured data into data shards and stores the data shards in the form of multiple copies into an object storage cluster, stores structured data into a relational database, stores business metadata into a cache cluster, listens to change logs of the relational database by using a change data capture technology, and synchronizes change events to the cache cluster. The traditional monomer platform needs 3 hours to complete the same task, the application only needs 1 hour and 30 minutes, the efficiency is improved by 50%, and the task queuing time is shortened from 1 hour to 10 minutes.

The application provides a database log analysis method, device and equipment, belongs to the technical field of computer information processing, and solves the problems of resource waste, structure change blockage and low scanning efficiency caused by the traditional change data capture mechanism when monitoring multiple tables. The method comprises the following steps: obtaining a metadata table and log data of a target database; performing query grouping processing on the metadata table to obtain a monitoring table; performing analysis and row record extraction processing on the monitoring table to obtain row record cache data; performing grabbing and updating processing on the log data to obtain log records; performing grouping filtering processing on the log records to obtain business change logs; performing cutting and conversion processing on the business change logs according to the monitoring table and the row record cache data to obtain structured data; and determining log change data according to the structured data. The scheme realizes millisecond-level change capture of multiple tables through single scanning, and significantly reduces resource consumption and data delay.

A computer-implemented method for maintaining unified guest identity records across heterogeneous hospitality data systems. The method receives guest interaction data from independent data sources operating on different schemas, normalizes identifiers through standardization algorithms, and applies two-stage matching comprising deterministic cryptographic hashing followed by probabilistic Bayesian inference. Core attributes are stored in a relational database with strong consistency guarantees while behavioral data is maintained in a document database with eventual consistency, synchronized through a change data capture mechanism providing near real-time synchronization. The identity resolution engine operates continuously in real-time, reducing storage requirements compared to systems allowing duplicate records. The system further implements a predictive segmentation model using online learning to adjust parameters in real-time while preserving accuracy for previously learned patterns. An event-driven workflow orchestration architecture records state-changing guest actions as immutable events and dispatches them to independent CRM, marketing automation, event management, and analytics modules that autonomously respond without centralized orchestration, enabling high-throughput processing with low latency.

The invention provides a data synchronization configuration optimization method and device based on metadata automatic discovery and electronic equipment, and relates to the technical field of data optimizing.The method comprises the steps that metadata obtaining logic of heterogeneous data sources is abstracted through a uniform interface, and table structures, field types and primary key constraints of the data sources are automatically connected and analyzed; based on a declarative configuration engine, automatically generating a field mapping rule and a target end DDL statement according to a source library name, a table name and a target end address input by a user; a source end DDL event is monitored in real time by integrating a change data capture technology, and a target end table structure is updated by adopting a dual-stage submission protocol; and constructing a cross-data source type mapping knowledge base, and dynamically adapting a type conversion rule in combination with semantic reasoning. According to the method, the metadata acquisition logic of the heterogeneous data source is abstracted through the uniform interface, the zero configuration access of a traditional database, a big data component and an API interface is supported, the compatibility is improved, and the tedious process of manually compiling field mapping rules and type conversion scripts is eliminated.

The invention discloses a change data capturing mechanism of a data synchronization framework program, belongs to the technical field of big data and databases, and particularly relates to the change data capturing mechanism of the data synchronization framework program. A storage directory of the mechanism distinguishes different WAL files according to namespace, each namespace creates a corresponding WAL file according to a node where the namespace is located, and the file directory can be abstracted into a tree structure. WAL writing is achieved through a Disruptor framework in the writing process, and the writing process comprises the four steps of writing data into Disruptor, writing Disruptor into a WAL file, writing data into a disk and returning a writing result. According to the mechanism, the log file is firstly written in before the data is written in, and then the data is written in the memory, and the writing process follows the sequence of'first disk and then memory ', so that even if the server crashes or the node is restarted, the data before crash can be recovered by fully reading the WAL log. And if WAL writing fails, the whole operation is judged to be failed, so that the integrity and consistency of the data are fully ensured.

The application relates to a business data processing method and device, computer equipment, a storage medium and a computer program product. The method comprises the following steps: receiving first changed business data in a source database; the first changed business data is determined by a business end through a change data capture tool; converting the data format of the first changed business data to obtain first target business data; obtaining target dimension business data having a first correlation relationship with the first target business data in a target database according to the data identifier of the first target business data; the business type corresponding to the target dimension business data is different from the business type corresponding to the first target business data; generating to-be-executed business data according to the first target business data and the target dimension business data, and executing a business processing operation corresponding to the to-be-executed business data. The method can improve the business data processing efficiency.

The invention discloses a data low-delay materialized view method, which comprises the following steps: S1, after an application server initiates a database transaction to a source database and executes a data operation instruction and before the database transaction is submitted, constructing a state intention event, and relates to the technical field of databases. According to the method, the data change information of the weight is taken as the state intention event, the submission confirmation signal of the light weight is taken as the submission state event, and the state intention event and the submission state event are separated and are subjected to asynchronous parallel processing, so that the update delay of the materialized view does not depend on the whole CDC link any more; the method only depends on the transmission delay of submitting the confirmation signal, so that the transmission delay is reduced, data synchronization with lower delay is realized, and the problem that the traditional materialized view update depends on the change data capture technology and has high delay is solved.

The invention provides a database log analysis method, device and equipment, belongs to the technical field of computer information processing, and solves the problems of resource waste, structural change blockage and low scanning efficiency caused by multi-table monitoring of a traditional change data capture mechanism. The method comprises the steps of obtaining a metadata table and log data of a target database; performing query grouping processing on the metadata table to obtain a monitoring table; performing analysis and row record extraction processing on the monitoring table to obtain row record cache data; capturing and updating the log data to obtain a log record; performing grouping filtering processing on the log records to obtain business change logs; cutting and converting the business change log according to the monitoring table and the row record cache data to obtain structured data; and determining log change data according to the structured data. According to the scheme, multi-table millisecond-level change capture is achieved through single scanning, and resource consumption and data delay are remarkably reduced.

Systems and methods herein combine the benefits of message passing through a message broker with the reliability of traditional databases for long-term data, storage. By utilizing a Change Data Capture (CDC) mechanism, various embodiments facilitate publishing of messages to the message broker while capturing the nature of the database changes. By establishing the database as the primary point of interaction, the two-transaction problem that arises from coordinating between disjoint systems, such as a database and a message broker is successfully avoided. Messages are generated directly from database interactions, captured by the CDC mechanism, and asynchronously transmitted. Advantageously, system components can receive messages and execute corresponding actions, without having to query the database. Asynchronous message processing thus reduces system latency and enhances efficiency by minimizing database load and the resources needed for tire database to keep connections open to various components.

The invention relates to the technical field of energy data synchronization, in particular to a multi-source heterogeneous energy data synchronization method and system, and the method comprises the steps: obtaining data through a plug-in connector, encrypting a data block through an AES algorithm, and transmitting and storing the data block to a distributed file system through an SSL / TLS protocol, processing the acquired data by adopting a protocol adapter, and analyzing unstructured data through a machine learning model; capturing a data change log in real time by adopting a change data capturing technology, dynamically synchronizing through a streaming processing engine, and storing based on a hot and cold data layering strategy; and monitoring a data synchronization process, and responding to an abnormal event through an automatic alarm mechanism. The method has the beneficial effects that the efficient access, processing, synchronization and monitoring of multi-source heterogeneous energy data are realized by creatively integrating the plug-in connector, the protocol adapter, the CDC technology and the SeaTunnel Zeta engine.