35 results about "Change data capture" patented technology

A method of change data captured is disclosed, in which modifications made to on-line transaction processing (OLTP) tables (e.g. inserts, updates, and deletes) are maintained in a database object, referred to as a change table. The change data in the change table is then made available to analysis applications in a controlled manner, such as in accordance with a publication-subscription model.

Utilizing change data capture methodology, a computer facilitated historical data warehousing system accepts transactional / operational data having no directly tied dates from a legacy source system and processes that data into a subject-oriented format that is optimized for analytical and query reports via a two step process comprised of pre-processing and transforming. The pre-processing step entails a serial modification of data with only the last modification being recorded. The transforming step involves linking related data by utilizing reusable primary keys and dates obtained from an RDBMS in an operation system of the legacy source system that supplied the transactional / operational data.

Systems and techniques are disclosed relating to management of a database. A method may include maintaining, by a computer system, a multi-tenant database operable to store a plurality of objects. Each object may be capable of including up to a maximum potential number of definable fields. Each field may have an associated index number that has been assigned sequentially. The method may include receiving user-defined data values for a subset of the maximum potential number of fields of a particular object, and storing, for the particular object, a boundary value indicating a range of index values that have been defined for the particular object. The method may further include, in response to determining that the particular object has been accessed, selectively processing a number of fields of the particular object. The number of fields may be determined based on the stored boundary value for the particular object.

A method and software for change data capture are disclosed, in which change data is extracted from a recovery log and stored in a database object. The change data indicates what modification has been performed to a source object on the OLTP system. In some embodiments of the present invention, a database statement may be generated and executed to extract and load the change data, and the recovery log itself may be shipped from an OLTP system to a staging system.

A synchronous change data capture system and methodology are described in which, for each statement of a transaction, a transaction identifier that uniquely identifies each transaction is recorded along with the change data. When the transaction is committed, the transaction identifier and a system change number for the commit is recorded in a transaction table. To identify the commit system change number for each statement in the change data, the transaction identifier in the change data is used to fetch the commit system change number from the transaction table.

A method for detecting granular data store changes in a computing environment. Data values stored in a first relational database are changed. Each data value is stored in a row and a column of the first relational database. Rows (i.e., changed rows) including the changed data values are identified. Changed rows are propagated to a second relational database. Columns including the changed data values are identified in the second relational database. The types of changes associated with the changed data values are identified. Filters optionally filter out one or more of the changed data values from further processing. At least one data value of the changed data values, along with their types of changes, are transformed into a language neutral format included in packaged data to be sent in response to a client requesting the at least one data value.

A method and software for change data capture is described, in which change data is extracted from a recovery log in a single pass and stored into multiple database objects, such as change tables. The change data indicates modifications that has been performed to multiple source objects that correspond to the multiple database objects. In a preferred embodiment, this can be accomplished with a single multi-table insert SQL statement.

A system, method and computer-readable medium for data capture for replication are provided. A data record from a transaction log of a source database indicative of a data element change is retrieved. A DDL command is generated from the retrieved data record. Once generated, the DDL command is distributed for replication to a target database such that the source database and the target database are synchronized.

Systems and methods for dynamically creating Entity-Specific Views from one or more Data Warehouses are provided. The system may involve multiple data sets within a Data Warehouse and may provide a Target Result data set which excludes fields of information not relevant to the desired view. The system may involve the use of derived fields and logical decision structures for the assignment of overriding boundary date information.

The invention relates to a vertical intelligent crawler data collecting method based on webpage data capture. The vertical intelligent crawler data collecting method is characterized by comprising the steps that firstly, a crawler data initial inlet address is configured to a starting module through starting / stopping of an inlet configuration module; then a crawler control system performs a depth-first algorithm to perform traversal webpage capture according to set capture rules and a set capture process; finally, crawlers perform analysis extraction on webpage data and storing extracted two-dimensional structure data through a rule sequence pair of a rule configuration system. Therefore, the universality requirements of the crawlers can be met, analysis rule configuration, webpage depth and thread capturing, database configuration or index configuration can be increased on specific service logic, and intelligent information capture can be started. An intelligent crawler frame can be effectively formed, meanwhile automatic data archival and classified storage can be achieved, and a mode that distributed key values are adopted for database storage can be adopted.

The present invention contemplated the use of Change Data Capture (CDC) between a source database and a target database. To execute CDC, one or more computer processors execute the steps of obtaining a plurality of log records comprising information on transactions processed in the source database, grouping the plurality of log records into a plurality of groups based on a predetermined condition, and determining that the plurality of obtained log records is to be replicated from the source database to the target database in parallel in a unit of a group.

The invention discloses a method for realizing incremental data extract based on a CDC (Change Data Capture) mode and belongs to the application field of computer networks and database technologies. The method comprises the following steps of: firstly, providing a set of complete and feasible solution schemes for incremental data synchronism from a source database table to a target database table; secondly, carrying out incremental extract by using a CDC technology without modifying a source list data structure. According to the method, while the operating performance of a system is improved, the data transmission quantity is reduced and the network load is alleviated.

The present invention contemplates steps for a change data capture (CDC) between a first database and a second database when executed by one or more processors of the computer system. The disclosed method contemplates extracting a data in a redo log of a first database by a first management module, the data comprising an encrypted column data included in an encrypted column and a non-encrypted column data included in a non-encrypted column, identifying whether the data extracted is the encrypted column data included in the encrypted column, and determining whether to change the encrypted data included in the encrypted column to a null data and to transmit the encrypted data, when the extracted data is the encrypted column data.

The invention discloses a MapReduce-based CDC (Change Data Capture) method of an MYSQL database. The MapReduce-based CDC method comprises the steps of (1) generating a query statement 'select into outfile' of an abstract, and setting a zone bit according to a FIELDS clause; inserting an 'attribute value separator' into a line of tuples obtained by searching the database by the 'select into outfile'; generating abstract md5value and generating an output format for a searching result of 'select into outfile' according to a zone bit value; writing the searching result into a disk file outfile; (2) calculating difference by adopting a Hadoop MapReduce parallel framework; reading in two snapshoot files of old.txt and new.txt from a map end, storing a value of same keys in a Key / value structure in an iterator by a shuffle function of MapReduce, and synthesizing an output file of reduce into an insert file and a delete file, i.e obtaining a CDC result. According to the MapReduce-based CDC method disclosed by the invention, both grammar and implementation of the query statement in MYSQL is improved, a snapshoot file with the abstract can be generated by searching a data file of the database in one step, one I / O (Input / Output) is reduced by the generation of one snapshoot file, and a large amount of I / O can be reduced by multiple continuous snapshoot difference processes.

The invention relates to a CDC (changed data capture) data distribution method and a CDC data distribution device, wherein the method comprises the following steps: step (1) of configuring extraction information, configuring the extraction information for extracting data in a database through a configuration interface; step (2) of extracting process, reading the configured extraction information, extracting the data in a source database from the database of a system related to businesses to generate a text file; step (3) of configuring loading information, configuring the loading information for loading the data in the database through the configuration interface; step (4) of loading process, reading the loading information, and loading the text file derived during the extraction process into a target database for storing the extracted text file. According to the CDC data distribution method and device provided by the invention, a configuration mode is graphically and flexibly opened, data is extracted fast, data extraction is fully implemented according to source data API, extraction and overloading are performed by the way of pipelining operation.

In accordance with an embodiment, described herein is a system and method for capture of change data from a distributed data source system, for example a distributed database or a distributed data stream, and preparation of a canonical format output, for use with one or more heterogeneous targets, for example a database or message queue. The change data capture system can include support for features such as distributed source topology-awareness, initial load, deduplication, and recovery. A technical purpose of the systems and methods described herein includes determination and communication of changes performed to data at a distributed data source that includes a large amount of data across a plurality of nodes, to one or more target computer systems.

According to an embodiment of the present disclosure, there may be provided a computer program stored in a computer readable storage medium, which includes encoded commands, wherein executing the computer program by one or more processors of a computer system allows the one or more processor to perform steps for change data capture (CDC) between a source database and a target database, in which the steps include: a replication step of permitting a real-time synchronization process for objects to be performed by a real-time CDC module; a recovery start step of encapsulating, when at least one non-integrity object is present between the source database and the target database, the at least one non-integrity object in a recovery target object; a non-stop replication step of permitting a real-time synchronization process to be performed with respect to objects other than the recovery target object among the objects by the real-time CDC module; and a recovery step of permitting the recovery target object to be recovered by a recovery module and a recovery CDC module, the recovery step being performed in parallel with the non-stop replication step.

A log is identified which includes one or more transactions to a source database. The log is ingested using a processor included in a storage processor complex, where the storage processor complex includes storage and a processor associated with the storage. The ingested information is transformed into change data appropriate for a target database stored at least in part on the storage processor complex. The change data is applied to the target database.

The invention discloses a bidirectional synchronization system and method for the internal and external network data of an ORACLE database. The system comprises a change data capture module and a dataextraction module, wherein the change data capture module obtains change data through the flashback query function of the ORACLE; and the data extraction module is used for extracting changed data and updating the changed data to a corresponding internal and external network database when new data is in the presence. Compared with the prior art, the bidirectional synchronization system and methodfor the internal and external network data of the ORACLE database solves the problems that the database requires mutually independent internal and external networks and internal and external networkdata needs to be kept consistent on an aspect of business since an application needs to have accessibility in the internal and external network can be solved.

The invention discloses a method and device for improving safety of data interaction between enterprises. The device is arranged in a DMZ space deployed between an enterprise intranet and an externalnetwork. The device comprises an MQ module used for organizing external incoming data in a message queue mode; a preprocessing module used for sequentially preprocessing the data organized by the message queue, wherein preprocessing comprises data cleaning, data integration and data transformation; and a CDC module used for synchronizing the preprocessed data flow to an enterprise intranet database in real time by adopting a changed data capture mode. According to the device and the method, high security and high stability of data exchange between enterprises are improved.

Disclosed is a computer program stored in a computer-readable storage including encoded commands. When the computer program is executed by one or more processors of a source database server, the computer program performs steps for Change Data Capture (CDC) by one or more processors. The steps may include: identifying a plurality of Structured Query Language (SQL) operations generated for one or more objects associated with a transaction by analyzing a log record including information about the transaction processed at a source database server; determining whether a previous SQL operation history for an object corresponding to an individual SQL operation is in a CDC file, based on whether an order in which the plurality of identified SQL operations occur; determining information to write on the CDC file based on whether a previous SQL operation history is in the CDC file; writing the determined information on the CDC file; and deciding to send the CDC file to a target database server.

An embodiment of the present invention is directed to configuring optimal parallelism on Hadoop processes at specific processing points for improved performance and throughput. An embodiment of the present invention forces Hadoop into high parallelism independent of a HDFS file size. An embodiment of the present invention achieves an order of magnitude improvement in performance and data throughputs to meet the SLAs. An embodiment of the present invention provides a design pattern unique to distributed processing in Hadoop to accomplish the required data throughputs for the CDC process.

Disclosed is a computer program stored in a computer readable storage medium, which includes encoded commands, wherein executing the computer program by one or more processors of a computer system allows the one or more processors to perform steps for change data capture (CDC) between a source database and a target database. The steps may include: a resource data acquisition step of acquiring resource data which is data obtained by monitoring a current resource of a source database server; a comparison information generation step of generating comparison information by comparing the resource data and a predetermined source database server load threshold value; a mode determination step of determining an operation mode of a changed data capture process which operates in the source database server based on the comparison information, the operation mode including a union mode and a division mode; and a mode changing step of changing the operation mode of the process to a mode determined in the mode determination step.

The invention relates to a business data processing method and device, computer equipment, a storage medium and a computer program product. The method comprises the following steps: receiving first change service data in a source database; the first change service data is determined by the service end through a change data capture tool; converting the data format of the first change service data to obtain first target service data; according to the data identifier of the first target business data, obtaining target dimension business data having a first association relationship with the first target business data from a target database; the business type corresponding to the target dimension business data is different from the business type corresponding to the first target business data; and generating to-be-executed service data according to the first target service data and the target dimension service data, and executing a service processing operation corresponding to the to-be-executed service data. By adopting the method, the business data processing efficiency can be improved.

The invention discloses a unified processing method for change data capture, and relates to the technical field of data capture, and the method comprises the steps: based on the data source type characteristics of a health medical area platform, appointing a processing method, obtaining data with a unique identifier, and storing the data in a target database; recording each piece of data of each data set in the target database; performing attribute marking on each data record of the target database; defining a feature calculation function, and taking the mark attribute of the data as input to obtain a feature code; acquiring the data again, storing the data in the target database, capturing changed data based on the data identifier and the feature code comparison, and marking a piece of changed data; and storing the change mark of the data in the target database for a downstream data processing flow to use. The invention also discloses a unified processing tool for capturing the change data, which is combined with the method, performs unified processing on the acquired data, and reduces the workload of development and maintenance.

The invention discloses a data change storage method and device, electronic equipment and a medium, and the method comprises the steps: obtaining change data capture logs of N source databases with started change data capture functions, N being a positive integer, and change data recorded by the change data capture logs comprising change positions and change contents; according to the change position, determining a target table in a target storage database corresponding to the change data in a distributed storage system; and according to a change type represented by the change content, storing the change data to a corresponding target column cluster in the target table. The method and device, the electronic equipment and the medium are used for solving the technical problems that in the prior art, data change storage needs to change the structure of a source database, and the storage amount is large, and the technical effects that data change storage does not need to change the structure of the source database, and storage consumption is reduced are achieved.

The invention discloses a MapReduce-based CDC (Change Data Capture) method of an MYSQL database. The MapReduce-based CDC method comprises the steps of (1) generating a query statement 'select into outfile' of an abstract, and setting a zone bit according to a FIELDS clause; inserting an 'attribute value separator' into a line of tuples obtained by searching the database by the 'select into outfile'; generating abstract md5value and generating an output format for a searching result of 'select into outfile' according to a zone bit value; writing the searching result into a disk file outfile; (2) calculating difference by adopting a Hadoop MapReduce parallel framework; reading in two snapshoot files of old.txt and new.txt from a map end, storing a value of same keys in a Key / value structure in an iterator by a shuffle function of MapReduce, and synthesizing an output file of reduce into an insert file and a delete file, i.e obtaining a CDC result. According to the MapReduce-based CDC method disclosed by the invention, both grammar and implementation of the query statement in MYSQL is improved, a snapshoot file with the abstract can be generated by searching a data file of the database in one step, one I / O (Input / Output) is reduced by the generation of one snapshoot file, and a large amount of I / O can be reduced by multiple continuous snapshoot difference processes.

The present invention contemplates a non-transitory computer readable medium including a computer program steps for a change data capture (CDC) between a first database and a second database when executed by one or more processors of the computer system. The disclosed non-transitory computer readable medium contemplates receiving data from a first database by a second transceiver module, the data including a null data contained in an encrypted column and a data contained in a non-encrypted column of a table, generating, by the second management module, a change data, by reproducing transactions performed in the first database with respect to the data transmitted from the first database and performing a processing on a transaction before commit, and determining, by the second management module, to request an encrypted column data corresponding to the null data to the first database.