Using change data capture to enable asynchronous processing in distributed database systems

Abstract

Systems and methods herein combine the benefits of message passing through a message broker with the reliability of traditional databases for long-term data, storage. By utilizing a Change Data Capture (CDC) mechanism, various embodiments facilitate publishing of messages to the message broker while capturing the nature of the database changes. By establishing the database as the primary point of interaction, the two-transaction problem that arises from coordinating between disjoint systems, such as a database and a message broker is successfully avoided. Messages are generated directly from database interactions, captured by the CDC mechanism, and asynchronously transmitted. Advantageously, system components can receive messages and execute corresponding actions, without having to query the database. Asynchronous message processing thus reduces system latency and enhances efficiency by minimizing database load and the resources needed for tire database to keep connections open to various components.

Description

Attorney Docket: 120179-0591W001USING CHANGE DATA CAPTURE TO ENABLE ASYNCHRONOUS PROCESSING IN DISTRIBUTED DATABASE SYSTEMSBACKGROUNDField

[0001] The present disclosure is generally directed to distributed database systems. More specifically, present disclosure relates to utilizing change data capture (CDC) mechanisms to enable asynchronous processing systems and methods in distributed database systems.Related Art

[0002] Many computer software systems integrate some sort of database. Tliis database stores the current state of some form of data interesting to the larger software system. This database is centralized within the system, with most other components interacting with it in one way or another. In such systems, multiple components interact indirectly with each other by directly querying the database for changes or mutating shared data. This leads to significant database load as components frequently query for relevant updates, especially in distributed systems where independent scaling of components exacerbates the issue. High database contention and complex interactions between components make it difficult to ensure consistency and scalability'. Furthermore, existing approaches to alleviate these issues, such as using message brokers, often result in additional complexity and potential transactional failures due to the disconnect between the database and the messaging systems.

[0003] Accordingly, it is desirable to have systems and methods that reduce tire reliance on database querying, simplify the coordination between components, and facilitate scalable asynchronous processing in distributed environments.SUMMARY

[0004] In some aspects of the disclosure, a database system for asynchronous processing comprises a database that comprises: an audit table; and a stored procedure configured to perform, in response to receiving from a component with access to the database system an instruction to change a value in the database, an update; and a CDC processor configured to perform steps comprising: performing the update; causing the stored procedure to insert into tlie audit table an audit record comprising information regarding the update; in response to the audit record being inserted into the audit table, automatically generating a CDC record that represents the change in the audit table; and an audit stream processor configured to: inAttorney Docket: 120179-0591W001response to asynchronously receiving at least part of the CDC record, process the CDC record; based on a content of the CDC record, determine an action without requiring another component to query tire database; and route the action to an audit action queue based on one or more predefined routing criteria comprising at least one of a priority or a an action type; and a set of audit action queue processors that each is configured to initiate one or more actions. GOOS] In some aspects, the stored procedure may be configured to update a value in the database when inserting the audit record into the audit table.

[0006] In some aspects, the set of audit action queue processors may be configured to process messages from one or more audit action queues to perform one or more actions associated with the messages. Exemplary actions may comprise a replication task that synchronizes data between different database instances.

[0007] In some aspects, each of the action queue processors in the set of audit action queue processors may be configured to scale independently based on at least one of action frequency or action complexity.

[0008] In some aspects, the database system may further comprise a raw stream processor that converts a raw CDC stream, which comprises a list of audit entries that is output by the CDC processor, into an audit stream compatible with the audit stream processor.

[0009] In some aspects, the CDC processor, tire audit stream processor, and / or the audit action queue processors may operate asynchronously.

[0010] In some aspects, the audit stream processor, the audit action queue processors, and / or the CDC processor may cooperate with each other to retry' a failed message transmission, without requiring reprocessing of the CDC record.

[0011] In some aspects, a database method for asynchronous processing in a database management system comprises: in response to receiving from a component, at a database system, a request to update a database, performing steps comprising: performing an update on the database; and inserting into an audit table within the database an audit record that comprises information regarding the update; using a CDC record, which has been automatically generated in response to the audit record being inserted into the audit table, to asynchronously communicate at least part of the CDC record to at least one of an audit stream processor that uses a content of the CDC record to determine an action without using another component to query the database, wherein the CDC record represents a change in the audit table; and communicating the action to an audit action queue processor that initiates the action.

[0012] In some aspects, the CDC record is communicated to the audit stream processor asynchronously with the action being communicated to the audit action queue processor.Attorney Docket: 120179-0591W001

[8013] In some aspects, the audit stream processor may determine the action by referencing a pre-configured mapping of CDC records to audit action types.

[0014] In some aspects, tire CDC record comprises information sufficient to perform the action, thereby reducing a need for the another component to query the database for additional state information.

[0015] In some aspects, the method further comprises independently scaling each action queue processors in a set of audit action queue processors based on at least one of action frequency or action complexity.

[0016] In some aspects, the method further comprises using the set of audit action queue processors to process messages from one or more audit action queues to perform one or more actions comprising a replication task that synchronizes data between different database instances.

[0017] In some aspects, the method further comprises converting a raw CDC stream output by a CDC processor into an audit stream that is compatible with the audit stream processor.

[0018] Aspects of the present disclosure can involve a system, which can involve means for, in response to receiving from a component, at a database system, a request to update a database, updating the database, and inserting into an audit table within the database an audit record that comprises information regarding the update; means for using a CDC record, which has been automatically generated in response to the audit record being inserted into the audit table, to asynchronously communicate at least part of the CDC record to at least one of an audit stream processor that uses a content of the CDC record to determine an action without using another component to query' the database, wherein the CDC record represents a change in the audit table; and means for communicating the action to an audit action queue processor that initiates the action.Attorney Docket: 120179-0591W001BRIEF DESCRIPTION OF DRAWINGS0019 FIG. 1 is a diagram illustrating the ‘‘double transaction” problem in common database system architectures.

[0020] FIG. 2 is a diagram illustrating a direct access architecture for traditional database system architectures.

[0021] FIG. 3 is a diagram illustrating a hybrid architecture for traditional database system architectures,

[0022] FIG. 4 is a diagram of an exemplar}' asynchronous processing system, according to various embodiments of the present disclosure.

[0023] FIG. 5A depicts an exemplary CDC record according to various embodiments of the present disclosure.

[0024] FIG. 5B depicts an exemplary audit record according to various embodiments of the present disclosure.

[0025] FIG. 6 is a flowchart illustrating a process for asynchronous processing in a database management system by using CDC to monitor and process database changes according to various embodiments of the present disclosure.

[0026] FIG. 7 illustrates an example computing environment according to various embodiments of the present disclosure.Attorney Docket: 120179-0591W001DETAILED DESCRIPTION OF THE INVENTION0027 The following detailed description provides details of the figures and example implementations of the present application. Reference numerals and descriptions of redundant elements between figures are omitted for clarity. Terms used throughout the description are provided as examples and are not intended to be limiting. For example, the use of the term “automatic” may involve fully automatic or semi-automatic implementations involving user or administrator control over certain aspects of the implementation, depending on the desired implementation of one of ordinary7skill in the art practicing implementations of the present application. Selection can be conducted by a user through a user interface or other input means, or can be implemented through a desired algorithm. Example implementations as described herein can be utilized either singularly or in combination and the functionality of the example implementations can be implemented through any means according to the desired implementations. In this document, tire terms “audit processor” and “audit stream processor” are used interchangeably. Similarly, the terms “raw processor” and “raw stream processor” are used interchangeably.[8028 As mentioned in the background, in many software systems, multiple components interact indirectly with each other by directly querying the database for changes or mutating shared data, which significantly increases the load on the database. After mutating the state, each component must query the database to identify the data with the matching state that component is interested in and take the action appropriate to that state. This process necessarily involves at least two database interactions: (1) the state mutation and (2) the query’ for the relevant state. When multiple components are simultaneously mutating and querying the same data, this results in a combinatorial effect for state mutations and queries placing a tremendous load on the database system.

[0029] The issue i s exacerbated when components perform periodic tasks, such as query-ring or updating the database at predetermined intervals (e.g., hourly). During these intervals, the database system often experiences large spikes in load, leading to latency and potential performance bottlenecks,

[0030] This is further compounded in distributed systems, where components can be added or scaled “independently.” Despite their scalability, all components remain dependent on the centralized database, which imposes a bottleneck on overall system performance.

[0031] One common approach to alleviate this issue is to use a dedicated message broker system alongside the database system, as shown in FIG. 1. By passing messages through the message broker, components can be notified of database state changes without querying theAttorney Docket: 120179-0591W001database directly. While this reduces the frequency of queries, it introduces the “double transaction” problem, where the database commit and message publication must be synchronized across two separate systems, thus adding complexity and enhancing the risk of inconsistency.

[0032] FIG. 1 is a diagram illustrating the “double transaction” problem in common database system architectures. Database system 100 in FIG. 1 comprises database 102, actor 240, and message broker 250.

[0033] As depicted, database 102 stores an audit table 104 that contains a set of actions; actor 240, coupled between database 102 and message broker 250, communicates database transactions 242 to database 102 and publishes messages via transactions 244 to message broker 250.

[0034] Various components (not shown) in database system 100 communicate by passing messages through message broker 250. In many cases, the messages that are published to message broker 250 in this manner are “notifications” of a change in a state of database 102. As a result, the receiver of the message does not need to query' database 102 for data of interest, as the message contains sufficient information for a task or action to be performed. However, this approach has a significant downside as it requires two separate systems that must perform two transactional tasks, while each system that can only guarantee transactionality within itself For example, a database commit may succeed, but publishing a message to message broker 250 may fail (and cannot be deleted — a property of message brokers) and vice-versa. This requires that components in system 102 be aware of the possibility' that each system may' experi ence a failure. Tirus, the components must interact with database 102 and message broker 250 in combination to ensure that information does not get inadvertently lost. This makes the management components more complex on both the writing / mutating side and on the reading / consuming side of system 100.

[0035] Tire problem of database load remains unsolved, since all components still need to query the database for missed messages, albeit at a lower frequency than if querying were the primary’ method of finding work. Some existing approaches, such as the one discussed further below with reference to FIG. 3, attempt to mitigate this problem by directly accessing the database and continuously scanning the audit table for potential changes. Other approaches record changes into row s of a dedicated table and then use another system component to query and retrieve changes from that table. However, these methods introduce additional operations, such as deletion of processed rows, requiring multiple database connections and additionalAttorney Docket: 120179-0591W001read / write operations. These activities increase the computational load on the system and, ultimately, reduce overall database performance.

[0036] FIG. 2 is a diagram illustrating a direct access architecture for traditional database system architectures. Database system 200 comprises database 102, transaction 140, and audit action query processors 124-132, including audit action 1 query processor 124 and audit action 2 & 3 query processor 132.

[0037] As depicted, action query processors 124-132 directly access database 102 to continuously scan audit table 104 for potential changes. Once a change has been detected, action query processors 124-132 may perform appropriate processing steps, such as initiating actions or updating dependent systems.

[0038] Such direct access approaches eliminates reliance on an external message broker but introduces its own challenges. Continuous scanning places a significant load on database 102, as each action query processor repeatedly queries audit table 104. This load grows exponentially as more processors or components are added, severely impacting database performance. Furthermore, since action query processors operate independently, conflicts or contention for resources can arise, further exacerbating system inefficiencies.

[0039] Additionally, this architecture lacks scalability for modern distributed systems. As the number of action query processors increases to handle more complex workflows or higher throughput, the direct interaction with the database creates a bottleneck. Scaling the system requires provisioning more resources for database 102, which may not be feasible or cost-effective in all environ ents.

[0040] As person of skill in the art will appreciate that as illustrated in FIG. 3, the approaches in FIG. 1 and FIG. 2 can be combined to produce a hybrid architecture. FIG. 3 is a diagram illustrating a hybrid architecture for traditional database system architectures. Database system 300 comprises database 102, actor 240, message broker 250, queue processor 260, and audit action queue processors 302. In this architecture, the hybrid approach integrates both direct database access and message broker-based communication.

[0041] In database system 300, actor 240 continues to manage transactional updates to database 102, while also publishing messages to message broker 250. At the same time, audit action queue processors 302 directly access audit table 104 within database 102 to scan for changes. Message broker 250 facilitates asynchronous communication between components, such as delivering notifications or processing updates based on published messages. To achieve this functionality, the message broker is associated with queue processor 260 that handles actions initiated through messages. In this manner, hybrid system 300 leverages bothAttorney Docket: 120179-0591W001audit action queue processor 302 tor direct database-based actions and message broker queue processor 260 for message-based actions.

[0042] This hybrid model allows for greater flexibility in processing database updates. For example, message broker 250 can handle tasks that benefit from asynchronous execution, such as notifications to external systems or downstream components, while audit action queue processors 302 handle tasks requiring direct database queries, such as real-time replication or immediate data transformations.

[0043] However, despite its flexibility, architecture 300 inherits the limitations of both prior approaches. Direct access by audit action queue processors 302 still places a significant load on database 102 due to continuous queries, particularly as the number of processors scales. Similarly, actor 240’s interactions with both database 102 and message broker 250 perpetuate tlie “double transaction” problem, requiring complex error handling to account for possible failures in either system.

[0044] Moreover, the coordination between message broker 250 and audit action queue processors 302 introduces new challenges. Ensuring consistency between data published to the message broker and changes detected directly in audit table 104 requires additional synchronization mechanisms. These mechanisms may include periodic reconciliation processes or integrity checks to prevent data loss or duplication. This adds further complexity to the system, reducing overall performance.

[0045] In contrast, in embodiments herein, when a component “A” wishes to update a value in a database, the update is performed by calling a stored procedure in the database. Part of that stored procedure performs the requested update, but also inserts a record into the audit table. The contents of this audit record depend on the action and are configured to capture all operations performed on the table. ITe audit table is further configured for CDC, allowing the database to automatically generate CDC records for changes.

[0046] These CDC records are sent to the Audit Processor Component of the system, which processes them to determine the appropriate action, such as sending an email notification.

[0047] FIG. 4 is a diagram of an exemplary' asynchronous processing system according to various embodiments of the present disclosure. As depicted, system 400 comprises database 402, which comprises audit table 404 and CDC mechanism 406; CDC processor 408; raw CDC stream 410; raw stream processor 412; audit stream 414; audit stream processor 416; audit action queue processors 424-432; transaction 440; and file table 442. As depicted, raw CDC stream 410; raw' stream processor 412; audit stream 414; audit stream processor 416; and audit action queues 420-422 may be part of message broker 450.Attorney Docket: 120179-0591W001

[0048] In embodiments, CDC mechanism 406 is configured to access and read audit table 404, e.g., to detect and monitor changes in table entries, in response to updates to database 402. For example, such updates may include a user account balance change or other data manipulations. Updates are performed via a stored procedure that, in embodiments, inserts audit records into audit table 404. Exemplar}' audit records may include details about the updates and other events or operations performed on audit table 404, including transaction information, such as the source of the request, type of update, present and historical database information, affected fields, and the like. Such records may be entered, for example, in response to transactions like transaction 440, which can subsequently be published and asynchronously processed in response to a query, e.g., based on predetermined priorities. It is understood that CDC mechanism 406 may automatically generate CDC records corresponding to the detected changes in audit table 404, and may report suitable changes by any mechanism known in the art.

[0049] In embodiments, CDC processor 408 obtains the CDC records, which represent a record of changes made to audit table 404, and asynchronously processes those records before communicating them to subsequent components for further processing. In embodiments, asynchronous transmission ensures that downstream processing is decoupled from database updates. Alternatively, in embodiments, CDC processor 408 may also handle requests for specific CDC record information from other components with access to database 402.

[0050] In embodiments, CDC processor 408 transmits the CDC records to audit stream processor 416, which determines one or more actions to be performed based on the content of the CDC records. For example, an audit record that reflects a user transaction may result in a message action being triggered, e.g., a notification email that informs a user of transaction 440. In embodiments, based on a detected or determined action, CDC processor 408 may cause CDC mechanism 406 to insert an audit record into audit table 404. The audit record characterizes a change in a manner such that the content of the audit record reflects the action that was performed.

[0051] In greater detail, in embodiments, CDC processor 408 may output the changed data in the form of raw CDC stream 410, which can be represented as a list of audit entries. In embodiments, raw CDC stream 410 is provided to raw' stream processor 412, which translates the raw stream into a structured audit stream 414, e.g., in a format that is understood by audit stream processor 416. Once audit stream processor 416 obtains audit stream 414, it may process it such as to route the determined actions into one or more audit action queues 418-Attorney Docket: 120179-0591W001422, e.g., based on predefined criteria, such as the type or priority of the action, as discussed further with reference to FIG. 5.

[0052] Unlike traditional designs, using CDC records to determine actions based on changes in audit table 404 in this fashion, advantageously, eliminates the need for additional components in system 400 to query database 402 directly or other additional database operations, thus reducing database load and avoiding redundant operations in system 400.

[0053] Further, in embodiments, audit action queues 418-422, which correspond to actions triggered by changes to database 402 in FIG. 4, may be asynchronously processed by audit action queue processors 424-432. Processing queued actions independently and at different times, advantageously enables scalability and improves resource allocation. Indeed, various components in system 400 may operate asynchronously. For example, CDC processor 408 and audit stream processor 416 may operate independently enabling CDC processor 408 to extract CDC records from CDC mechanism 406 long before audit stream processor 416 uses their contents to determine what actions to perform. In embodiments, audit stream processor 416 may asynchronously generate different two or more audit action queues 418-422. Similarly, the messages in resulting queues 418-422 may be processed asynchronously, i.e., at different times.

[0054] As a further advantage of system 400 depicted in FIG. 4, is that audit action queue processors 424-432 may scale differently and independently from each other, e.g., based on the complexity or frequency of the actions they are tasked with. For example, some audit action queue processors 424-432 may handle relatively simple tasks such as sending user notifications, while others may perform more resource-intensive tasks, such as replicating data between different network regions or across geographic areas, which by far surpasses the processing capabilities of conventional database architectures.

[0055] It is understood that, in embodiments, any of CDC processor 408, raw CDC stream 410, raw' stream processor 412, audit stream 414, audit stream processor 416, or their combination, may be integrated into one or more components, such as a CDC stream processor, without altering the functionality or scope of system 400.

[0056] FIG. 5A depicts an exemplar}' CDC record according to various embodiments of the present disclosure. As shown, the CDC record comprises an entry having a value of “action 1,’’ which indicates that action 1 is to be inserted into the audit table. Conversely, the exemplary audit record of FIG. 5B that also has the value “action 1” that, when processed, causes the action 1 to be triggered.Attorney Docket: 120179-0591W001

[0057] FIG. 6 is a flowchart illustrating a process for asynchronous processing in a database management system by using CDC to monitor and process database changes according to various embodiments of the present disclosure.

[0058] In embodiments, process 600 may start at step 602, when a database system receives, e.g., from a component within the database system, a request to update a database, e.g., a certain value stored in the database.

[0059] In response to receiving the request, a stored procedure may then perform the update. A CDC processor may then cause the stored procedure to insert an audit record, which comprises information regarding the update, into an audit table within the database.

[0060] At step 604, in response to the audit record being inserted into the audit table, the CDC processor may automatically generate a CDC record that represents the change in the audit table and asynchronously communicate some or all of the CDC record to an audit stream processor or a message broker.

[0061] At step 606, the audit stream processor uses some or all of the content of the CDC record to determine one or more actions, without using another component within the database system to query the database. An exemplary action compromises a replication task that synchronizes data between different database instances.

[0062] In embodiments, the audit stream processor routes the action to an audit action queue, e.g., based on one or more predefined routing criteria, such as a priority or a an action type.

[0063] Finally, at step 608 the audit stream processor communicates the action(s) to an audit action queue processor that each is configured to initiate action(s), e.g., in response to process messages from one or more audit action queues. In embodiments, each action queue processor may be configured to scale independently from each other, e.g., based on an action frequency or complexity.

[0064] One skilled in the art shall recognize that: (1) certain steps may optionally be performed; (2) steps may not be limited to the specific order set forth herein; (3) certain steps may be performed in different orders; and (4) certain steps may be done concurrently.

[0065] FIG. 7 illustrates an example computing environment with an example computer device suitable for use in some example implementations, according to various embodiments of the present disclosure. Computer device 705 in computing environment 700 can include one or more processing units, cores, or processors 710, memory 715 (e.g., RAM, ROM, and / or the like), internal storage 720 (e.g., magnetic, optical, solid-state storage, and / or organic), and / or I / O interface 725, any of which can be coupled on a communication mechanism or busAttorney Docket: 120179-0591W001730 for communicating information or embedded in the computer device 705. I / O interface 725 is also configured to receive images from cameras or provide images to projectors or displays, depending on the desired implementation.

[0066] Computer device 705 can be communicatively coupled to input / user interface 735 and output device / interface 740. Either one or both of input / user interface 735 and output device / interface 740 can be a wired or wireless interface and can be detachable. Input / user interface 735 may include any device, component, sensor, or interface, physical or virtual, that can be used to provide input (e.g., buttons, touch-screen interface, keyboard, a pointing / cursor control, microphone, camera, braille, motion sensor, optical reader, and / or the like). Output device / interface 740 may include a display, television, monitor, printer, speaker, braille, or the like. In some example implementations, input / user interface 735 and output device / interface 740 can be embedded with or physically coupled to the computer device 705. In other example implementations, other computer devices may function as or provide the functions of input / user interface 735 and output device / interface 740 for a computer device 705.

[0067] Examples of computer device 705 may include highly mobile devices (e.g., smartphones, devices in vehicles and other machines, devices carried by humans and animals, and the like), mobile devices (e.g., tablets, notebooks, laptops, personal computers, portable televisions, radios, and the like), and devices not designed for mobility' (e.g., desktop computers, other computers, information kiosks, televisions with one or more processors embedded therein and / or coupled thereto, radios, and the like).

[0068] Computer device 705 can be communicatively coupled (e.g., via I / O interface 725) to external storage 745 and network 750 for communicating with any number of networked components, devices, and systems, including one or more computer devices of the same or different configurations. Computer device 705 or any connected computer device can be functioning as, providing services of, or referred to as a server, client, thin server, general machine, special-purpose machine, or another label.

[0069] I / O interface 725 can include wired and / or wireless interfaces using any communication or I / O protocols or standards (e.g., Ethernet, 802.1 lx. Universal System Bus, WiMax, modem, a. cellular network protocol, and the like) for communicating information to and / or from at least all the connected components, devices, and network in computing environment 700. Network 750 can be any network or combination of networks (e.g., the Internet, local area network, wide area network, a telephonic network, a cellular network, a satellite network, and the like).Attorney Docket: 120179-0591W001

[0070] Computer device 705 can use and / or communicate using computer-usable or computer-readable media, including transitory media and non-transitory media. Transitory media include transmission media (e.g., metal cables, fiber optics), signals, carrier waves, and the like. Non-transitory media include magnetic media (e.g., disks and tapes), optical media (e.g., CD ROM, digital video disks, Blu-ray disks), solid-state media (e.g., RAM, ROM, flash memory, solid-state storage), and other non-volatile storage or memory'.

[0071] Computer device 705 can be used to implement techniques, methods, applications, processes, or computer-executable instructions in some example computing environments. Computer-executable instructions can be retrieved from transitory media, and stored on and retrieved from non-transitory media. The executable instructions can originate from one or more of any programming, scripting, and machine languages (e.g., C, C++, C#, Java, Visual Basic, Python, Perl, JavaScript, and others).

[0072] Processors) 710 can execute under any operating system (OS) (not shown), in a native or virtual environment. One or more applications can be deployed that include logic unit 760, application programming interface (API) unit 765, input unit 770, output unit 775, and inter-unit communication mechanism 795 for the different units to communicate with each other, with the OS, and with other applications (not shown). The described units and elements can be varied in design, function, configuration, or implementation and are not limited to the descriptions provided. Processor(s) 710 can be in the form of hardware processors such as central processing units (CPUs) or a combination of hardware and software units.

[0073] In some example implementations, when information or an execution instruction is received by API unit 765, it may be communicated to one or more other units (e.g., logic unit 760, input unit 770, output unit 775). In some instances, logic unit 760 may be configured to control the information flow among the units and direct the services provided by API unit 765, input unit 770, and output unit 775, in some example implementations described above. For example, the flow of one or more processes or implementations may be controlled by logic unit 760 alone or in conjunction with API unit 765. The input unit 770 may be configured to obtain input for the calculations described in the example implementations, and the output unit 775 may be configured to provide output based on the calculations described in example implementations.

[0074] Processor(s) 710 can be configured to execute a method or computer instructions which can involve, in response to receiving from a component, at a database system, a request to update a database, updating the database, and inserting into an audit table within the databaseAttorney Docket: 120179-0591W001an audit record that comprises information regarding the update, as described, for example, with respect to FIG. 4 and FIG. 6.

[0075] Processor(s) 710 can be configured to execute a method or computer instructions which can involve using a CDC record, which has been automatically generated in response to the audit record being inserted into the audit table, to asynchronously communicate at least part of the CDC record to at least one of an audit stream processor or a message broker that uses a content of the CDC record to determine an action without using another component to query the database, wherein the CDC record represents a change in the audit table, as described, for example, with respect to FIG. 4 and FIG. 6. Processor(s) 710 can be further configured to execute a method or computer instructions which can involve communicating the action to an audit action queue processor that initiates the action, as described, for example, with respect to FIG. 4.

[0076] Some portions of the detailed description are presented in terms of algorithms and symbolic representations of operations within a computer. These algorithmic descriptions and symbolic representations are the means used by those skilled in the data processing arts to convey the essence of their innovations to others skilled in the art. An algorithm is a series of defined steps leading to a desired end state or result. In example implementations, the steps carried out require physical manipulations of tangible quantities to achieve a tangible result.

[0077] Unless specifically stated otherwise, as apparent from the discussion, it is appreciated that throughout the description, discussions utilizing terms such as “processing,” “computing,” “calculating,” “determining,” “displaying,” or the like, can include the actions and processes of a computer system or other information processing device that manipulates and transforms data represented as physical (electronic) quantities within the computer system’s registers and memories into other data similarly represented as physical quantities within the computer system’s memories or registers or other information storage, transmission or display devices.

[0078] Example implementations may also relate to an apparatus for performing the operations herein. This apparatus may be specially constructed for the required purposes, or it may include one or more general-purpose computers selectively activated or reconfigured by one or more computer programs. Such computer programs may be stored in a computer- readable medium, such as a computer-readable storage medium or a computer-readable signal medium, A computer-readable storage medium may involve tangible mediums such as optical disks, magnetic disks, read-only memories, random access memories, solid-state devices, drives, or any other types of tangible or non-transitory media suitable for storing electronicAttorney Docket: 120179-0591W001information. A computer-readable signal medium may include mediums such as carrier waves. The algorithms and displays presented herein are not inherently related to any particular computer or other apparatus. Computer programs can involve pure software implementations that involve instructions that perform the operations of the desired implementation.

[0079] Various general-purpose systems may be used with programs and modules in accordance with the examples herein, or it may prove convenient to construct a more specialized apparatus to perform desired method steps. In addition, the example implementations are not described with reference to any particular programming language. It will be appreciated that a variety of programming languages may be used to implement the techniques of the example implementations as described herein, The instructions of the programming language(s) may be executed by one or more processing devices, e.g., central processing units (CPUs), processors, or controllers.

[0080] As is known m the art, the operations described above can be performed by¬ hardware, software, or some combination of software and hardware. Various aspects of the example implementations may be implemented using circuits and logic devices (hardware), while other aspects may be implemented using instructions stored on a machine -readable medium (software), which if executed by a processor, would cause the processor to perform a method to carry out implementations of the present application. Further, some example implementations of the present application may be performed solely in hardware, whereas other example implementations may be performed solely in software. Moreover, the various functions described can be performed in a single unit, or can be spread across a number of components in any number of ways. When performed by software, the methods may be executed by a processor, such as a general-purpose computer, based on instructions stored on a computer-readable medium. If desired, the instructions can be stored on the medium in a compressed and / or encrypted format.

[0081] Moreover, other implementations of the present application will be apparent to those skilled in the art from consideration of the specification and practice of the techniques of the present application. Various aspects and / or components of the described example implementations may be used singly or in any combination. It is intended that the specification and example implementations be considered as examples only, with the true scope and spirit of the present application being indicated by the following claims.

Claims

Attorney Docket: 120179-0591W001ClaimsWhat is claimed is:

1. A database system for asynchronous processing, the database system comprising:a database comprising:an audit table; anda stored procedure configured to perform, in response to receiving from a component with access to the database system an instruction to change a value in the database, an update: anda change data capture (CDC) processor configured to perform steps comprising: performing the update;causing the stored procedure to insert into the audit table an audit record comprising information regarding the update;in response to the audit record being inserted into the audit table, automatically generating a CDC record that represents the change in the audit table; and and an audit stream processor configured to:in response to asynchronously receiving at least part of the CDC record, process the CDC record;based on a content of the CDC record, determine an action without requiring another component to query the database; androute the action to an audit action queue based on one or more predefined routing criteria comprising at least one of a priority or a an action type; and a set of audit action queue processors that each is configured to initiate one or more actions.

2. The system of claim 1, wherein each of the action queue processors in the set of audit action queue processors is configured to scale independently based on at least one of action frequency or action complexity.

3. The system of claim 1, wherein the stored procedure is configured to update a value in the database when inserting the audit record into the audit table.

4. The system of claim 1, wherein the set of audit action queue processors is configured to process messages from one or more audit action queues to perform one or more actions associated with the messages.

5. The database system of claim 4, wherein the one or more actions comprise a replication task that synchronizes data between different database instances.Attorney Docket: 120179-0591W0016. The system of claim 1, further comprising a raw stream processor configured to convert a raw CDC stream that comprises a list of audit entries that is output by the CDC processor into an audit stream that is compatible with the audit stream processor.

7. The system of claim 1, wherein at least two of the CDC processor, the audit stream processor, and the set of audit action queue processors operate asynchronously.

8. The system of claim 1, wherein at least one of the audit stream processor, the set of audit action queue processors, or the CDC processor cooperate with each other to retry a failed message transmission without requiring reprocessing of the CDC record.

9. A database method for asynchronous processing in a database management system, the method comprising:in response to receiving from a component, at a database system, a request to update a database, performing steps comprising:performing an update on the database; andinserting into an audit table within the database an audit record that comprises information regarding the update;using a change data capture (CDC) record, which has been automatically generated in response to the audit record being inserted into the audit table, to asynchronously communicate at least part of the CDC record to at least one of an audit stream processor that uses a content of the CDC record to determine an action without using another component to query the database, wherein the CDC record represents a change in the audit table; andcommunicating the action to an audit action queue processor that initiates the action.

10. The database method of claim 9, wherein the CDC record is communicated to the audit stream processor asynchronously with the action being communicated to the audit action queue processor.

11. The database method of claim 9, wherein the audit stream processor determines the action by referencing a pre-configured mapping of CDC records to audit action types.

12. The database method of claim 9, wherein the CDC record comprises information sufficient to perform the action, thereby reducing a need for the another component to query the database for additional state information.Attorney Docket: 120179-0591W00113. The database method of claim 9, further comprising independently scaling each action queue processors in a set of audit action queue processors based on at least one of action frequency or action complexity.

14. The database method of claim 13, further comprising using the set of audit action queue processors to process messages from one or more audit action queues to perform one or more actions comprising a replication task that synchronizes data between different database instances.

15. The database method of claim 9, further comprising converting a raw CDC stream output by a CDC processor into an audit stream that is compatible with the audit stream processor.

Images