Event Cleaning and Publication for Database Streaming
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Legacy databases fail to adequately support data cleansing, organization, and real-time streaming of data, limiting their value for downstream applications like AI and ML, and preventing concurrent delivery of messages related to events or transactions to downstream systems.
Innovation Solution
A method and system for receiving messages from a database, identifying changed tables, assigning event types based on these changes, and publishing them on an event streaming platform, ensuring that messages related to a specific event are streamed together, using a computing device with processors and memory to execute instructions for message extraction, parsing, and publishing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If legacy databases are used to store data, then data storage capacity is maintained, but data cleansing, organization, and real-time streaming capabilities are insufficient
Solution Approach 1:
The system segments the database architecture into multiple components: the legacy database itself, a message extraction layer, a message cleaning layer, and an event streaming platform. This segmentation allows each component to specialize in specific functions (storage, extraction, cleansing, streaming) thereby improving overall adaptability without requiring complete replacement of the legacy database infrastructure.
Solution Approach 2:
The patent introduces intermediary components between the legacy database and downstream systems, including message extraction mechanisms and cleaning processors. These intermediaries enable the legacy database to connect with modern streaming platforms without direct integration, improving versatility while maintaining the existing database architecture.
2Productivity
If messages related to events are forwarded to downstream systems, then data accessibility is improved, but concurrent delivery of all messages related to a specific event cannot be ensured
Solution Approach 1:
The system performs preliminary actions by assigning event types to messages before forwarding them to downstream systems. Messages are grouped and identified as belonging to specific events in advance, allowing the system to ensure concurrent delivery of all messages related to a particular event by routing them together through the event streaming platform.
Solution Approach 2:
The system implements feedback mechanisms to track and monitor message delivery status. By monitoring which messages have been delivered and ensuring that all messages belonging to the same event are received concurrently by downstream systems, the system can detect and correct delivery issues, thereby improving reliability.
3Loss of information
If data is extracted and cleansed for downstream use, then data value for AI and ML applications is enhanced, but processing time and system complexity increase
Solution Approach 1:
The patent implements continuous data extraction and cleansing operations that run in parallel with database operations. Rather than batch-processing data periodically, the system continuously extracts messages from the database, cleanses them in real-time, and streams them to downstream systems, thereby minimizing processing delays and maintaining data usability without significant time loss.
Data Source
AI summary
Systems and methods for cleaning and publishing event messages. An example method includes receiving messages corresponding to a plurality of events, identifying, for a first event of the plurality of events, one or more changed tables, assigning an event type to the first event based at least in part on the one or more changed tables, and publishing the first event on an event streaming platform.