Multithreaded Change Data Writing for Chronological Update Integrity
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In complex environments where database updates from multiple sources arrive in a non-deterministic order, existing systems compromise data integrity by allowing later updates to overwrite earlier updates with stale data, and these systems introduce processing delays or are limited to structured data formats.
Innovation Solution
A system utilizing a data integrity compute device that employs multiple threads to write transaction data to a staging table with millisecond precision, ensuring chronological order and preventing out-of-order updates by using a data pump to reconstruct the correct sequence before writing to a target data set like a data lake.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If protective measures are employed to guard against out of order writes, then data integrity is maintained, but processing delays occur that are unsuitable for high frequency or real time transaction updates
Solution Approach 1:
The system assigns a sequence number to each data update upon receipt, before the update is processed or written to the database. This preliminary tagging of updates with their temporal order information allows the system to maintain data integrity without requiring complex protective measures during processing, thereby avoiding processing delays.
Solution Approach 2:
The patent introduces an intermediary mechanism (sequence number) that mediates between the incoming updates and the database writing process. This sequence number acts as a mediator that carries the temporal ordering information through the system, allowing updates to be processed in parallel while ensuring correct chronological ordering when written to the database.
2Productivity
If updates from multiple sources are processed concurrently, then processing speed is improved, but data integrity is compromised when later updates overwrite earlier updates with stale data
Solution Approach 1:
The system performs preliminary action by assigning sequence numbers to updates as they are received, before any processing or database operations occur. This allows concurrent processing of multiple updates while maintaining the ability to reconstruct chronological order, thus enabling both high processing speed and data integrity.
Solution Approach 2:
The patent adds another dimension (sequence number) to the update data structure. This additional dimension allows the system to process updates concurrently in parallel while maintaining the temporal ordering information, effectively separating the processing dimension from the ordering dimension.
3Reliability
If protective measures are used to prevent out of order writes, then chronological order is maintained, but the system is restricted to data sets with rigid structure rather than supporting varying formats
Solution Approach 1:
The sequence number mechanism is a universal solution that works with any data format. By attaching the sequence number as metadata to the update record rather than requiring the data itself to have a specific structure, the system can handle structured, semi-structured, and unstructured data while maintaining chronological ordering.
Solution Approach 2:
The sequence number serves as an intermediary that connects the temporal ordering requirement with the diverse data formats. It mediates between the need for chronological order and the flexibility of varying data structures, allowing the system to maintain order without imposing rigid structural constraints on the actual data being processed.
Data Source
AI summary
Technologies for writing change data in real time as a transaction occurs using multithreading include a compute device. The compute device includes circuitry configured to obtain transaction data indicative of transactions associated with threads of a multithreaded environment. The circuitry may be further configured to provide the transaction data to a staging table to prevent overwrites of data present in a target data set with older data. Additionally, the circuitry may be configured to selectively write data from the staging table to the target data set to reconstruct a chronological sequence associated with the transactions.