Difference-Specific Database Updates for Lower Data I/O
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Updating data in databases often involves high data input/output (I/O) requirements due to the need to rewrite entire sets of data, which consumes excessive processing resources.
Innovation Solution
Implementing a data storage system that identifies and updates only the differences (deltas) between current and updated data sets, reducing the need to rewrite unchanged data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If entire data set is rewritten during database update, then data consistency is maintained, but data I/O operations and processing resource usage increase significantly
Solution Approach 1:
The patent segments the database update process into two distinct phases: (1) writing new data to a separate change data feed (CDF) table while leaving the original table unchanged, and (2) subsequently applying only the changed data to the target table. This segmentation eliminates the need to rewrite entire data sets, reducing I/O operations and processing resource usage while maintaining data consistency through the intermediary CDF mechanism.
Solution Approach 2:
The patent performs preliminary action by capturing and storing data changes in a Change Data Feed (CDF) table before applying them to the target database table. This preliminary capture of changes allows the system to update only the necessary data portions in subsequent operations, avoiding redundant rewriting of unchanged data and thereby reducing processing resource consumption while ensuring data consistency.
2Reliability
If entire data set is rewritten during database update, then updated data is ensured, but update time increases
Solution Approach 1:
The update process is segmented into change capture phase and change application phase. The CDF table stores changes separately, allowing the target table to be updated only with changed data in the second phase. This segmentation ensures updated data is achieved while significantly reducing update time by avoiding rewriting of unchanged data portions.
Solution Approach 2:
Instead of performing excessive action by rewriting the entire data set, the patent applies partial action by updating only the specific rows or columns that have changed, as identified and stored in the CDF table. This partial update approach ensures data is updated correctly while minimizing the time required for the update operation.
3Loss of energy
If difference-specific update is implemented, then data I/O operations are reduced, but complexity of identifying changes increases
Solution Approach 1:
The patent introduces a CDF table as an intermediary structure that automatically captures data changes through triggers or change data capture mechanisms. This intermediary simplifies the complexity of identifying changes by providing a centralized, structured repository of changes that can be easily queried and applied, reducing the overall system complexity compared to implementing complex change detection logic throughout the entire data processing pipeline.
Data Source
AI summary
In some implementations, a data storage system may receive target data associated with a first set of rows and a first set of columns. The data storage system may receive historical data associated with a second set of rows and a second set of columns. The data storage system may identify a set of differences between the target data and the historical data. The data storage system may perform, based on the set of differences between the target data and the historical data, a difference-specific update to a target database storing the historical data.


