Iterative Database Partition Update With Memory Evaluation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Updating large databases efficiently is hindered by the need for significant computer resources and memory, leading to long update operations or failures due to memory limitations during batch-wise updates, especially when decompressing, updating, and recompressing data.
Innovation Solution
A database system and method that utilize two storage devices and a computing device to perform iterative updates of database partitions, where the available memory on one storage device is evaluated and used to generate and merge updated portions, minimizing memory usage and data traffic between devices.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If batch-wise update operation is executed on compressed database data, then the database can be updated, but significant computer resources and memory volume are required leading to long duration or failure
Solution Approach 1:
The partition to be updated is divided into multiple portions, each updated separately in iterative fashion. This segmentation allows the update operation to proceed with limited memory by processing one portion at a time rather than loading the entire partition into memory simultaneously.
Solution Approach 2:
The necessary memory space is evaluated and prepared before the update operation begins. By pre-assessing memory requirements and ensuring sufficient space is available, the system prevents update failures due to memory constraints while optimizing resource allocation.
2Reliability
If batch-wise update operation is executed on compressed database data, then the database can be updated, but the operation duration becomes very long
Solution Approach 1:
The update operation is segmented into iterative steps where portions are updated sequentially. This approach balances memory usage with processing efficiency, allowing the system to make progress on the update without requiring excessive memory that would slow down operations.
Solution Approach 2:
The iterative update process maintains continuous progress on the database update by systematically processing portions until completion. This continuous action ensures the update operation completes reliably without unnecessary interruptions or retries.
3Productivity
If memory volume is limited during update operation, then resource constraints are managed, but the update operation may fail
Solution Approach 1:
Memory availability is evaluated before initiating the update operation. This preliminary assessment ensures that the system only proceeds with updates when sufficient memory space is available, preventing failures while optimizing resource utilization by avoiding unnecessary memory allocation.
Solution Approach 2:
The partition is segmented into portions that can be updated within available memory constraints. This segmentation allows the system to successfully complete updates even with limited memory by processing data in manageable chunks rather than requiring the entire partition to fit in memory simultaneously.
Data Source
AI summary
A system includes a first and a second storage device and a computing device. A set of partitions includes a first subset of partitions and a second subset of partitions. The first storage device stores the first subset of partitions. The second storage device stores the second subset of partitions. The computing device is configured to execute an iterative update of a selected partition until a fully updated partition is generated. The updated partition generated in a current iteration is used as a selected partition in the next iteration, and a not yet updated partition is updated in every next following iteration.