Dynamic Data Partition Management for Fast Database Pruning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional database management systems face inefficiencies in handling large volumes of data, particularly in managing data addition and deletion operations, leading to resource-intensive and time-consuming processes due to the sequential processing of row operations, which is impractical for modern data objects.
Innovation Solution
A system and method for dynamic partition management that allows for adaptable data management by configuring database parameters, generating SQL statements, and concurrently executing operations using multiple threads and processes, enabling attach, detach, and drop operations based on partition keys, optimizing database utilization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional database management executes row operations sequentially, then data addition and deletion operations can be performed, but computation cycle time increases and resource consumption increases
Solution Approach 1:
The patent divides the data into partitions based on partition keys, allowing the database to process only relevant partitions during addition or deletion operations. This segmentation enables parallel processing of independent partitions, significantly reducing computation cycle time and increasing throughput for large-scale data operations.
Solution Approach 2:
The patent implements dynamic partition management where partitions can be attached to or detached from data objects based on operational needs. This dynamic structure allows the system to optimize data access patterns by bringing only necessary partitions into active memory, reducing overall computation time and resource consumption.
2Productivity
If the database executes addition or deletion operations line by line, then data management can be performed, but the operations become time-consuming and resource-expensive for large volumes of data
Solution Approach 1:
By segmenting data into partitions managed through a partition manager, the system can process data in batches rather than line-by-line. This enables parallel execution of addition and deletion operations across multiple partitions, dramatically improving efficiency and reducing computational resource consumption for large datasets.
Solution Approach 2:
The patent changes the fundamental parameter of data organization from individual row-level operations to partition-level operations. This parameter change allows the database to execute operations at a higher abstraction level, improving throughput and reducing the cumulative resource consumption associated with sequential row processing.
3Adaptability or versatility
If data objects are merged by parsing and executing row operations individually, then data can be incorporated, but the process becomes resource intensive and time-consuming
Solution Approach 1:
The patent segments data objects into partitions that can be independently managed and merged. When incorporating new data objects, the system can identify relevant partitions and execute merge operations at the partition level rather than parsing individual rows, significantly increasing merging speed while maintaining the ability to handle diverse data structures.
Solution Approach 2:
The patent implements preliminary partition creation and indexing before data merging operations. This preliminary structure allows the system to quickly locate and process only the necessary partitions during data object integration, avoiding the need to parse entire datasets and dramatically improving merging throughput.
Data Source
AI summary
A system and method for managing data lifecycles that can provide an adaptable data management system configured to optimize the lifecycles of data and the reliability of databases. The system can provide data management systems with the ability to attach, detach, and drop segments of data based on key partitions of the data objects to optimize database utilization. By including the key partitions to the data objects, the database including the data objects can now perform data management operations based on the key partitions. The system, based on the key partitions, can identify a location of the data object to attach row entries, detach row entries from the existing tables, and delete the detached row entries from the database altogether. The system can operate on any relational database management system.