Autonomous Table Partitioning for Bulk Expired Metadata Cleanup
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data management systems face inefficiencies in managing expired metadata, leading to excessive processing resource consumption, prolonged operation times, and increased disk fragmentation due to row-by-row deletion, which degrades database performance and requires significant developer effort for table-specific optimizations.
Innovation Solution
Implementing table partitioning techniques with policy files that define partition generation and retention policies, allowing for autonomous partition management based on data retention requirements, enabling bulk deletion of expired data while preserving unexpired data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If row-by-row deletion is used to remove expired metadata, then the deletion process is simple to implement, but processing resource consumption increases and operation time prolongs
Solution Approach 1:
The patent segments the table into multiple partitions based on the partitioning category (e.g., time-based columns). Each partition contains a subset of rows, allowing the system to delete expired metadata at the partition level rather than row-by-row. This segmentation enables bulk deletion operations that are both efficient and maintainable.
Solution Approach 2:
The patent extracts and deletes entire partitions that contain only expired metadata, rather than individually removing rows. By identifying partitions where all rows have expired based on the partitioning category, the system can efficiently remove large volumes of expired data in single operations, significantly reducing processing overhead.
2Ease of manufacture
If row-by-row deletion is used to remove expired metadata, then the implementation is straightforward, but disk fragmentation increases
Solution Approach 1:
By organizing data into partitions based on the partitioning category, the patent creates structured segments that can be managed as units. When a partition contains only expired metadata, the entire partition can be deleted as a single unit, preventing the disk fragmentation that occurs with scattered row deletions and maintaining better storage efficiency.
3Productivity
If table-specific optimization is implemented for each table, then database performance improves, but developer effort and system complexity increase
Solution Approach 1:
The patent implements a universal table partitioning mechanism that can be applied across all tables in the database. By using a standardized partitioning category and policy file structure, the system achieves performance optimization for all tables without requiring table-specific customizations, thereby reducing developer effort and maintaining system simplicity.
Solution Approach 2:
The system automatically manages partition creation, maintenance, and deletion based on the partitioning category and retention policies stored in policy files. This self-service approach eliminates the need for manual table-specific optimizations, as the partitioning mechanism autonomously adapts to different tables and their expiration requirements.
4Productivity
If partitioning is implemented for all tables, then resource utilization is optimized, but implementation complexity increases
Solution Approach 1:
The patent creates a universal partitioning framework that works across all tables using a standard partitioning category. This approach optimizes resource utilization for the entire database while avoiding the complexity of implementing custom solutions for each table, as the same partitioning mechanism and policy file structure apply universally.
Solution Approach 2:
The partitioning system automatically configures and manages partitions based on metadata in policy files, eliminating the need for manual configuration. This self-service capability reduces implementation complexity while still achieving optimized resource utilization across all tables through consistent application of the partitioning framework.
Data Source
AI summary
Methods, systems, and devices for data management are described. A database instance associated with managing data of one or more users may be generated. In the database instance, a table of a database for a user of the one or more users may be generated. The table may be configured to store metadata for managing data of the user and may include a partitioning category. Based on generating the table, a file that indicates a policy for partitioning the table and for retaining partitions in the table may be identified. The policy may be based on a data retention policy associated with the data of the user. Based on identifying the file, partitions for the table may be generated in accordance with the partitioning category and the file, and based on generating the partitions, one or more partitions of the partitions may be deleted in accordance with the file.