Bulk Delete API Asynchronous Batch Processing Multitenant Database
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional database systems face inefficiencies when deleting large numbers of records, as they require significant resources and can lead to system bottlenecks, especially in multitenant databases, where multiple tenants share resources.
Innovation Solution
Implementing a bulk delete API that allows clients to make a single request for multiple deletions, which are then executed asynchronously in smaller batch jobs, optimizing resource usage and reducing the burden on the database system.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a large request is made to delete a large number of records all at once, then the deletion operation can be completed in a single request, but it requires significant network bandwidth, processing resources, process threads, and database connections
Solution Approach 1:
The patent divides a large delete operation into multiple smaller batch requests. Instead of processing all records in one request, the system segments the deletion into batches (e.g., 200 records per batch) and processes them sequentially. This reduces the resource burden of each individual request while still achieving complete deletion of large data sets.
Solution Approach 2:
The system implements periodic processing of delete batches with intervals between requests. Rather than sending all delete requests continuously, the system waits for resource availability and processes batches periodically, allowing other operations to share system resources and preventing resource monopolization.
2Use of energy by moving object
If many individual requests are generated to delete records, then resource consumption per request is reduced, but the time to generate and execute numerous requests increases
Solution Approach 1:
The system performs preliminary actions by preparing and queuing multiple delete requests in advance. Records are identified and batched for deletion before the actual delete operation begins. This preliminary preparation allows the system to efficiently manage the sequence of delete operations without time-consuming decision-making during execution.
Solution Approach 2:
The system maintains continuous useful action by processing delete batches back-to-back without unnecessary delays. Once a batch is ready, it is executed immediately, and the next batch follows continuously. This continuous processing minimizes idle time while still allowing resource management between batches.
3Use of energy by moving object
If manual deletion is performed for small numbers of records, then resource consumption is minimal, but the approach becomes inefficient when dealing with larger numbers of records
Solution Approach 1:
The system applies partial action by deleting records in batches rather than all at once. Each batch processes a manageable number of records (e.g., 200) which is excessive compared to manual deletion but controlled enough to avoid resource overload. This partial batching approach scales efficiently from small to large data sets.
Data Source
AI summary
A bulk delete for remote database access is provided. A delete request is received over an API for a multitenant database. The delete can be for either a soft delete or a hard delete, where hard delete allows the system to skip use of a recycle bin. Multiple batch jobs are created to implement the delete request, and the batch jobs are executed asynchronously with respect to the request until the request is satisfied. Results for each batch job can be stored and provided to a client to indicate results of the delete operation.