Object Paging for Archiving Under Resource Starvation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Archiving systems face resource starvation due to incorrectly labeled objects that should not be archived, leading to a situation where archivable objects are not archived, as the archiving component is overwhelmed by objects that are mistakenly marked as 'not allowed to be archived, resulting in a failure to delete them from the database.
Innovation Solution
Implementing a paging mechanism where the archiving component tracks the number of objects not allowed to be archived and adjusts the archiving process by skipping over a certain number of oldest objects based on this count, allowing for eventual archiving of all eligible objects.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If the archiving component attempts to archive all objects in a single batch, then the archiving process is simple, but the system experiences resource starvation when overwhelmed by objects incorrectly marked as 'not allowed to be archived'
Solution Approach 1:
The archiving process is divided into multiple pages or batches, where each page processes a limited number of objects. The system retrieves objects in pages, attempts to archive them, and tracks failures. This segmentation prevents resource starvation by limiting the number of objects processed simultaneously, while maintaining overall archiving productivity through iterative page processing.
2Speed
If the system processes a large number of objects in each archiving batch, then archiving speed increases, but objects incorrectly marked as 'not allowed to be archived' consume system resources and prevent valid objects from being archived
Solution Approach 1:
The system implements feedback by tracking the number of failed archiving attempts and using this information to adjust subsequent processing. When objects fail to archive (due to incorrect markings), the system records this and modifies the retrieval query to skip a proportional number of objects in future pages. This feedback mechanism ensures archiving completeness by preventing resource starvation, while maintaining speed through optimized query parameters.
3Productivity
If the archiving component skips objects marked as 'not allowed to be archived
Solution Approach 1:
The system performs preliminary actions by calculating and setting the skip parameter before retrieving object pages. Based on the failure rate from previous pages, the system pre-determines how many objects to skip in subsequent pages using the formula: skip parameter = (failure rate × total objects) / page size. This preliminary calculation prevents wasting time on objects that will fail, improving the successful archiving rate while minimizing total time through optimized processing paths.
Data Source
AI summary
A paging mechanism is provided for objects that are to be archived. An archiving component keeps track of the number of objects for which it has determined it is not allowed to archive (here called the paging amount). When a request to archive data is received by the archiving component, rather than necessarily attempt to archive the oldest objects, the archiving component instead skips a number of the oldest objects equal to the paging amount. The archiving component is invoked (e.g., each day), and then the paging amount can change based on the paging amount from the prior period (e.g., yesterday) and the newly requested objects to be archived that it determines are not allowed to be archived.