Metadata Processing Bypassing Logs During High Busyness
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Legacy metadata processing solutions face performance issues when computing devices are in a busy state, as they rely heavily on log creation, leading to poor execution performance due to insufficient storage space in the log storage apparatus, preventing timely execution of modification transactions.
Innovation Solution
The method involves determining the busy degree of the computing device and, if it exceeds a threshold, modifying metadata directly in the metadata storage apparatus without generating a log in the log storage apparatus, allowing immediate transaction execution even in busy states, while generating logs when the device is not busy to ensure fast response times.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If logs are generated for metadata modification transactions, then data consistency and reliability are improved, but processing speed and productivity deteriorate when the computing device is in a busy state
Solution Approach 1:
The system dynamically adjusts the metadata processing approach based on the busy degree of the computing device. When the device is not busy, logs are generated to ensure data consistency. When the device is busy, direct modification is performed to maintain processing speed. This dynamic adaptation resolves the contradiction between reliability and productivity.
Solution Approach 2:
The invention changes the processing parameter (whether to generate logs) based on the system state (busy degree). By monitoring the busy degree parameter and adjusting the processing mode accordingly, the system achieves both data consistency when resources are available and high processing speed when resources are constrained.
2Reliability
If logs are generated in the log storage apparatus, then transaction tracking and reliability are improved, but storage space availability and processing efficiency worsen when the log storage apparatus is full
Solution Approach 1:
The invention extracts the log generation step from the mandatory processing path and makes it conditional. By taking out the log creation operation and placing it within a conditional check (based on busy degree and storage space), the system maintains transaction tracking reliability when possible while avoiding storage space exhaustion that would prevent processing.
Solution Approach 2:
The system performs partial logging - generating logs only when the storage apparatus has sufficient space and the device is not busy. This partial action approach ensures that logging operations are performed selectively, maintaining reliability for transactions that can be logged while avoiding the harmful effect of storage exhaustion.
3Productivity
If direct metadata modification is performed without logs, then processing speed and productivity are improved, but data consistency and reliability deteriorate
Solution Approach 1:
The system dynamically switches between two processing modes: direct modification mode (for speed) and logged modification mode (for consistency). The switching decision is based on the busy degree of the computing device, ensuring that direct modification is only used when the system state allows for potential recovery if needed.
4Reliability
If the computing device processes all metadata transactions through log creation, then data integrity is maintained, but IOPS performance and response time deteriorate in busy states
Solution Approach 1:
The invention changes the processing parameter (log generation) based on the system state parameter (busy degree). When the busy degree exceeds a threshold, the system switches to direct modification to reduce response time. This parameter-based adaptation resolves the contradiction between maintaining data integrity and reducing response time losses.
Data Source
AI summary
Techniques to process metadata involve: in response to receiving a modification transaction for metadata on a computing device, acquiring information related to a busyness degree of the computing device; and according to at least a determination that the busyness degree is greater than a threshold degree, modifying the metadata in a metadata storage apparatus of the computing device directly without generating a log related to the modification transaction in a log storage apparatus of the computing device. Accordingly, such techniques can achieve good processing performance irrespective of a busy state or non-busy state of a computing device. In particular, when the computing device is in a busy state, a newly triggered modification transaction can still be timely executed, thereby improving IOPS performance of the computing device in a busy state.