Retention Lock Audit Log for Backup System Compliance
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for generating retention lock reports in large-scale data processing systems are resource-intensive and time-consuming, especially when dealing with hundreds of millions of files.
Innovation Solution
The implementation of a retention lock audit log (RLA log) stored in a hidden MTree, which allows for efficient generation and storage of retention lock reports, reducing the need for iterative file system walks and minimizing resource usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If iterative file system walk process is used to generate retention lock reports, then complete file locking history can be captured, but excessive time and resources are consumed
Solution Approach 1:
The system performs preliminary actions by maintaining an audit log that continuously records file locking events as they occur. This pre-captured data is stored in a structured format with timestamps, allowing the report generation process to simply retrieve and display pre-processed information rather than performing resource-intensive file system walks at report generation time.
Solution Approach 2:
The system creates a copy of the essential retention lock information in an audit log structure that is separate from the actual file system. This copied data structure contains all necessary locking history information in a compact, queryable format, eliminating the need to traverse the entire file system hierarchy when generating reports.
2Measurement precision
If iterative file system walk process is used to generate retention lock reports, then accurate file locking history is obtained, but excessive computational resources are consumed
Solution Approach 1:
The audit log performs preliminary recording of all file locking events with precise timestamps and file identifiers. This pre-captured data maintains accurate locking history while eliminating the need for resource-intensive file system traversal during report generation, as all necessary information is already recorded in the audit log structure.
3Productivity
If retention lock reports are generated frequently for data auditing, then data compliance can be monitored, but system performance degrades
Solution Approach 1:
The system maintains a copied audit log structure that contains all retention lock information in a compact, pre-processed format. This allows frequent report generation by simply querying the audit log copy rather than performing full file system walks, enabling high-frequency auditing without degrading system performance.
Solution Approach 2:
The audit log structure is designed to be self-sufficient for report generation, containing all necessary locking history information in a queryable format. This self-service capability allows the system to generate reports rapidly without requiring intensive computational resources or disrupting normal backup operations.
Data Source
AI summary
A retention lock report generation process creates, upon deployment of a backup server in the backup system, a hidden directory structure in local storage of the backup server. Upon retention locking (RL) of a file, the process writes details of the file and lock operation to a retention lock audit (RLA) log. The RLA log is stored in the hidden directory structure in the filesystem, and maintained as a rotating log containing only a latest timestamped RLA log file. Previous RLA log files are retention locked and stored differently from the RLA log. In a cascaded storage target system, a plurality of targets store copies of the sourced data. RLA log information is generated for locked files at each target for new lock attributes and lock state changes as propagated through the targets for both replication and back replication operations.