Container Location Database Snapshot Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The existing container location database (CLDB) in map-reduce file systems experiences inefficiencies due to high disk activity and contention from large snapshot information tables, leading to slowed transactions and increased resync times, which compromises overall cluster operations.
Innovation Solution
The solution involves reducing the number of key-value store transactions during snapshot creation and deletion by storing snapshot information compactly, eliminating temporary data, and leveraging data from file servers, thereby decreasing disk I/O and improving scalability and resilience.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If snapshot information tables are maintained in the CLDB to track all snapshots, then snapshot management capability is improved, but disk activity and transaction contention increase significantly
Solution Approach 1:
The patent extracts snapshot information management from the CLDB by introducing a separate snapshot information table maintained by the file server. This separates the snapshot tracking function from the container location database, allowing the CLDB to focus on container location services while snapshot information is managed independently, thereby reducing transaction contention and disk activity in the CLDB.
Solution Approach 2:
The file server acts as an intermediary between the CLDB and snapshot information. Instead of the CLDB directly managing snapshot tables, the file server mediates by maintaining snapshot information locally and interacting with the CLDB only when necessary for container location updates, reducing the frequency and intensity of transactions in the CLDB.
2Measurement precision
If comprehensive snapshot information is stored in the CLDB including snapcid lists and sizes, then snapshot tracking accuracy is improved, but resync time increases proportionally
Solution Approach 1:
The patent extracts detailed snapshot information (snapcid lists, sizes, and metadata) from the CLDB and stores it in a separate snapshot information table maintained by the file server. This extraction allows the CLDB to maintain only essential container location information, significantly reducing the data volume that needs to be synchronized during resync operations while preserving complete snapshot tracking capability in the separate table.
3Reliability
If the CLDB maintains large snapshot information tables to service transaction requests, then transaction completeness is improved, but cache churn increases and access to critical data slows down
Solution Approach 1:
The patent extracts snapshot information from the CLDB cache and maintains it in a separate table in the file server's local storage. This extraction reduces the size of data that must be cached in the CLDB, decreasing cache churn caused by frequent snapshot transactions and improving access speed to critical container location data that remains in the CLDB cache.
Data Source
AI summary
The efficiency of a container location database (“CLDB”) that is used for retrieving location information of data containers stored on nodes in a cluster is improved by decreasing the number of key-value store transactions performed during snapshot creation and deletion, thereby decreasing both disk I/O and computing cycles. The CLDB is made more scalable by improving the deletion and creation time of snapshots, allowing one to create and delete more snapshots during a given interval of time. CLDB resilience is improved by decreasing the time required to resync a CLDB node. CLDB data is reduced to decrease IO operations, improve resync times, and to cache most of the data in memory.