Immutable LSM Trees for Live Database Corruption Repair
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Multi-tenant database systems face significant challenges in addressing logical or physical corruption, which can lead to catastrophic failures, affecting numerous customers and resulting in substantial financial losses and reputational damage, due to the complexity and cost of rectifying corruption in conventional database systems.
Innovation Solution
The implementation of immutable LSM trees, which store records in a way that allows for the identification and correction of corruption within immutable extents through a log-based system, enabling corruption to be fixed without taking the database offline by replaying transactions and generating compensating transactions, thus maintaining system availability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional database systems detect data corruption, then data integrity is protected by crashing the database, but system availability and service continuity deteriorate
Solution Approach 1:
The database is segmented into multiple independent tenants, allowing corruption in one tenant to be isolated and corrected without affecting other tenants or the entire database system. This enables localized repair operations that maintain overall system availability while ensuring data integrity for affected segments.
Solution Approach 2:
The system performs preliminary actions by maintaining transaction logs and version histories before corruption occurs. When corruption is detected, these pre-recorded transactions are replayed to restore data to a consistent state, avoiding the need to crash the database and enabling continuous operation.
2Reliability
If database corruption is detected in multi-tenant systems, then catastrophic failure affecting multiple customers is prevented, but the complexity and cost of rectifying corruption increases
Solution Approach 1:
By dividing the database into tenant-specific segments with isolated transaction logs, the system reduces rectification complexity. Each tenant's corruption can be addressed using only their own transaction history, rather than requiring complex analysis of the entire multi-tenant database system.
Solution Approach 2:
The system creates copies of transaction logs and maintains historical versions of data. When corruption is detected, these copies are used to reconstruct the correct state, simplifying the rectification process by providing ready-made restoration sources without requiring complex repair algorithms.
3Ease of repair
If database corruption is addressed by taking the database offline, then thorough corruption repair is enabled, but downtime and financial losses increase
Solution Approach 1:
Transaction logs and data versions are continuously recorded and maintained before corruption occurs. When corruption is detected, these pre-existing records enable immediate repair operations without taking the database offline, as all necessary restoration information is already available in the logs.
Solution Approach 2:
The database system performs self-repair by automatically detecting corruption, replaying transactions from logs, and restoring data consistency without requiring external intervention or system shutdown. This autonomous repair capability eliminates downtime while maintaining thorough corruption remediation.
Data Source
AI summary
In accordance with disclosed embodiments, there are provided methods, systems, and apparatuses for fixing logical or physical corruption in databases using immutable LSM trees including, for example, means for storing records in a LSM database at the host organization; processing transactions to the LSM database by writing the transactions into immutable extents for subsequent merge into the LSM database; recording a history of the transactions in a log; identifying corruption in at least one of the records of the LSM database represented within the immutable extents prior to merge of the immutable extents to the LSM database; and correcting the identified corruption to the at least one of the records of the LSM database based at least in part on the recorded history of the transactions in the log during live operation of the LSM database. Other related embodiments are disclosed.