Database Crawler Detecting Skewed Data in Multitenant Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In multitenant database systems, performance degradation occurs due to data skew, where excessive relationship depth between data elements strains resources, affecting all tenants and degrading access performance, as assumptions about access and resource allocation often mismatch the actual implementation.
Innovation Solution
Implementing a database crawler and scheduler to detect data skew by calculating relationship depth and setting soft limits, providing warnings, restricting access, and automatically managing resource usage to prevent performance degradation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If data relationships are allowed to grow deeply to satisfy user preferences and data associations, then data can be stored and accessed according to user needs, but system resources are consumed excessively and performance degrades
Solution Approach 1:
The system performs preliminary actions by calculating relationship depth before data access occurs. The database crawler proactively identifies skewed data relationships and the scheduler preemptively blocks further data access for affected tenants, preventing resource exhaustion before it impacts overall system performance.
Solution Approach 2:
The system implements feedback mechanisms where the database crawler continuously monitors relationship depths and provides feedback to the scheduler. When skewed data is detected, the feedback loop triggers automatic blocking of data access for specific tenants, creating a closed-loop control system that adapts to real-time conditions.
2Reliability
If a database crawler and scheduler are implemented to detect and manage data skew, then performance degradation can be prevented, but system complexity increases
Solution Approach 1:
The database management system performs self-service by automatically detecting data skew through the database crawler and autonomously managing the situation through the scheduler. The system identifies and blocks skewed data access without requiring external intervention or complex manual configuration, reducing operational complexity while maintaining reliability.
3Adaptability or versatility
If relationship depth is increased to accommodate complex data associations, then more comprehensive data relationships can be maintained, but resource strain increases affecting all tenants
Solution Approach 1:
The system extracts the harmful element by isolating and blocking access to specific data relationships that cause skew. The scheduler identifies which data associations are problematic and selectively blocks only those, allowing other legitimate data access to continue uninterrupted, thus removing the harmful resource strain while preserving useful functionality.
Data Source
AI summary
Detection of skew in an on-demand database services environment is provided. A request is generated to scan a multitenant database for skew indicated by relationship depth exceeding an expected limit. A database crawler calculates skew for tenant identifier for a particular table in the database. Any skew that is detected is identified for later resolution.