Database Index Parallelization for Multi-Tenant Data Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Relational database systems are inefficient when handling data for multiple tenants due to interleaved data blocks, leading to expensive data-intensive processing as they often read data from other tenants, especially when dealing with thousands of tenants.
Innovation Solution
Implementing a system that uses an index structure to parallelize operations by dividing indexes into groups and assigning them to multiple threads for parallel processing, leveraging distributed compute resources to reduce runtime and scale with data growth.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data for multiple tenants is interleaved in a single table, then storage efficiency is improved, but data access efficiency deteriorates due to reading other tenants' data
Solution Approach 1:
The patent divides the single multi-tenant table into multiple separate tables, one for each tenant. This segmentation eliminates the interleaving of data from different tenants, allowing each tenant's data to be stored contiguously. When querying a specific tenant's data, the system only reads from that tenant's dedicated table, avoiding the need to scan and filter out other tenants' data, thus resolving the contradiction between storage efficiency and data access efficiency.
2Ease of operation
If a single table stores data for thousands of tenants, then ease of management is improved, but processing cost increases due to scanning all data
Solution Approach 1:
The patent segments the large multi-tenant table into multiple smaller tenant-specific tables. This segmentation reduces the amount of data that needs to be scanned during processing operations. When a processing task needs to operate on a specific tenant's data, it only scans that tenant's table rather than scanning through all data in a large combined table, significantly reducing processing costs and energy consumption while maintaining manageable access patterns.
3Reliability
If data-intensive processing scans all tenant data, then completeness of processing is improved, but processing time increases
Solution Approach 1:
The patent segments data processing by tenant, allowing processing operations to be performed on individual tenant tables rather than scanning all data in a large combined table. This segmentation maintains completeness of processing for each tenant's data while dramatically reducing processing time by eliminating unnecessary scans of other tenants' data. The system can also parallelize processing across multiple tenant tables simultaneously.
Solution Approach 2:
The patent performs preliminary organization of data into tenant-specific tables before processing operations are executed. This preliminary action of structuring data by tenant enables subsequent processing operations to directly access only the relevant data without needing to scan through unrelated data, thus maintaining processing completeness while reducing processing time.
Data Source
AI summary
Techniques are disclosed relating to processing data stored within database tables in a parallelized manner. In various embodiments, a computer system maintains a database having a database table storing data for a plurality of tenants and an index structure storing indexes for accessing data stored in the database table. In some embodiments, the computer system receives, from a particular tenant, a request to perform a task that includes operating on particular data. In some embodiments, the computer system determines a portion of the index structure corresponding to the particular tenant and divides a range of indexes within that portion into a plurality of index sub-ranges. In various embodiments, the computer system processes portions of the task in parallel by determining an index sub-range for each of a plurality of threads and causing each of the threads to operate on a portion of the particular data specified by its index sub-range.