Decoupled Index Caching for Flat-Wide Table Data Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In software-as-a-service (SaaS) platforms, especially in logical multi-tenant scenarios, customization of database tables leads to conflicts when multiple tenants modify the same database table, affecting live-network services and causing performance issues due to the need for simultaneous index record updates.
Innovation Solution
A data processing method that writes data records to a database in two phases: initially inserting the record and caching the index record, followed by a scheduled batch synchronization of the index records, using field and relationship indexes generated based on object type identifiers, field values, and relational metadata to improve performance and reduce conflict.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If index records are updated in real-time during data insertion, then data consistency is maintained, but system performance deteriorates and conflicts occur in multi-tenant environments
Solution Approach 1:
The patent applies preliminary action by pre-generating index records during data insertion and storing them in a buffer, then synchronizing to the database in batches at scheduled intervals. This allows data to be inserted and made queryable immediately without waiting for index creation, while maintaining eventual consistency through the batch synchronization process.
Solution Approach 2:
The patent segments the index update process into two distinct phases: (1) real-time buffer insertion during data insertion, and (2) batch synchronization to the database. This segmentation isolates the performance-critical insertion operation from the resource-intensive index building process, resolving the contradiction between speed and consistency.
2Adaptability or versatility
If multiple tenants modify the same database table simultaneously, then customization flexibility is achieved, but conflicts occur affecting live-network services
Solution Approach 1:
The patent introduces an index buffer as an intermediary layer between multiple tenants and the shared database table. Each tenant's data insertion operations write to this buffer independently without conflict, and the batch synchronization process handles multiple tenants' index records together, eliminating conflicts while preserving customization flexibility.
3Speed
If data records are written with full index updates, then query performance is improved, but writing speed deteriorates
Solution Approach 1:
The patent implements periodic action by synchronizing index records from the buffer to the database in scheduled batches rather than continuously. This periodic synchronization allows rapid data insertion at any time while maintaining query performance through regular index updates, resolving the speed-productivity trade-off.
Data Source
AI summary
A data processing method, apparatus, and system, where the method includes: receiving a data insertion request sent by service software, where the data insertion request includes a data record, and the data record includes an object type identifier, a field value, a field identifier, and an object instance identifier; generating a field index based on the object type identifier, the field value, and the object instance identifier; generating a relationship index based on the object type identifier; writing the data record to a database; and caching an index record and an operation for the index record into a cache apparatus, where the index record includes the relationship index and the field index. Using the foregoing solution, high-performance data processing of a flat-wide table is implemented.