Index Data Processing via Log Queue Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In database systems with significant scale and heavy loads, index generation and write processes cause increased delays, leading to user query delays.
Innovation Solution
A method and apparatus for processing index data that involves obtaining index data corresponding to target data written in a data table, determining a target log queue and log file based on the data table's log queue, and writing the index data into a write-ahead log (WAL) file for data playback to generate an index, with load balancing and dynamic allocation of log queues to ensure efficient index generation and reduce delays.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If index data is generated and written during data write operations in database systems with significant scale and heavy loads, then the index can be constructed for data inquiry, but the index generation and write processes result in increased delays in index writing, thereby causing user query delays
Solution Approach 1:
The patent segments the index writing process by introducing multiple log queues and multiple index log files. Instead of writing all index data to a single queue/file, the system divides index data into multiple segments that are distributed across different log queues and files. This segmentation reduces the writing load on any single component, thereby reducing index writing delays while maintaining complete index generation through the collective segments.
Solution Approach 2:
The patent implements preliminary action by using a write-ahead log (WAL) mechanism where index data is first written to index log files before being applied to the actual index structure. This preliminary writing to logs ensures that index data is captured and preserved before any potential failures, allowing for efficient batch processing and recovery without redoing the entire index generation process, thus reducing overall delays.
2Device complexity
If index data is written into a single log file during data write operations, then the index generation process is simple, but the system cannot handle large-scale data efficiently and may lose index data after power failure
Solution Approach 1:
The patent divides the single log file structure into multiple index log files organized under different log queues. Each log queue contains multiple index log files that store segments of index data. This segmentation allows the system to distribute the writing load across multiple files, improving data processing efficiency for large-scale data while maintaining a manageable structure through the queue-based organization.
Solution Approach 2:
The patent applies local quality by assigning different log queues and index log files to different data tables or index types. Each log queue/file can be optimized and managed independently based on its specific requirements, allowing efficient processing of different data segments while maintaining overall system integrity through the queue hierarchy.
3Productivity
If multiple log queues and index log files are used for index data writing, then load balancing and efficient processing are achieved, but the system complexity increases
Solution Approach 1:
The patent reduces management complexity through preliminary action by pre-establishing the log queue and index log file structure before data writing begins. The system pre-configures multiple log queues and their associated index log files, creating a ready-to-use framework that simplifies subsequent index data writing operations. This preliminary setup avoids dynamic complexity during runtime while enabling efficient load-balanced processing.
Data Source
AI summary
The present disclosure relates to a method for processing index data, a method for index generation, a medium and a device. The method for processing index data includes: in response to data being written into a data table, obtaining index data corresponding to target data written in the data table, the index data is data corresponding to an index field used for constructing an index in the target data; determining a target log queue and a target index log file into which each of the index data is to be written based on a log queue corresponding to the data table; and writing the index data corresponding to the target data into the target index log file in the target log queue.