Parallel Scan Manager for Time Series Data Search
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data search systems in DBMS face inefficiencies when handling large-volume time series data, particularly due to high CPU processing loads from decompression and conditional clause checks, and are limited in search performance despite improvements in disk speed, with existing indexing methods like bitmap and B+ tree being inefficient for large datasets with repeated keys.
Innovation Solution
A data search system that employs a query optimizer to determine if a parallel scan is possible, generating a query execution plan and utilizing parallel scan threads with bitmap indexes to divide record identifiers into ranges, allowing concurrent searches without reading repeated records, thereby optimizing search speed.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If sequential data access method or indexed search based on bitmap indexing is used, then search can be performed on large-volume time series data, but search speed is insufficient and CPU processing loads are high
Solution Approach 1:
The patent divides the large-volume time series data into multiple partitions or segments, allowing parallel processing of different data segments. This segmentation enables multiple search operations to be performed simultaneously across different partitions, reducing the overall search time and distributing the CPU processing load across multiple cores or processors, thereby resolving the contradiction between search speed and CPU processing load.
2Speed
If disk speed is improved using RAID or SSD, then input and output speed is increased, but CPU processing load for decompression and search target condition checks increases
Solution Approach 1:
The patent applies bitmap indexing during the data ingestion phase, pre-computing and storing the bitmap structures before search operations. This preliminary action ensures that when searches are performed, the CPU does not need to perform complex decompression and condition checking on the raw compressed data, as the bitmap indexes are already prepared and can be queried efficiently, thus reducing CPU processing load during search operations while maintaining high disk I/O speed benefits.
3Productivity
If bitmap-based indexing method is used, then indexes can be generated quickly for high-speed data input, but update operations are slow and not suitable for time series data with large amounts of repeated keys
Solution Approach 1:
The patent implements a dynamic indexing strategy where the choice of indexing method adapts based on the data characteristics and operational requirements. For time series data with repeated keys where updates are minimal, bitmap indexing is used to leverage its fast generation speed. When update operations become more frequent or data characteristics change, the system can transition to or combine with other indexing methods like B+ trees, providing adaptability while maintaining the benefits of fast index generation where applicable.
4Speed
If B+ tree search method is used, then search is efficient for small amounts of data or single record searches, but performance is highly inefficient for time series data having large amounts of repeated keys
Solution Approach 1:
The patent applies different indexing strategies to different portions or types of data based on their characteristics. For data portions with repeated keys in time series data, bitmap indexing is applied locally to handle the repetition efficiently. For unique keys or small data sets where B+ tree excels, that method is used locally. This local quality approach allows the system to optimize search performance for each data characteristic rather than using a single indexing method for all data, resolving the contradiction between search efficiency for small data and performance for large repeated keys.
Data Source
AI summary
The present disclosure relates to a data search system capable of providing increased speed in searching large-volume time series data in a DBMS. A data search system according to an embodiment of the invention may be implemented by a computing terminal as a system for searching data. The system may include: a query optimizer that checks whether or not a parallel scan is possible for a search target data table and generates a query execution plan; a parallel scan manager module that conducts a parallel scan on the table; and a query executer that executes the query execution plan generated by the query optimizer and, if a parallel scan is possible for the table, calls the parallel scan manager module, where the parallel scan manager module determines whether or not to execute a parallel scan by comparing the number of records forming the table with a preset value.