Database Random Sampling via Index Tree Scanning and Page Selection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The existing method of randomly sampling database tables frequently accesses the disk, consuming excessive disk I/O resources and reducing query performance, especially in distributed database systems with large data volumes.
Innovation Solution
A data processing method that utilizes an index tree scanning approach to traverse an index tree, randomly sampling data pages based on a generated random number, thereby reducing the need for full-table scanning and minimizing disk I/O operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If full-table scanning is performed to randomly sample database tables, then random sampling data can be obtained, but the number of disk I/O accesses increases significantly
Solution Approach 1:
The patent pre-loads data pages into memory before performing random sampling operations. By anticipating the sampling need and loading data in advance, the system reduces the number of disk I/O operations required during actual sampling, thus resolving the contradiction between obtaining accurate random samples and minimizing disk I/O resource consumption
Solution Approach 2:
The patent introduces memory as an intermediary between disk storage and sampling operations. Instead of directly accessing disk for random sampling, data is first transferred to memory where sampling can occur with minimal I/O overhead, effectively mediating between the disk storage system and the sampling requirement
2Measurement precision
If multiple data pages are loaded from disk to memory for random sampling, then sampling data can be obtained, but query performance deteriorates
Solution Approach 1:
The system performs preliminary data page loading into memory before random sampling operations. This advance preparation ensures that when sampling is needed, the data is already in memory, maintaining high query performance while ensuring accurate sampling results can be obtained
Solution Approach 2:
The patent loads more data pages into memory than the minimum required for sampling. By loading excessive data pages in advance, the system ensures that sufficient data is available in memory for random sampling operations, maintaining both sampling accuracy and query performance without repeated disk accesses
Data Source
AI summary
Embodiments of the present disclosure provide a data processing method, an electronic device, and a storage medium. In the embodiments of the present disclosure, in a process of obtaining random sampling data from the target database table, through an index tree scanning manner in combine with a mechanism for random sampling a data page of the database table, full-table scanning is not performed on the target database table, thereby effectively reducing the number of times that I/O access is performed on the disk, saving disk I/O resources, and enhancing the query performance of the database system, especially in the case that the distributed database system has mass data.