Segmented Data Query Files for Faster Iceberg Queries
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data query methods in the big data era suffer from low efficiency due to the need to traverse large amounts of data, resulting in prolonged query times for 'iceberg queries', where only a small part of the massive data is required.
Innovation Solution
A data query method that involves generating multiple smaller files from a larger initial file, allowing traversal of these smaller files to locate and retrieve target data, thereby reducing the amount of data that needs to be traversed.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is queried from a large first file, then complete data is available, but traversal time increases and query efficiency decreases
Solution Approach 1:
The patent divides the large first file into multiple smaller second files based on attribute values. Each second file contains a subset of the original data, allowing the system to traverse only relevant portions during queries rather than the entire large file, thus reducing query time while maintaining data completeness through the collective set of segmented files.
Solution Approach 2:
The patent performs preliminary segmentation of the large file into smaller second files before the actual query operation. This pre-processing action organizes data by attribute values in advance, enabling faster query execution by directly accessing relevant segmented files without traversing the entire dataset during query time.
2Productivity
If the first file is divided into multiple second files, then data traversal amount decreases and query efficiency improves, but file management complexity increases
Solution Approach 1:
The patent creates second files that maintain the same data structure and attribute schema as the original first file, allowing the same query processing logic to be applied universally across all segmented files. This multi-functionality approach enables the system to handle multiple files using a single unified processing framework, reducing the actual management complexity despite the increased number of files.
3Quantity of substance
If aggregation operation is performed on the first file, then data amount is reduced, but aggregation computation time is consumed
Solution Approach 1:
The patent performs aggregation operations on segmented portions of data rather than the entire dataset at once. By dividing the first file into smaller second files based on attribute values and performing aggregation on these smaller units, the system reduces the computational burden of each aggregation operation while achieving the same overall data reduction effect.
Solution Approach 2:
The patent performs aggregation operations as a preliminary step before creating the segmented second files. This pre-aggregation action reduces the data amount in advance, so that when queries are executed on the segmented files, the data volume to be processed is already minimized, thereby reducing subsequent query computation time.
Data Source
AI summary
Example data query methods, apparatuses, and computer-readable medium are provided. After receiving a data query request, a data processing system queries target data required by a user from second files with a smaller data amount instead of a first file with a larger data amount. The second files are specifically generated based on the first file. Both the first file and the second file may be files formed by aggregated data or files formed by original data that is not aggregated. Alternatively, the first file is a file formed by original data, and the second file is a file formed by aggregated data. When data is queried, the second file with the smaller data amount is traversed.