Hive Table Scanning via Update Cycle Rules
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The existing Hive table scanning methods are inefficient, requiring 200 hosts and at least a day to complete scan statistics, significantly slowing down machine learning processes due to the large magnitude of data in Hive tables, typically exceeding 100 million or more than one billion rows.
Innovation Solution
A Hive table scanning method that acquires update time information, determines an update cycle level based on a preset correspondence rule, and applies a target scanning rule from a preset correspondence table to optimize scan statistics, allowing for different scanning rules to be used based on the update cycle, thereby improving efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional Hive table scanning method is used, then complete scan statistics can be obtained, but it requires 200 hosts and at least one day to complete, significantly reducing scanning efficiency
Solution Approach 1:
The patent applies dynamics by making the scanning rule adjustable based on the update cycle of the Hive table. Different update cycles (daily, weekly, monthly) correspond to different scanning frequencies and rules, allowing the system to adapt dynamically to data change patterns and optimize scanning efficiency accordingly
Solution Approach 2:
The patent changes the scanning parameters (scanning frequency, scanning scope, scanning depth) based on the update cycle parameter. By establishing a correspondence relationship between update cycles and scanning rules, the system adjusts scanning parameters to match data update patterns, thereby reducing unnecessary scanning time while maintaining data accuracy
2Measurement precision
If Hive tables with large data magnitude (100 million or more than one billion rows) are scanned, then complete data statistics can be obtained, but the scanning process takes at least one day and requires 200 hosts, severely dragging down machine learning efficiency
Solution Approach 1:
The patent applies partial action by scanning only the necessary portion of data based on the update cycle. Instead of scanning the entire Hive table every time, the system scans only the updated portions or uses sampling methods for tables with long update cycles, thereby reducing scanning time while maintaining sufficient accuracy for machine learning tasks
Solution Approach 2:
The patent performs preliminary action by pre-establishing a correspondence relationship between update cycles and scanning rules. This pre-configured mapping allows the system to automatically select the optimal scanning strategy based on the table's update pattern, avoiding unnecessary scanning operations and improving machine learning efficiency
Data Source
AI summary
The present application discloses a Hive table scanning method, device, computer apparatus and storage medium, wherein the method includes: acquiring an update cycle of a Hive table; determining an update cycle level corresponding to the update cycle; determining a target scanning rule corresponding to the update cycle according to a preset correspondence table between the update cycle level and a preset scanning rule; and performing scan statistics to the Hive table according to the target scanning rule.