Virtual Column Construction for Data Lake Query Automation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data query solutions for data lakes often result in a poor user experience due to manual input requirements and complexity in writing correct data query requests.
Innovation Solution
A virtual column construction method and data query method based on a data lake, which automatically determines expressions from analyzed statements, constructs virtual columns, and facilitates data queries through a user-friendly interface.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If manual input is required for data query requests, then users can directly access data, but user experience deteriorates due to complexity and error-proneness
Solution Approach 1:
The system automatically generates data query requests by analyzing user behavior and historical data without requiring manual input from users. The virtual column construction method autonomously identifies data patterns, formulates queries, and executes them, allowing the system to serve itself rather than relying on user complexity
Solution Approach 2:
A virtual column is introduced as an intermediary between the user and the underlying data query mechanism. This virtual column abstracts the complex query logic, presenting a simplified interface to users while handling the sophisticated data retrieval operations in the background
2Extent of automation
If virtual column construction is automated, then user experience improves, but system complexity increases
Solution Approach 1:
The system performs preliminary analysis of data patterns and user behavior in advance to pre-construct virtual columns before they are actually needed. By anticipating data access patterns and pre-processing the information, the system reduces the complexity of on-demand virtual column construction
Solution Approach 2:
The virtual column construction process is divided into distinct segments: data pattern analysis, expression determination, virtual column definition, and query generation. Each segment handles a specific aspect of the construction process, making the overall complex task more manageable and maintainable
3Measurement precision
If statistical analysis is performed on multiple expressions, then expression selection accuracy improves, but processing time increases
Solution Approach 1:
The system performs statistical analysis on a representative subset of expressions rather than exhaustively analyzing all possible expressions. By selecting key expressions for analysis based on relevance and frequency, the system achieves sufficient accuracy without the full time cost of complete analysis
Data Source
AI summary
The present application discloses a virtual column construction method based on data lake and a data query method. The present disclosure includes: first, automatically performing expression statistical analysis on a large quantity of statements to be analyzed in a data lake, so as to obtain an expression to be used; next, automatically constructing a virtual column construction request corresponding to the expression to be used, according to virtual column construction description information corresponding to the expression to be used, so that the virtual column construction request is used to request construction of a virtual column that can represent the expression to be used; then, according to the virtual column construction request, constructing a virtual column corresponding to the expression to be used, to enable the virtual column to represent the expression to be used.