Real-Time Data Lake File Merging Based on Query-Aware Timing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing file merging methods in real-time data lakes, performed at fixed periods, lead to inefficient resource usage and increased costs due to varying data stream writing speeds, affecting query performance and resource overhead.
Innovation Solution
A method that determines the necessity of file merging based on query task characteristics and file attributes, predicting resource overheads before and after merging to optimize timing and reduce costs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If file merging is performed at fixed periods, then the data lake table can be maintained, but resource usage becomes inefficient and costs increase due to varying data stream writing speeds
Solution Approach 1:
The patent transforms the static fixed-period merging approach into a dynamic decision-making process that adapts to varying data stream writing speeds. The system evaluates query task characteristics and file attributes in real-time, adjusting merging decisions based on current system state and workload conditions, thereby optimizing resource usage while maintaining data lake table reliability.
Solution Approach 2:
The patent changes the parameter of merging timing from a fixed time interval to a condition-based decision variable. By introducing evaluation criteria involving query task characteristics (e.g., query frequency, data access patterns) and file attributes (e.g., file size, file age), the system determines merging timing based on optimal conditions rather than predetermined schedules, reducing unnecessary merging operations and improving resource efficiency.
2Productivity
If file merging is performed frequently, then query performance can be improved, but user costs increase due to unnecessary merging operations
Solution Approach 1:
The patent implements a feedback mechanism where the system continuously monitors query task characteristics and file attributes, evaluates the current state of the data lake table, and adjusts merging decisions accordingly. This feedback loop prevents unnecessary merging operations by only triggering merges when evaluation criteria indicate that merging will actually improve query performance, thereby reducing user costs while maintaining productivity benefits.
Solution Approach 2:
The patent applies partial action by performing merging operations only when necessary, rather than executing them at every fixed interval. The evaluation mechanism determines the precise conditions under which merging provides value, allowing the system to perform fewer, more targeted merging operations that achieve the necessary query performance improvement without incurring excessive user costs from unnecessary merges.
3Quantity of substance
If file merging is delayed, then user costs are reduced, but query performance deteriorates due to accumulation of small files
Solution Approach 1:
The patent performs preliminary evaluation of query task characteristics and file attributes before executing merging operations. By assessing conditions in advance and predicting whether merging will improve query performance, the system can proactively initiate merges when beneficial while avoiding unnecessary operations, thus balancing cost reduction with query performance maintenance without relying on delayed or reactive merging strategies.
Data Source
AI summary
Embodiments of the present disclosure provide a file merging method for a real-time data lake, a device and a storage medium. The method comprises: obtaining characteristic information of a query task for a real-time data lake table and attribute information of files to be merged in the real-time data lake table; determining, based on the characteristic information of the query task and the attribute information of the files to be merged, whether to initiate a merging task; and in accordance with a determination that the merging task is initiated, merging the files to be merged.