Lambda Application Query Processing via Speed and Batch Layers
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Ad hoc queries in big data systems, particularly those involving historical data, face latency issues due to slow batch processing in conventional query processing techniques, especially when the query timeframe is modified repeatedly.
Innovation Solution
The method involves computing partial query results within an initial timeframe for both speed and batch processing layers, and upon modifications, identifies and computes additional results for unaddressed data segments, combining them into an updated final query result, utilizing a lambda architecture to minimize latency and optimize computing power usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If batch processing is used for historical data queries, then data accuracy is improved, but query latency increases
Solution Approach 1:
The patent segments the query processing into two independent layers: a speed processing layer that handles recent data in real-time with low latency, and a batch processing layer that handles historical data with high accuracy. By dividing the data processing into these segments, the system achieves both fast response for recent data and accurate results for historical data without the trade-off present in conventional single-layer systems.
Solution Approach 2:
The patent introduces a temporal dimension to data processing by maintaining different processing speeds for different time periods. Recent data is processed in real-time through the speed layer, while historical data is processed through the batch layer. This dimensional approach allows the system to optimize for both speed and accuracy simultaneously by applying different processing strategies to different time segments of the data.
2Speed
If pre-calculation of query results is performed, then query response speed is improved, but system complexity increases
Solution Approach 1:
The speed processing layer performs preliminary action by continuously processing and storing recent data in real-time, so that when a query is received, this pre-processed data is immediately available. This eliminates the need to reprocess recent data for each query, providing fast response speeds without requiring complex pre-calculation mechanisms for every possible query scenario.
Solution Approach 2:
The patent introduces an intermediary component - the serving layer - that coordinates between the speed processing layer and the batch processing layer. This intermediary manages query routing, result aggregation, and data flow between layers, simplifying the overall system architecture by providing a unified interface rather than requiring direct complex interactions between processing components.
3Adaptability or versatility
If query timeframe is modified repeatedly, then query adaptability is improved, but processing efficiency decreases
Solution Approach 1:
By segmenting data processing into speed and batch layers with different time horizons, the system can efficiently handle timeframe modifications. When a query timeframe is adjusted, only the relevant layer needs to be queried - recent timeframe adjustments affect only the speed layer, while historical timeframe changes affect the batch layer, avoiding complete reprocessing and maintaining high efficiency.
Solution Approach 2:
The system dynamically adapts to query timeframe modifications by selectively querying different layers based on the requested time range. The serving layer determines which layers to query based on the timeframe parameters, enabling the system to efficiently handle dynamic query requirements without fixed processing patterns, thus maintaining both adaptability and efficiency.
Data Source
AI summary
An approach is described of query processing via a lambda application including a batch processing layer and a speed processing layer. The approach may improve speed and efficiency with respect to computation of query results associated with the batch processing layer. A method pertaining to such approach may include receiving a query including an initial timeframe and computing partial query results within the initial timeframe. The partial query results within the initial timeframe may include a result for data associated with the speed processing layer and results for a first set of data segments associated with the batch processing layer. The method further may include combining the partial query results within the initial timeframe into a final query result. The method further may include computing partial query results outside of the initial timeframe including results for a second set of data segments associated with the batch processing layer.