Precreated Compute Sessions for Online Big Data Query Execution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing big data techniques take unacceptably long times (e.g., 5-10 minutes) to execute queries and return results, making them unsuitable for online applications due to inefficiencies in creating compute sessions reactively and using conventional data processing methods.
Innovation Solution
A distributed computing system with preemptively created compute sessions and column-oriented data storage formats like Apache Parquet, combined with a distributed computing engine, enables efficient online execution of queries by proactively creating compute sessions and using formats like Apache Hive for direct query generation, facilitating fast data retrieval and processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional big data processing techniques are used, then data processing capability is maintained, but query execution time becomes unacceptably long (5-10 minutes)
Solution Approach 1:
The system pre-creates compute sessions and pre-processes data into column-oriented formats (Parquet, Hive) before queries are executed. This preliminary preparation eliminates the need for time-consuming session creation and data parsing during query execution, reducing query response time from minutes to seconds while maintaining processing capability.
Solution Approach 2:
The invention changes the data storage format parameter from conventional row-oriented formats to column-oriented formats (Parquet, Hive). This parameter change enables faster data retrieval and processing by allowing selective reading of only the required columns, significantly reducing query execution time while preserving data processing capabilities.
2Device complexity
If compute sessions are created reactively, then system simplicity is maintained, but query execution time increases significantly
Solution Approach 1:
The system implements preliminary creation of compute sessions before queries are received. By pre-establishing compute sessions and maintaining them in a ready state, the system eliminates the time-consuming reactive session creation process during query execution, reducing query response time from minutes to seconds.
3Reliability
If conventional data processing methods are used, then processing robustness is maintained, but query execution time becomes unacceptable for online applications
Solution Approach 1:
The invention changes the data storage format parameter to column-oriented formats (Parquet, Hive) which enable faster data retrieval and processing. This parameter change maintains processing robustness while significantly improving query execution speed from minutes to seconds, making the system suitable for online applications.
Solution Approach 2:
By pre-processing data into optimized column-oriented formats and pre-creating compute sessions, the system prepares everything needed for fast query execution in advance. This preliminary action maintains processing robustness while eliminating the need for time-consuming processing steps during query execution.
Data Source
AI summary
Techniques are disclosed relating to the execution of queries in an online manner. For example, in some embodiments, a server system may include a distributed computing system that, in turn, includes a distributed storage system operable to store transaction data associated with a plurality of users, and a distributed computing engine operable to perform distributed processing jobs based on the transaction data. In various embodiments, the server system preemptively creates a compute session on the distributed computing engine, where the compute session provides access to various functionalities of the distributed computing engine. The distributed computing engine may then use these preemptively created compute sessions to execute queries (e.g., for end users of the server system) against the transaction data and return the results dataset to the requesting users in an online manner.