Platform-agnostic Abstract Syntax Tree for Big Data Query Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The complexity of query languages and their processing frameworks, particularly in big data systems, makes it challenging to generate insights across diverse technologies like Hadoop, Hive, and Spark, as existing solutions are disruptive and not platform-agnostic, limiting real-time processing capabilities.
Innovation Solution
A method is developed to generate platform-agnostic abstract syntax trees (ASTs) by receiving data in a predefined format, parsing it to extract model information, transforming it into an AST structure, and enriching it with metadata, which allows for the creation of a platform-agnostic AST that can be executed across different engines like Hadoop, Hive, and Spark.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If query languages use diverse processing frameworks (Hadoop, Hive, Spark), then processing capabilities are enhanced, but interface layer complexity increases
Solution Approach 1:
The patent introduces an Abstract Syntax Tree (AST) as an intermediary layer between diverse query languages and processing frameworks. The AST serves as a universal intermediate representation that decouples the interface layer from specific framework implementations, allowing queries to be transformed into AST and then executed on different platforms without increasing interface complexity
Solution Approach 2:
The AST structure is designed as a universal data model that can represent queries across multiple processing frameworks (Hadoop, Hive, Spark, Map-Reduce). This single universal representation handles diverse query paradigms (SQL, procedural, object-oriented, graphical) without requiring separate interface layers for each framework
2Adaptability or versatility
If query languages follow different paradigms (SQL, procedural, object-oriented), then expressive power increases, but disruption to application development increases
Solution Approach 1:
The AST acts as a stable intermediary that absorbs the variability of different query paradigms. Applications interact with the AST interface rather than directly with framework-specific languages, isolating applications from paradigm changes and reducing disruption while maintaining support for multiple expressive paradigms
3Productivity
If big data processing uses Map-Reduce on Hadoop, then batch processing capability is achieved, but real-time processing capability is lost
Solution Approach 1:
The system dynamically adapts the execution plan based on the AST representation. The same AST can be executed on different processing engines (batch-oriented Map-Reduce or real-time Spark/Flink) without changing the query semantics, allowing the system to switch between batch and real-time processing modes as needed
4Productivity
If applications are hard-wired to specific technologies, then processing efficiency for that technology is optimized, but technology agnosticism is lost
Solution Approach 1:
The AST serves as a universal intermediate representation that maintains technology agnosticism while enabling optimized execution. The query compilation process generates AST that can be executed on any supported platform, allowing the system to leverage platform-specific optimizations without hard-wiring applications to specific technologies
Data Source
AI summary
A method generating a platform-agnostic abstract syntax tree (AST) comprises receiving data in a predefined format, through an input unit; subsequently parsing the data to extract model information corresponding to the predefined format of the data; and transforming, by a processing server, the model information to an abstract syntax tree (AST) structure. The above steps aid in generating, by the processing server, a platform-agnostic AST by combining predefined metadata and the abstract syntax tree (AST) structure.